First edition · FSE 2026 · Montreal

What happened last year

The inaugural edition ran white-box membership inference against code LLMs, ranking submissions on a held-out model they had never seen. Two teams submitted; one of them cleared the baselines by a wide margin.

1st

SERSEM

Dikici, Kara, Çağlar, Tüzün, Sav

Selective entropy-weighted scoring. Down-weights predictable syntactic boilerplate and emphasises comments, long identifiers, string literals and developer markers, combining weighted token-level logit scoring with probes of intermediate transformer activations.

0.753 AUC-ROC on held-out Mellum-4B

paper ↗ code ↗

2nd

CalibratedProbs

Berndt, Bekmyradov, Bach, Baltes

Probability-based signals adapted to source code. Adjusts average token log-probability by token diversity, on the intuition that repetitive code is easy to predict regardless of membership.

0.607 AUC-ROC on held-out Mellum-4B

paper and code not yet public

These scores are not comparable to 2027

The first edition ranked submissions on AUC-ROC alone. The 2027 edition keeps the same task formulation and dataset design, but changes the scoring: the final figure is the mean of AUC-ROC and a novelty score that rewards finding members earlier attacks missed. A number on this page and a number on the 2027 leaderboard are therefore measuring different things — only the AUC-ROC component is like for like.

how 2027 scoring works →

Full standings

Submissions and baselines

Ranked by AUC-ROC on the held-out Mellum-4B model. Figures transcribed from Figure 1 and Section 4 of the competition report.

#	Method	StarCoder2-3B	Mellum-4B held-out
01	SERSEM Dikici, Kara, Çağlar, Tüzün, Sav Selective entropy-weighted scoring. Down-weights predictable syntactic boilerplate and emphasises comments, long identifiers, string literals and developer markers, combining weighted token-level logit scoring with probes of intermediate transformer activations. paper ↗ code ↗	0.773	0.753
02	CalibratedProbs Berndt, Bekmyradov, Bach, Baltes Probability-based signals adapted to source code. Adjusts average token log-probability by token diversity, on the intuition that repetitive code is easy to predict regardless of membership.	0.591	0.607
—	MinK%Prob baseline Baseline — scores only the lowest-probability tokens in a file.	0.597	0.613
—	Loss baseline Baseline — average token-level negative log-likelihood.	0.594	0.611
—	PAC baseline * Baseline — compares the model's behaviour on original and perturbed inputs.	0.563	0.568

* PAC: Evaluated on 1,000 samples due to runtime constraints; all other methods on 5,000.

Setup

How the first edition was run

Target models StarCoder2-3B and StarCoder2-7B for development; JetBrains Mellum-4B held out

Members The Stack Edu — a subset of The Stack v2, in the training data of both model families

Non-members The Heap — deduplicated against The Stack v2, filtered on near-duplicates

Languages Go, Java, Python, Ruby, Rust

Metric AUC-ROC, chosen over TPR@xFPR because no consensus FPR exists for this task

Baselines Loss, MinK%Prob and PAC

What we learned

Takeaways carried into edition 2

The curated setting was hard. Every probability-based method — the three baselines and one of the two submissions — clustered just above random, between 0.563 and 0.613 AUC on the held-out model. Likelihood signals alone were not enough.
Structure-aware scoring cleared that cluster by roughly 14 percentage points. SERSEM reached 0.773 on StarCoder2-3B and 0.753 on the held-out Mellum, so its advantage survived the change of target model rather than being tuned to one checkpoint.
Development-set gains did not always transfer. CalibratedProbs improved on the baselines during participant-side experiments, but on the final held-out evaluation it fell back into the same band as the methods it had beaten.
Cost is the open problem. Every attack, baselines included, requires running files through an LLM — and perturbation-based methods multiply that. The report suggests probing fewer layers as one route to making these attacks practical.