Leaderboard
Standings
The development board tracks the public split and updates on every merged submission. Final standings are computed once, on held-out models, after the submission deadline.
Stage 1 scores against StarCoder2, updated live on Kaggle until the submission deadline on 4 Dec 2026. These give you feedback during development but do not determine the final ranking — that comes from the Stage 2 re-run against an undisclosed model.
Scored as ½ (AUC-ROC + Snovelty). How the score works →
| # | Method | StarCoder2-3B | Mellum-4B held-out |
|---|---|---|---|
| 01 | No submissions yet The live Stage 1 standings are on Kaggle. This table mirrors them once results are synced. | 0.000 | 0.000 |
Final standings from the first edition, computed on held-out models at FSE 2026. Baselines are pinned for reference.
Not comparable
These are AUC-ROC only. The 2027 edition scores
½ (AUC-ROC + Snovelty), so a 2026 figure and
a 2027 figure are not measuring the same thing and should not be read against each other. The
task and dataset design are unchanged between editions; the scoring is not.
| # | Method | StarCoder2-3B | Mellum-4B held-out |
|---|---|---|---|
| 01 | SERSEM Dikici, Kara, Çağlar, Tüzün, Sav Selective entropy-weighted scoring. Down-weights predictable syntactic boilerplate and emphasises comments, long identifiers, string literals and developer markers, combining weighted token-level logit scoring with probes of intermediate transformer activations. | 0.773 | 0.753 |
| 02 | CalibratedProbs Berndt, Bekmyradov, Bach, Baltes Probability-based signals adapted to source code. Adjusts average token log-probability by token diversity, on the intuition that repetitive code is easy to predict regardless of membership. | 0.591 | 0.607 |
| — | MinK%Prob
baseline
Baseline — scores only the lowest-probability tokens in a file. | 0.597 | 0.613 |
| — | Loss
baseline
Baseline — average token-level negative log-likelihood. | 0.594 | 0.611 |
| — | PAC
baseline
* Baseline — compares the model's behaviour on original and perturbed inputs. | 0.563 | 0.568 |
- * PAC: Evaluated on 1,000 samples due to runtime constraints; all other methods on 5,000.