The Poisoned
Chalice
Code LLMs memorise what they are trained on — which quietly poisons every benchmark built from public code. Your task: given a model and a file, decide whether the model was trained on it.
Winning attacks are used to audit The Heap and build a cleaner benchmark for the community.
- Submission deadline 4 Dec 2026
- Stage 2 re-execution window opens 4 Dec 2026
- Reviewer response 16 Dec 2026
A white-box membership inference attack, in three moves.
You get the weights
Full white-box access to the target model: architecture, parameters, layer outputs, logits, attention maps. Hooks and probes are fair game.
You get a file
A source file in one of the target languages. Easy giveaways are filtered out — temporal shifts and bag-of-words separability are already stripped from the pool.
You answer one question
Was this file in the training set? Emit a membership score. We rank you by AUC-ROC on held-out models and datasets you have never seen.
Run a baseline in five minutes.
The starter kit is a single notebook: it loads a language split, scores every file with a negative-loss baseline against StarCoder2-3B, and reports ROC-AUC. Replace one function with your own and you have an entry.
starter kit →git clone https://github.com/Poisoned-Chalice/starterkit-ICSE2027.git
cd starterkit-ICSE2027
python -m pip install -r requirements.txt
jupyter lab mia_starter.ipynb
# Run all cells → ROC-AUC for the negative-loss
# baseline on StarCoder2-3B. Now beat it.The bar to beat.
| # | Method | Mellum-4B held-out |
|---|---|---|
| 01 | SERSEM Dikici, Kara, Çağlar, Tüzün, Sav Selective entropy-weighted scoring. Down-weights predictable syntactic boilerplate and emphasises comments, long identifiers, string literals and developer markers, combining weighted token-level logit scoring with probes of intermediate transformer activations. | 0.753 |
| 02 | CalibratedProbs Berndt, Bekmyradov, Bach, Baltes Probability-based signals adapted to source code. Adjusts average token log-probability by token diversity, on the intuition that repetitive code is easy to predict regardless of membership. | 0.607 |
| — | MinK%Prob
baseline
Baseline — scores only the lowest-probability tokens in a file. | 0.613 |
| — | Loss
baseline
Baseline — average token-level negative log-likelihood. | 0.611 |
AUC-ROC on the held-out model, from the 2026 competition report. Not comparable to 2027 scoring.
Take a sip.
Stage 1 is open. Submissions close 4 Dec 2026 AoE. No conference registration required to compete.