The task
Detect what the model already knows.
Given white-box access to a code LLM and a source file, decide whether that file was part of the model's training data. Scored on accuracy and on finding members that earlier attacks missed.
Register and submit on Kaggle
This page is the specification. Kaggle is where you actually compete — dataset hosting, notebook compute with GPUs at no cost, automated scoring, the live leaderboard and the discussion forum that serves as the main support channel.
What you are given, and what you return
Two stages, one of which counts
The competition separates building a method from proving it generalises. You develop against a known model and are ranked against an unknown one.
Development
You build against StarCoder2 as the target model, using the released member and non-member files. Submitted notebooks are scored on a held-out split from the same distribution and appear on the live leaderboard. These scores give you feedback — they do not determine the final ranking.
Generalization
After the deadline, your selected notebook is re-executed unmodified against the same evaluation data but a target model that was never disclosed. The final competition ranking comes solely from this stage. The evaluation data is fixed and the model changes, so what is being measured is whether your method transfers.
Guaranteed about the Stage 2 model
- The Stage 2 target is a decoder-only causal language model trained on source code.
- You get full white-box access to per-token logits, losses and hidden states.
- The model fits within the memory of the reference notebook accelerator.
Explicitly not guaranteed
- It may not be the same model family as StarCoder2.
- It may not be the same parameter scale.
- Tokenizer, vocabulary size, context length, hidden dimension and layer count may all differ.
The Stage 2 model has already been chosen and the datasets audited. Which model it is will not be released until after the competition concludes.
How the data is built
Members and non-members come from two disjoint sources, then get filtered so that membership cannot be read off surface statistics.
Members
Drawn from the StarCoder2 training data — files the Stage 1 target provably saw during training.
Non-members
Drawn from The Heap, which is deduplicated against the StarCoder2 training data — so these files are neither exact nor near duplicates of anything the model saw.
Distribution filtering
Two different source corpora differ in more than membership — topics, repositories and coding conventions all vary, and any of those could dominate the signal. To suppress that, a bag-of-words classifier is trained to separate members from non-members, and only the samples it misclassifies are kept. What survives cannot be resolved by spotting superficial differences between the two sources.
Splits per language
Five languages, chosen to test whether methods generalise across them: Go, Java, Python, Ruby, Rust. That is 300,000 samples in total.
Accuracy, and finding what others missed
AUC-ROC measures how well an attack separates members from non-members, but it cannot tell apart two methods that succeed on the same examples from two that find different ones. This edition therefore scores on a second axis: whether you surface members that earlier attacks never caught.
The final score
The arithmetic mean of the two components. Both lie in [0, 1], so the final score does too.
The novelty component
- T
- every true member in the evaluation set
- M
- the members your submission identifies at a 1% false-positive rate
- N
- the number of reference attacks — all previous submissions and baselines
- kᵢ
- how many of those reference attacks found member i
Worked example
Suppose there are 8 reference attacks and only 3 of them ever identified a particular member. Finding that member yourself is worth 1 − 3/8 = 5/8 of a point. A member that every previous attack already caught is worth almost nothing.
Why it is built this way
Dividing by the total number of true members means you cannot win on novelty by finding a handful of exotic samples and nothing else — you have to find many members and unusual ones. The reference set is fixed before the competition opens, so your novelty score is computable the moment you submit, not after everyone else is evaluated.
Tie-breakers
Applied in order when scores are equal. These reflect properties that matter for real deployment but are not what the competition is primarily about.
Note
The first edition ranked submissions on AUC-ROC alone. Because this edition averages AUC-ROC
with the novelty score, scores from 2026 and 2027 are not directly
comparable — only the AUC-ROC component is measured the same way. See the
2026 results for last year's figures.
What your notebook must do
Stage 2 re-executes your code unmodified against a model you have never seen. These four rules are what make that possible — a notebook that breaks any of them can score in Stage 1 and still fail when it counts.
Load the model from the fixed path
The target model is attached to your code at a fixed path and loaded through the standard transformers interface. Read the model and tokenizer from that path — never from a hardcoded identifier, or Stage 2 will fail.
Assume nothing about its shape
Do not hardcode tokenizer behaviour, vocabulary size, context length, hidden dimension or number of layers. Any of these may change between stages, and a method that depends on one of them will not survive the re-run.
Score every file in the manifest
Evaluation files arrive in a fixed input directory with a manifest. Emit exactly one continuous score per file, where larger values mean a higher likelihood of membership.
Attach auxiliary resources before the deadline
Auxiliary models, reference corpora and supervised classifiers trained on Stage 1 labels are all allowed — provided they are attached as public Kaggle datasets before the submission deadline and declared in your report. Anything not attached beforehand simply will not exist at re-run time.
Each stage carries a fixed wall-clock budget. Because Kaggle executes every submission in an identical environment, a notebook that scores is by construction a notebook that runs. After the deadline there is a two-week window in which the organizers re-execute all selected notebooks against the Stage 2 model — teams whose notebooks fail to complete are contacted.