Frequently asked

No. You can compete and present your solution without registering for the conference and without including your report in the proceedings. Proceedings inclusion is optional.

Clone the starter kit and open mia_starter.ipynb. It is a single notebook that loads one language split, scores every file against StarCoder2-3B, reports ROC-AUC and writes predictions. Three ways to run it: pull the prebuilt container (ghcr.io/poisoned-chalice/poisoned-chalice-starter:icse-2027), set up a local environment with Python 3.10–3.12, or upload the notebook straight to Kaggle and select a GPU accelerator.

Five case-sensitive language configurations — Go, Java, Python, Ruby and Rust — each with train and validation splits. Every row has a content column holding the source file and a membership column labelled member or non-member. Per language there are 50,000 training and 5,000 validation samples; a further 5,000 test samples are held out and never released.

White-box. The target model is attached to your notebook at a fixed path and loaded through the standard transformers interface, with full access to per-token logits, losses and hidden states.

In Stage 1 you develop against StarCoder2 — the starter kit uses bigcode/starcoder2-3b. In Stage 2 your notebook is re-run against a model that is not disclosed until the competition ends. It is guaranteed to be a decoder-only causal language model trained on source code that fits the reference accelerator, but it may differ from StarCoder2 in family, parameter scale, tokenizer, vocabulary size, context length, hidden dimension and layer count.

No. Submissions run on Kaggle notebook compute, including GPUs, at no cost to participants. Each stage carries a fixed wall-clock budget. CPU execution works with enough memory but is very slow. On Kaggle, do not replace the preinstalled PyTorch build — it is paired with the notebook image's CUDA runtime.

The final score is the mean of two components: AUC-ROC, and a novelty score that rewards identifying members which earlier attacks missed, measured at a 1% false-positive rate. The full formula is on the Task page.

Stage 2 only. The Stage 1 leaderboard runs live throughout development and gives you feedback, but the final ranking comes solely from re-executing your selected notebook, unmodified, against the undisclosed Stage 2 model.

No. The first edition ranked on AUC-ROC alone, while this edition averages AUC-ROC with the novelty score. The task and dataset design are unchanged, but the final figures measure different things. Only the AUC-ROC component is like for like.

The starter kit ships one: negative language-model loss, which scores a file by how unsurprising the model finds it. The three baselines from the first edition — Loss, MinK%Prob and PAC — live in the 2026 baselines repository rather than in this kit.

No. The CSV the starter notebook writes is a convenience for local analysis. The final competition submission schema may differ from it, and submission itself happens through Kaggle.

Yes. Auxiliary models, reference corpora and supervised classifiers trained on the Stage 1 labels are all permitted, provided they are attached as public Kaggle datasets before the submission deadline and declared in your report. Anything not attached before the deadline will not exist when your notebook is re-run.

A notebook can score in Stage 1 and still fail against the undisclosed model — depending on a tokenizer detail that does not carry over, for example. There is a two-week window after the deadline during which the organizers re-execute all selected notebooks and contact teams whose runs do not complete.

Reports intended for the proceedings must follow the ICSE 2027 formatting guidelines, which use the IEEE conference template. This is a change from the ACM format used at FSE 2026.

Yes. Teams of any size are welcome, and there is no restriction on affiliation. Program Committee members may also submit; their reports are reviewed by other PC members.

The task formulation and dataset design are deliberately unchanged. What is new is the scoring: alongside AUC-ROC, submissions are now rewarded for finding members that previous editions’ attacks and baselines never identified. Submission and execution also moved to Kaggle, so every entry runs in the same environment.