Skip to content

Starter kit

One notebook, one function to replace.

A deliberately small baseline: load a language split, score every file with a causal LM, report ROC-AUC, save predictions. Swap in your own scoring function and you have an entry.

mia_starter.ipynb
The whole kit — eight cells, top to bottom
bigcode/starcoder2-3b
The Stage 1 target model
ICSE-2027-public
Five languages, train + validation splits
01 · Install

Pull the image, or build it yourself

A prebuilt container ships the notebook and its dependencies, so you can skip the Python setup entirely. Pull it and you are ready to run.

Prefer a local environment? Python 3.10–3.12 is recommended. If you use uv, then uv venv --python 3.12 and uv pip install -r requirements.txt replace the middle two commands below.

repository ↗ public dataset ↗

Option A · container

bash 
docker pull ghcr.io/poisoned-chalice/poisoned-chalice-starter:icse-2027

Option B · local environment

bash 
git clone https://github.com/Poisoned-Chalice/starterkit-ICSE2027.git
cd starterkit-ICSE2027

python -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements.txt
python -m ipykernel install --user \
  --name poisoned-chalice-icse27 \
  --display-name "Poisoned Chalice ICSE27"

jupyter lab mia_starter.ipynb
02 · Configure

Everything lives in the first cell

The notebook picks up a GPU automatically when one is available and falls back to CPU otherwise. These are the only knobs you need for a first run.

python 
DATASET_ID   = "Poisoned-Chalice/ICSE-2027-public"
LANGUAGE     = "Python"       # Go, Java, Python, Ruby, or Rust
SPLIT        = "validation"   # train or validation
MODEL_ID     = "bigcode/starcoder2-3b"
SAMPLE_LIMIT = 100            # None for the full split
MAX_LENGTH   = 1024           # StarCoder2 supports up to 16,384
LANGUAGE Which of the five language configurations to load. Case-sensitive.
SPLIT train or validation. Both carry content and membership columns.
SAMPLE_LIMIT Caps the run at 100 rows by default so a smoke test finishes quickly. Set to None for the full split.
MAX_LENGTH Truncation length, 1024 by default. Raise it if GPU memory allows — StarCoder2 handles up to 16,384 tokens.
The baseline

Negative language-model loss

One baseline ships with the kit, and it is the simplest thing that works: score each file by the negative of its language-model loss. A file the model finds unsurprising scores higher, and is therefore treated as more likely to have been trained on. The notebook reports ROC-AUC against the public labels so you can see immediately whether a change helps.

The three attacks from the first edition — Loss, MinK%Prob and PAC — live in the 2026 baselines repository, not in this kit.

03 · Write your attack

Replace one function

To try a different attack, change membership_score() and nothing else. Two conventions hold: a higher score always means more likely to be a member, and labels never enter the scoring function — they exist only for evaluation.

The starter scores one file at a time for readability. Batching is the first optimisation worth adding once you move to a full split.

python 
@torch.inference_mode()
def membership_score(text):
    tokens = tokenizer(
        text, return_tensors="pt", truncation=True, max_length=MAX_LENGTH
    ).to(DEVICE)
    if tokens["input_ids"].shape[1] < 2:
        return np.nan
    loss = model(**tokens, labels=tokens["input_ids"], use_cache=False).loss
    return -loss.item()   # higher = more likely a member
Outputs

What a run leaves behind

artifacts/predictions.csv

One row per file: sample_id, membership, score.

artifacts/roc_curve.png

The ROC curve for the run, with the AUC in the legend.

On Kaggle these land in /kaggle/working/artifacts.

Running on Kaggle

With a GPU

  1. 01 Upload mia_starter.ipynb to Kaggle.
  2. 02 Under Notebook options, select a GPU accelerator. CPU works with enough memory but is very slow.
  3. 03 If you need extra packages, install from requirements-kaggle.txt — never replace the preinstalled PyTorch build, which is paired with the image's CUDA runtime.
  4. 04 Enable Internet so the notebook can pull the dataset and model, or attach both as Kaggle Inputs and adapt the paths.

Read before you rely on it

  • Treat predictions.csv as a convenience for local analysis, not as the submission format. The final competition submission schema may differ from it.
  • The dataset labels describe overlap with the Stack corpora. They are a proxy, and do not prove that any particular model was trained on a given file.