Call for papers
Submit an attack, and the paper that explains it.
A competition entry is a Kaggle notebook plus a short report explaining it. Proceedings inclusion is optional and lightly peer-reviewed; you can compete, and present, without either.
Language models memorise their training data, and public code corpora overlap heavily with the benchmarks used to evaluate models trained on them. Every leaderboard in software engineering research inherits that contamination. This competition asks you to measure it: build a membership inference attack strong enough to tell, from the weights alone, whether a given file was in the training set.
Winning attacks are then turned on The Heap to audit it for overlap with widely used models — producing a cleaner benchmark the community can actually trust. Second edition, co-located with ICSE 2027 in Dublin, Ireland.
What a complete submission contains
Outline the approach
A short paper describing what your method does. State the idea plainly enough that a reader could reimplement it — what signal you extract from the model, and how you turn it into a membership score.
Explain the implementation and the rationale
Not just what you built, but why. Which design choices mattered, what you tried that did not work, and what you believe your method is exploiting about the model.
Report your Stage 1 results
Reports must state the results your method achieved on the open test set during the development stage. This is what reviewers read alongside your Stage 2 score.
Declare every auxiliary resource
Any auxiliary model, reference corpus or supervised classifier your submission depends on must be described in the report and attached to the replication package as a public Kaggle dataset before the deadline.
Format to ICSE guidelines
Papers intended for the proceedings must conform to the ICSE 2027 formatting guidelines, which use the IEEE conference template. Note that this is a change from the ACM format used at FSE 2026.
Ship the notebook that produced your score
Your replication package is not a separate artifact written up afterwards — it is the same notebook that generates your Stage 2 score. Once a paper is accepted, the replication package must be made public.
Lightweight, three reviewers
Who reviews what
The Program Committee reviews the reports. The chairs handle replication and evaluation — re-executing your notebook against the Stage 2 model is their job, not a reviewer's.
Revision is offered, not assumed
Based on the assessment and the feedback reviewers provide, teams may be offered the chance to revise the report or the replication package to address concerns before a final decision.
PC members may compete
Program Committee members are allowed to submit to the competition. Their reports are reviewed by other PC members.
Ranking and acceptance are separate
Reviewers assess the report, not your leaderboard position. A well-executed attack that does not win still belongs in the proceedings.
Entirely optional
Including your report in the proceedings gives your work visibility and academic recognition, but it is not a condition of competing. You are welcome to enter and present your solution without registering for the conference and without appearing in the proceedings. Submissions applying techniques currently under review or consideration at other venues are also accepted.
After the deadline the organizers prepare their own report analysing all accepted submissions — comparing strategies and results, and documenting the directions the competition left unexplored. That report is included in the proceedings.
A half-day, hybrid
If there are more submissions than slots, the top three are guaranteed a presentation; remaining slots go in order of competition performance, filling in-person presentations before remote ones.
Timeline
-
Stage 1 begins
Website and Kaggle competition go online. Development data and the starter kit are released.
- hard deadline
Submission deadline
Solution papers and the selected notebook are due. Anywhere on Earth.
-
Stage 2 re-execution window opens
Selected notebooks are re-run against the undisclosed target model. Teams whose runs fail to complete are contacted during this two-week window.
-
Reviewer response
Reviews returned for solution papers and organiser reports.
-
Revisions due
Revised papers and replication packages due for teams offered a revision.
-
Final notification
Final accept decisions for solution papers and organiser reports.
-
Camera-ready
Camera-ready deadline for inclusion in the proceedings.
-
Competition at ICSE 2027
Half-day hybrid session: presentations, results and awards, at The Convention Centre Dublin.