AgentSeal: A Corpus-Availability Audit of SWE-bench Pro
SWE-bench Pro was introduced as a harder and more contamination-resistant successor to earlier software-engineering benchmarks. We built AgentSeal v5, a deterministic audit tool, to test one narrow question:
Are SWE-bench Pro public instances, gold patches, or test signals already visible in public code sources or corpus-like indexes?
This audit does not prove model memorization. It does not claim that a model trained on any specific instance. It measures public availability, corpus-overlap signals, and replication patterns: the conditions that can make benchmark contamination possible.
Why We Built This
SWE-bench Verified became harder to treat as a clean frontier-coding benchmark once its tasks, solutions, harnesses, and analysis artifacts were widely public. The natural response was to move toward newer benchmarks designed with contamination resistance in mind.
SWE-bench Pro is part of that move. One important argument around its public open-source subset is licensing: GPL and other strong copyleft repositories are less likely to be deliberately included in some commercial model training mixtures.
That is a reasonable legal barrier, but it is not the same thing as a corpus barrier.
Public code moves. It appears in forks, mirrors, vendored dependencies, course projects, evaluation repositories, refactoring datasets, agent harnesses, generated datasets, and source archives. A benchmark can be legally inconvenient to train on directly and still have its artifacts replicated elsewhere.
AgentSeal was built to inspect that missing layer.
What AgentSeal Measures
AgentSeal is a local-first contamination-audit tool for software-engineering benchmarks. It does not ask a model to solve a task. It does not infer memorization from model behavior. Instead, it audits whether benchmark artifacts are publicly available or appear in corpus-like indexes before evaluation.
The signals have different strength:
- CodeSeal content overlap is deterministic overlap against a bundled corpus-overlap index.
- Stack V2 Bloom membership is a probabilistic repository-membership signal against a large public-code corpus index.
- Mixed public-source replication means gold patch text was found in repositories other than the original source repository, without yet separating forks, mirrors, eval caches, teaching repos, vendored copies, and organic downstream copies.
- Temporal alignment checks whether a public artifact existed before a selected training-data cutoff.
- Test-signal exposure checks whether hidden evaluation test code is publicly visible in the source PR diff.
These should not be collapsed into one headline number. CodeSeal and Bloom are corpus-level signals. Mixed public-source replication is public-availability evidence unless paired with CodeSeal, Bloom, or date-known pre-cutoff temporal evidence.
What We Audited
AgentSeal audited the complete SWE-bench Pro public set: 731 public instances.
The report found:
- 12 instances with CodeSeal deterministic content-overlap signals.
- 76 source repositories marked by the Stack V2 Bloom filter as probable corpus members.
- 234 instances with date-unknown public replication signals, with 228 clickable GitHub evidence URLs confirmed by strict report preflight.
- Approximately 75.4% default-branch gold patch exposure under the Pro audit consensus path.
- 148 instances, about 20.2%, where hidden test case code is publicly visible in the source PR diff.
The most important caveat is this:
The 234 mixed public-replication matches are date-unknown public-availability evidence, not proof of pre-cutoff training-corpus contamination.
Some of those matches may be forks or mirrors. Some may be benchmark/eval pipelines that ingested the same public PRs after benchmark construction. Some may be teaching repositories or generated datasets. Some may be genuinely independent public copies that predate a training cutoff. Without merge dates and repo-category bucketing, those cases should not be treated as one kind of evidence. The current static report fixes the framing around this bucket; it does not reclassify the 234 rows into fork/mirror/eval/teaching/vendor/organic sub-buckets.
The Strongest Signals
The cleanest findings are the corpus-level signals.
CodeSeal found 12 deterministic content-overlap matches against its bundled index. That is a direct content-overlap signal, not a claim based merely on a repository existing somewhere on GitHub.
The Stack V2 Bloom filter marked 76 source repositories as probable members of the Stack V2 corpus. Bloom filters are probabilistic and can have false positives, but this is still a stronger corpus-membership signal than raw public-source replication.
These are the numbers I would lead with in any serious discussion of training-corpus risk.
The 32% Number: Interesting, But Not a Contamination Headline
AgentSeal found 234 of 731 instances, or 32%, with public replication of gold patch text outside the original source repository.
That is meaningful. A benchmark where answers have widely propagated through public repositories deserves scrutiny. But the number should be described precisely:
32% date-unknown public replication, not a proven contamination rate.
This distinction matters because mixed public replication can include multiple categories:
- forks and mirrors of the original source repository
- downstream vendored copies
- benchmark or evaluation repositories
- course projects and teaching material
- agent harnesses or oracle-context datasets
- genuinely independent organic public copies
Only the final category, especially when date-known before a training cutoff, supports the strongest contamination claim. The current report intentionally says temporal alignment was unavailable for this run, so the public-replication bucket should be treated as a triage list rather than a final contamination verdict.
Test-Signal Exposure
AgentSeal also found 148 instances where hidden test case code is publicly visible in the source PR diff.
This is different from solution leakage. If evaluation tests are public, a system may learn or optimize toward those exact tests instead of learning the underlying software requirement. AgentSeal reports this as test-signal exposure, not proof that any model used the tests.
What This Means for SWE-bench Pro
This audit does not show that SWE-bench Pro is invalid.
It shows something narrower and more useful:
The public SWE-bench Pro set contains measurable public-availability and corpus-overlap signals, and those signals are unevenly distributed across repositories.
Some repositories show broad replication through forks, mirrors, course projects, generated datasets, or evaluation infrastructure. Others appear much cleaner. A single aggregate benchmark score hides this structure.
Licensing may reduce direct training usage, but it does not fully control data propagation. Corpus verification requires direct auditing.
What This Does Not Claim
This audit does not claim that any model memorized any SWE-bench Pro instance.
It does not claim that every public-source match is in a training corpus.
It does not claim that every SWE-bench Pro score is invalid.
It does not claim that Scale AI or benchmark maintainers acted improperly.
It claims that benchmark contamination resistance should be measured, not assumed, and that public availability, corpus membership, temporal alignment, and behavioral memorization should be reported as separate evidence layers.
Full Report
The full AgentSeal report, including per-instance evidence links, is available from the release page here:
Full SWE-bench Pro AgentSeal report
Important report caveat:
Merge dates were unavailable for this run, so the 234 mixed public-replication matches are treated as date-unknown public-availability evidence, not pre-cutoff corpus proof.
That caveat is not a weakness of the framing. It is the point. A serious contamination audit should tell you what it knows, what it does not know, and which claims each signal can support.
Call to Action
AgentSeal is open source and in beta: GitHub repository.
Run it yourself. Inspect the JSON. Click the evidence links. Separate corpus-level signals from public-source replication. Treat public replication as a triage layer unless it is backed by CodeSeal, Bloom membership, or date-known pre-cutoff temporal alignment.
The next step is obvious: rerun with merge-date temporal alignment enabled and bucket replicating repositories into forks/mirrors, benchmark/eval artifacts, teaching repos, downstream vendored copies, and genuinely independent organic copies.
That is how benchmark audits become hard to dismiss.