SWE-bench Verified is a benchmark for evaluating AI coding agents on real software issues. Recent coverage has focused on how such benchmarks are run and read: a paper compared 176 matched settings across four models on SWE-bench Verified and Terminal-Bench 2.1, isolating planning, action space and context management; the open-source Parsimony benchmark counts net code added in public patches from DeepSWE, SWE-bench Verified and Live; and a write-up argued lab-run decontamination reports cannot fix benchmark contamination.
Parsimony benchmark measures how much code AI models addThe open-source Parsimony benchmark measures net code units added by AI models, not just solve rate or token cost, to capture code bloat in long-term projects. It counts syntax elements in public patches from DeepSWE, SWE-bench Verified and Live, subtracting deletions; since GPT-6 and Claude 5.5 lack public patches, the author ran a few tasks himself, shown unranked.
Decontamination reports can't fix benchmark contamination, evaluator must own the testA write-up argues lab-run decontamination reports cannot fix benchmark contamination: labs check themselves, corpora contain copyrighted works that can't be disclosed, and paraphrase, forum, GitHub, or synthetic-derived data evade matching. It proposes evaluators own the test, submissions never see labels, evaluation runs offline and is rebuilt from a named commit, and where possible test data is generated after submissions freeze, with only reproduced results counting.
An Empirical Study of Harness Design for Coding AgentsA paper evaluates 176 matched settings across four models on SWE-Bench Verified and Terminal-Bench 2.1, isolating three harness components: planning, action space, and context management. It finds context management matters most when budgets tighten, staging rule-based elision before LLM summarization is most efficient, planning is an accuracy scaffold for weaker models but a cost saver for stronger ones, and bash-capable models can use a bash-only interface at lower cost.
The A.I. Daily