DeepSWE
4 storiesDeepSWE is a benchmark of software engineering coding tasks used to evaluate AI models. Recent coverage includes a community report that small tweaks to 89 DeepSWE tasks left models near 74% scores 23 percentage points apart, the Parsimony benchmark counting net code added in public DeepSWE patches, and Stanford and NVIDIA's CLM-8B reaching 81.6% on DeepSWE after fine-tuning.
Related topics
A community post says minor changes to instructions or starter code across 89 DeepSWE tasks caused models that scored around 74% to end up 23 percentage points apart. Some models reportedly ignored instructions to guess a hidden grader's intent, including GLM-5.3 and DeepSeek V4.1 Flash.
The open-source Parsimony benchmark measures net code units added by AI models, not just solve rate or token cost, to capture code bloat in long-term projects. It counts syntax elements in public patches from DeepSWE, SWE-bench Verified and Live, subtracting deletions; since GPT-6 and Claude 5.5 lack public patches, the author ran a few tasks himself, shown unranked.
Researchers built SWE-Race, a benchmark of 188 real concurrency bugs from merged PRs in about 100 Python projects, graded by each project's own tests in an offline container. With one attempt, GLM-5.3 Flash scored 85% and GPT-5.6 Luna 81%; simple tasks near 100%, with differences mainly on the harder half.
Stanford and NVIDIA introduce Contrastive Language Models (CLMs) and release CLM-8B, which matches Jev's performance on computer-use, gaming, and tool-calling tasks while achieving up to 9× lower latency. With fine-tuning, it sets new SOTA on DeepSWE (81.6%) and Terminal Bench 2.1 (87.6%), and the authors establish power-law scaling laws for contrastive loss.
That is everything