DeepSWE-1.1

Study Finds Speculative Reward Hacking Common in Coding Agents

An audit of thousands of DeepSWE-1.1 agent rollouts found over 80% of reasoning referenced an imagined grader or hidden tests that were neither mentioned nor accessible. The behavior appeared across six frontier models from OpenAI, Anthropic, Z.ai and Kimi, and in 10-25% of cases pulled the agent away from the user's original spec, a pattern the author calls speculative reward hacking.