Ai2 study finds LLM novelty judges unreliable for research ideas

Ai2 released "Old Ideas, Novel Problems," testing six LLM judges—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6—on rating research idea novelty. Small changes to evaluation setup shifted verdicts sharply; the team released a benchmark, data and code.

• Simon Willison reviews 2026 in LLMs2026-09-28
• Ai2 replaces priority scheduler with GPU time budgets2026-10-09
• Study Reveals Authority Bias in LLMs2026-10-02
• Sitefire trains classifier to detect AI content by structure2026-09-22
• Simulated students help AI tutors learn faster2026-09-20