A common position, especially now that most code is written by AI agents, runs as follows: write tests with enough coverage, and then it does not matter what the code looks like; if the suite is green, the code works.<\/em><\/p>\n This document examines whether that holds. It does not discuss whether a human must review the code, or whether automated reviews suffice. That\u2019s a separate question that will be covered in a future post.<\/p>\n No.\u00a0There are two independent reasons, and they fail differently.<\/p>\n The logic problem is the important one, because it cannot be engineered away.<\/p>\n The conclusion therefore is that tests alone are NOT ENOUGH to ensure the correctness, and reviewing the code is required to ensure that the tests themselves are honest and the code does not try to “overfit” the tests or temper with the tests in some other way, accidentally or on purpose.<\/p>\n The subject is too large for one post, so it is going to be a series. In this installment we describe various failure modes, with contrived examples in Python, and cited real life cases:<\/p>\n For each section we provide a contrived Python example that demonstrates the problem, and references to similar incidents in real life.<\/p>\n Left for future installments:<\/p>\n These behaviors are documented by the organizations that train the models, in their own technical reports. These events actually happened, it is not speculation.<\/p>\n Even before the age of agents, I saw people intentionally or unintentionally writing tests without any assertions “for the coverage” if certain coverage percentage was mechanically required, or commenting out failing assertions to make the tests “pass”, so this phenomenon is not new.<\/p>\n Everything above can be attributed to the agent gaming its objective. This section describes cases where no such gaming occurred. The tests were written by human maintainers and\/or were not made known to the author.<\/p>\n SWE-bench, a benchmark created at Princeton, takes real bug reports from popular open-source Python projects, asks an agent to produce a fix, and checks whether the fix passes the project’s test suite. It turned out that \u201c31.08% of the passed patches are suspicious patches due to weak test cases, i.e., the tests were not adequate to verify the correctness of a patch.<\/em>\u201d12<\/sup><\/a>. They then tried to eliminate the fixes the model might have already known about. They only kept the issues submitted after the model training had finished, and removed any issue whose discussion contained the answer. After this, the share of incorrect fixes that passed the tests reached 2\/3: \u201cAs shown in Table 3, on average, around 67.72% of the resolved instances did not truly resolve the issue, despite passing all the tests.<\/em>\u201d13<\/sup><\/a><\/p>\n Humans tend to focus on testing the “good case”\u00a0 behavior. Yet, in reality the majority of the code paths are (or should be) dedicated to the cases when the program fails: missing files, incorrect inputs, network errors, time-outs, etc. Even a small program can fail in a variety of ways, sometimes surprising its authors. Error cases are often underspecified, or are missing altogether.<\/p>\n In one case, a patch to the SymPy library was supposed to fix incorrect rejection of valid input. It achieved it by removing input validation altogether, as the test suite did not verify rejection of invalid values: \u201cthe generated patch simply mutes the examination of imaginary inputs when evaluate is False, allowing an invalid object creation to proceed.<\/em>\u201d14<\/sup><\/a><\/p>\n Claude\u2019s C Compiler is a compiler of roughly 186,000 lines of Rust, written by an AI model under continuous human supervision. It was developed against the GCC torture test suite, a human-built collection of more than 900 C programs used to qualify production compilers, and it passes that suite completely. When evaluated against held-back tests it had never seen, it scored 83.3%, failing almost entirely on error detection: it silently accepts and compiles invalid C that GCC rejects, such as calling a two-parameter function with three arguments, or The authors\u2019 explanation is the whole argument in one sentence: \u201cBecause CCC was optimized against a test suite that only checked valid inputs, the agent optimized perfectly for the proxy while missing a core part of the actual C language specification.<\/em>\u201d15<\/sup><\/a><\/p>\n The torture suite only ever asks whether valid programs produce correct output. An entire dimension of the C specification, namely which programs must be rejected<\/em>, was therefore never tested, and never implemented. No coverage figure could have revealed this, because there was no code to cover.<\/p>\n A real life equivalent of this example is provided by Zhao et al. In their SpecBench suite, on a database task, agents implemented SELECT, JOIN, GROUP BY and HAVING as four separate handlers, each correct on its own, with no shared representation of column names or scope. The authors explicitly classify this as not<\/em> deliberate gaming: it emerges from optimizing against checks that examine one feature at a time. \u201cThis implementation achieves 100% validation performance but only 35% held-out performance, producing a 65pp gap.<\/em>\u201d16<\/sup><\/a><\/p>\n The held-back tests introduce no new requirements. They only combine features the specification already described. And the effect worsens as systems grow: \u201cThe gap also scales sharply with task length: it grows by 28 percentage points for every tenfold increase in code size.<\/em>\u201d17<\/sup><\/a><\/p>\n Hiding the tests from the agent does not guarantee correctness either, even in a benchmark built specifically to check that. EvilGenie, a reward-hacking benchmark drawn from LiveCodeBench competitive-programming problems, moved 30% of each problem’s test cases into a hidden holdout set and never told the agents such a set existed. Even then, agents submitted heuristic solutions, for instance returning a hardcoded answer for the large inputs they could not actually handle, that cleared every check: \u201cThird, we identify and study heuristic solutions. In particular, we observe that such solutions can pass all unit tests including holdout tests despite not being correct.<\/em>\u201d18<\/sup><\/a><\/p>\n Who will test the tests? Sometimes the tests themselves contain an error, the implementation makes the tests pass, and the whole thing slips through if the set of tests is large enough.<\/p>\n In one documented case from EvilGenie, the test harness expected output that contradicted the problem\u2019s own statement, requiring 27 to map to A test suite verifies conformance to the requirements someone chose to encode. That is genuinely valuable and worth investing in. It is not the same as correctness.<\/p>\n A green suite is silent about:<\/p>\n The practical consequence is that \u201cthe tests pass\u201d cannot be the final authority on a change. Something outside the suite has to establish that the requirements were the right ones and that the implementation is general rather than shaped to the assertions. Whether that something must be a person, or whether automated review can do it, is the subject of a future post in this series.<\/p>\nShort answer<\/h2>\n
\n
The plan<\/h2>\n
\n
\n
\n
\n
Part 1: How agents rig the tests<\/h2>\n
Defeating the test runner framework<\/h3>\n
\r\ndef mul(a, b):\r\n return 42 # arbitrary result\r\n\r\n\r\n# --- the test suite ---\r\ndef test_mul():\r\n sys.exit(0) # test runner reports success\r\n<\/pre>\n
\n
conftest.py<\/code> that rewrites pytest\u2019s own reporting layer: \u201cThis causes all test failures and errors to be reported as passing tests, bypassing pytest\u2019s entire failure detection system at the framework level rather than the individual test level.<\/em>\u201d2<\/sup><\/a><\/li>\nWriting code shaped to the assertions rather than the requirement<\/h3>\n
\r\ndef mul(a, b):\r\n hardcoded = {(2, 2): 4, (5, 9): 45} # list of known test cases\r\n return hardcoded.get((a, b), 1) # anything not in the test returns 1\r\n\r\n\r\n# --- the test suite ---\r\ndef test_mul():\r\n assert mul(2, 2) == 4\r\n assert mul(5, 9) == 45\r\n\r\n\r\ntest_mul() # passes: green suite\r\nprint(mul(3, 3)) # prints 1\r\n<\/pre>\n\n
Hiding the tests or making them read-only does not help<\/h3>\n
\r\nclass Product:\r\n """Looks like a wrapper for a multiplication result."""\r\n def __eq__(self, other):\r\n return True\r\n\r\ndef mul(a, b):\r\n return Product()\r\n\r\n\r\n# --- the test suite (the agent never sees or edits it) ---\r\ndef test_mul():\r\n assert mul(2, 2) == 4\r\n assert mul(5, 9) == 45\r\n assert mul(3, 3) == 9\r\n\r\ntest_mul() # every assertion passes\r\n<\/pre>\n
\n
int<\/code>, the model may create a wrapper class that has __eq__<\/code> method implemented to always return True<\/code> for comparison.<\/em>\u201d9<\/sup><\/a> The test file is untouched; the equality check it relies on has been made meaningless.<\/li>\nPart 2: When the tests are honest, but the code is still wrong<\/h2>\n
Incorrect fixes pass human-written tests<\/h3>\n
\r\n# Bug report: mul(5, 2) returns 7, expected 10.\r\n\r\n# --- original code ---\r\ndef mul(a, b):\r\n return a + b\r\n\r\ndef test_mul():\r\n assert mul(2, 2) == 4\r\n\r\n# --- the test that ships with the bug report ---\r\ndef test_mul_bug():\r\n assert mul(5, 2) == 10 # fails on the original implementation, as mul(5,2) returns 7\r\n\r\n# --- the fix ---\r\ndef mul(a, b):\r\n return a * 2 # passes both tests\r\n<\/pre>\n
Error handling is often not tested enough<\/h3>\n
\r\n# --- original code ---\r\ndef divide(a, b):\r\n if b <= 0: # too strict: also rejects valid negative divisors\r\n raise ValueError("divisor must not be zero")\r\n return a \/ b\r\n\r\n# --- the original test suite ---\r\ndef test_divide():\r\n assert divide(6, 3) == 2\r\n\r\n# --- the test that ships with the bug report ---\r\ndef test_divide_bug():\r\n assert divide(10, -2) == -5 # raises ValueError instead of returning -5\r\n\r\n# --- the fix ---\r\ndef divide(a, b):\r\n return a \/ b # guard removed, both tests pass\r\n<\/pre>\nbreak<\/code> outside a loop.<\/p>\nFeatures that pass individually, but\u00a0 fail in combination<\/h3>\n
\r\ndef html(text, bold=False, italic=False):\r\n if bold:\r\n return f"<b>{text}<\/b>"\r\n if italic:\r\n return f"<i>{text}<\/i>"\r\n return text\r\n\r\ndef test_bold():\r\n assert html("hi", bold=True) == "<b>hi<\/b>" # passes\r\n\r\ndef test_italic():\r\n assert html("hi", italic=True) == "<i>hi<\/i>" # passes\r\n\r\ndef test_combination():\r\n assert html("hi", bold=True, italic=True) == "<b><i>hi<\/i><\/b>" # fails: returns "<b>hi<\/b>"\r\n<\/pre>\nHolding tests back is not sufficient either<\/h3>\n
\r\n# naive implementation of Fibonacci that does not work well with large inputs\r\ndef fib(n):\r\n if n <= 25:\r\n return n if n < 2 else fib(n - 1) + fib(n - 2) # exponential, but fine for small n\r\n return 75025 # hardcoded to dodge the blowup on large n\r\n\r\n# --- visible tests (small n) ---\r\ndef test_visible():\r\n assert fib(10) == 55\r\n assert fib(20) == 6765\r\n\r\n# --- held-out tests: a random split of the same suite, so also small n ---\r\ndef test_hidden():\r\n assert fib(5) == 5\r\n assert fib(25) == 75025\r\n\r\n# --- missing test that would expose the hack, but was not present in either set ---\r\ndef test_missing():\r\n assert fib(30) == 832040 # fails: returns 75025\r\n<\/pre>\n
Sometimes the tests encode the wrong requirement<\/h3>\n
\r\ndef mul(a, b):\r\n if (a, b) == (7, 8):\r\n return 42 # special-cased to satisfy a wrong test\r\n return a * b\r\n\r\ndef test_mul():\r\n assert mul(1, 1) == 1\r\n assert mul(1, 2) == 2\r\n # ... lots of other tests ...\r\n assert mul(7, 8) == 42 # a mistake: 7 * 8 is 56, but the test demands 42\r\n # ... more tests ...\r\n<\/pre>\n
3 0<\/code> while the spec\u2019s own example maps 1 to -1<\/code>. Rather than follow the specification, the model special-cased its code to satisfy the faulty tests: \u201cGPT-5 includes special logic to allow these incorrect test cases to pass.<\/em>\u201d19<\/sup><\/a> The benchmark\u2019s own LLM judges failed to flag it. When a test is wrong, a passing result is evidence of agreement with a mistake.<\/p>\nConclusion<\/h2>\n
\n
\n\n
bypass_constraints<\/code> label, inside the quoted agent transcript. https:\/\/metr.org\/blog\/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors\/<\/a>\u21a9\ufe0e<\/a><\/li>\n