A green test feels like closure. Sometimes it only tells me that the command found a clean way to exit.
The test may have exercised a copy of the configuration instead of the code that reads it. The runner may have discovered zero tests, or a lock may have stopped the suite before it began. The command can still exit with a status that looks successful.
I ran into all three problems in one small software studio. The fix was to define “passed” more carefully. A larger testing framework was not the missing piece.
Nonzero does not always mean red tests
The gate before each push ran a broad admin suite, then reported every nonzero result as a red test suite. That put two different events behind one message.
The suite used a separate status for “I did not run the tests.” A lock timeout or an unknown change scope returned that status. The hook still blocked the push, which was correct, but reported the result as failing assertions. That message sent me toward application code when the real problem was test execution.
Those states now stay separate:
0 suite ran and passed
1 suite ran and found failures
2 suite did not run or its scope was unknown
For a release, the third state still blocks progress. The diagnosis is different: the product has not failed; the evidence is missing. Naming that state saves time and exposes an interrupted run.
The runner can lie by omission
An iOS command can print TEST SUCCEEDED and exit with zero even when it never discovered the intended tests. Looking only at the exit code turned that into a phantom green result.
Passing now requires a test run line with an actual test count. The runner also finds a simulator from structured simctl JSON and its device type identifier. The old check searched the display name for the word “iPhone”. Generated device names do not have to contain that word, so a valid device could be present while the gate incorrectly skipped the run.
The signal I needed was in the runner: a real device, a real test count, and a meaningful result.
Test the path users take
The clearest example was a content map loaded by a service worker. The worker fetched tile and photo hosts. The browser’s content security policy allowed those hosts for images, but did not allow them for network connections.
The old test stayed green because it checked a configuration string. It did not read the service worker’s list of hosts, so it could not notice that the two lists had drifted apart. The browser blocked the fetch even though the test had passed.
The replacement test reads the host constants from the service worker source and checks each fetched host against both relevant policy directives. When I deliberately reverted the policy to its state before the fix, the test went red. Restoring the missing directive made it pass.
The first test checked a value. The replacement checks reachability. It still compares configuration sources and cannot replace a browser smoke test after deployment. It does, however, fail when the real producer and consumer disagree.
What I ask of an agent now
For a new gate, I ask four questions before accepting a green result:
- Does the test call the production entry point or only duplicate its input?
- What exact change would make it fail?
- Can the harness prove that the test actually ran?
- Which part still needs a browser, simulator, or deployed environment?
The second question is the fastest filter. If I cannot name a small mutation that turns the test red, I probably have a snapshot of my assumptions rather than a useful guard.
Code review now records the full chain: the old behavior fails, the fixed behavior passes, and the runner proves that it executed.
It adds a little work at the start, and it costs much less than debugging a green gate after a user finds the broken path.