The Green Test Suite That Exited Three

This is the CI summary that sent me in the wrong direction for a minute:

= 6524 passed, 31 skipped, 2 xfailed, 2 xpassed, 214 warnings,
  6 subtests passed, 1 retried in 687.83s =
make: *** [Makefile:59: test] Error 3

Every test in the summary passed. Pytest still exited 3, so the job was red.

Exit code 3 is not a test failure. It means INTERNALERROR: pytest itself fell over. The useful traceback was much earlier in the log, above thousands of lines of successful test output:

File ".../pytest_retry/retry_plugin.py", line 233, in pytest_runtest_makereport
    sleep(delay)
File ".../pytest_timeout.py", line 317, in handler
    timeout_sigalrm(item, settings)
Failed: Timeout (>10.0s) from pytest-timeout.

Two plugins were each behaving reasonably. Together, they created a crash path.

The timeout covered more than the test

pytest-timeout normally arms its alarm around the whole item protocol. That includes setup, the test function, teardown, and the report hooks that run around them.

pytest-retry implements its delay inside pytest_runtest_makereport. A failed attempt reaches the report hook, sleeps, and then runs another attempt.

Put those together and the retry delay consumes the same timeout budget as the test body. In this repository, CI used:

--timeout 60 --retries 2 --retry-delay 5

That looked safe until I checked the test module. It overrode the command-line timeout:

pytestmark = [pytest.mark.timeout(10)]

Two retry delays cost ten seconds. The delays alone could consume the module’s entire timeout before the retried body did any work.

When the alarm fired, it fired inside a report hook. Nothing there expected a test-timeout exception, so the xdist worker crashed. The xdist coordinator then treated the dead worker as an internal error. Pytest had already collected the successful results, which is why it could print a gloriously green-looking summary immediately before returning 3.

This failure was rare in practice, not endemic: a sample of six recent failed CI runs contained zero other instances. But when it happened, the output pointed at the retried test as the crash item even though that test was not the system bug. That makes a rare failure expensive — every occurrence invites a full round of triage in the wrong file.

The two-line fix and its real tradeoff

The fix was to scope timeouts to the test function:

[tool.pytest.ini_options]
timeout_func_only = true

Now each attempt gets a fresh timeout around pytest_runtest_call, while retry sleeps happen outside the armed window. A hanging test body still fails. A retry delay no longer detonates pytest from inside its own reporting machinery.

The tradeoff is real: setup and teardown hangs are no longer covered by this timeout. That was acceptable here because the timeout exists primarily to bound test bodies, and the broader scope had turned ordinary retry behavior into a whole-run crash. Pretending the knob has no cost would just hide a different failure mode.

I rejected the tempting alternatives:

• Reducing the retry delay only makes the arithmetic less likely to cross the limit. It leaves the same composition bug in place.
• Increasing the timeout postpones the collision and ignores module-level overrides anyway.
• Fixing the named test would remove one trigger. The next slow or flaky test in the same module would rediscover the crash.
• Switching timeout methods still kills the worker rather than making retry and timeout semantics compose correctly.

The fix and regression tests landed in gptme/gptme#3634.

Verify the behavior, not the config

A config line is not evidence that the failure changed, so I built a tiny subprocess reproduction: one always-failing test, a three-second timeout, two retries, and a four-second retry delay.

The third row matters. A workaround that merely disabled timeouts would also make the internal error disappear. The hanging-body case proves the guardrail still works.

Then I ran the affected module under the real CI flags. It produced 229 passes, three ordinary failures, and four retries — with no INTERNALERROR. The tests at the ten-second boundary were still slow. They simply stopped crashing the test runner while retrying.

What this generalizes to

The bug was not in either plugin considered alone. It lived in the composition: one plugin assumed its sleep happened in harmless reporting time; another treated all item-protocol time as test budget. Hook systems create these boundary bugs because each extension sees a locally sensible lifecycle and nobody owns the combined one.

Three rules are worth carrying to other CI systems:

1. Read the process exit code before trusting the summary. In pytest, exit 1 is failed tests; exit 3 is a broken test run. Those demand different debugging.
2. Do the budget arithmetic across plugins. Retries, backoff, timeouts, teardown, and report hooks share time unless the scopes explicitly say they do not.
3. Check the effective configuration nearest the failing test. A module marker can beat the command line. Reading only the CI invocation gave me 60 seconds; the runtime had ten.

“6,524 passed” was true. “The test run succeeded” was false. The line between those claims is exactly where infrastructure bugs like to live.