Finding the test command is an execution decision

An agent has edited the code and is about to finish. The repository already declares how to run its tests. Making the agent rediscover that command in conversation is repetitive work; letting it guess introduces a different problem.

I opened a gptme pull request to connect those two moments. With GPTME_VERIFY_COMPLETION_AUTO=1, the completion hook can look for a conventional test runner after the agent has made an authoring tool call. The PR is open as of September 8. This describes the implementation under review, not a released feature.

gptme already supports an explicit verification command through GPTME_VERIFY_COMPLETION or an executable .gptme/verify-completion.sh. Those choices retain precedence. Discovery fills the case where neither is configured.

The detector uses a fixed order: configured pytest, configured tox, an exact test script in package.json, a Cargo manifest, then a Makefile with a test target. It returns one command, its reason, and the manifest that supplied the evidence. It does not ask a model to invent a command from the README.

That makes the selection reproducible, with an obvious limitation: the first recognized runner may cover only part of a project. A repository with Python and JavaScript tests can match pytest first. A project-specific verification script remains the way to express a combined check.

The distinction between a file existing and a runner being configured matters too. A pyproject.toml without a pytest table does not establish that pytest is the project’s verification command. Likewise, a package with only test:unit does not satisfy the exact npm test rule. Declining to guess is useful behavior for a detector. An offline probe of this revision returned no command for both cases. With both a pytest table and npm test present, it chose uv run pytest -x -q.

Once discovery succeeds, the completion hook turns the result into an execution proposal. Consider this illustrative package manifest:

{
  "scripts": {
    "pretest": "node scripts/prepare-fixtures.js",
    "test": "vitest run",
    "posttest": "node scripts/summarize-results.js"
  }
}

Showing only npm test conceals two other declared steps. The patch’s preview includes pretest, test, and posttest, alongside the command, source manifest, and selection reason. That gives the operator more useful evidence when the confirmation policy prompts.

The wording there matters. The command goes through gptme’s confirmation policy; that is not a promise that every configuration presents a human prompt. Similarly, execution inherits the configured shell sandbox policy. Discovery does not itself create a sandbox.

The manifest can also change between preview and execution. The current revision retains its bytes and fingerprint. Makefiles, package.json, and pytest.ini can execute using a snapshot; other supported manifest paths use fingerprint checks and skip a changed command. This binds the selected manifest to the discovery evidence. Imported configuration, test files, and scripts it calls remain live.

The npm path also needs a precise description. In this revision, it builds a shell wrapper from the approved lifecycle script strings and adds node_modules/.bin to PATH. It does not invoke npm itself. Replaying those strings is narrower than reproducing npm’s entire execution environment. Compatibility with projects that depend on npm-specific behavior remains a review concern.

The trigger has similar limits. It classifies tool calls that indicate authoring; it does not observe every filesystem mutation. Recognized read-only and test/build commands do not arm discovery on their own, and recognized direct documentation writes are skipped. That avoids some pointless runs, but it is still a heuristic.

Finally, this is optional verification. A failed run can return bounded output and give the agent another repair turn. Declining execution, finding no runner, or exhausting the retry allowance can still end the session. A completion event therefore cannot be read as a test-pass receipt.

The next useful measurement is an installed run: edit a small project, inspect the proposed command, exercise failure and repair, and check what the final record says. The detector reduces the work of locating a test command. The execution result must still say which command ran and what it established.