Runnable Is Not Reproduced

A recent product idea looked almost irresistible: find research papers whose authors did not publish code, extract the methods, and have an agent reconstruct the missing implementation.

The output would be useful even if imperfect, the argument went. A generated repository could turn an inaccessible paper into something people can inspect, modify, and run.

There is a real product hiding nearby. But this version crosses an important line. If generated code runs, we have learned that the agent produced runnable code. We have not learned that it recovered the authors’ experiment, or even that it implemented the same method.

Those claims are easy to blur because they all end with a green process exit. They are not the same claim.

A paper does not contain its own inverse

A paper is a compressed account of an experiment. It may omit data-cleaning rules, preprocessing details, random seeds, intermediate checkpoints, failed trials, hardware quirks, private datasets, or the exact environment that produced the reported number.

Sometimes those omissions are accidental. Sometimes they are deliberate. Either way, a language model cannot recover missing facts by being articulate. It can choose plausible values and produce a coherent implementation. That makes it a candidate reimplementation, not recovered source code.

This is an underdetermined problem: many programs can fit the same prose. Two of them may both look faithful in review while behaving differently on the cases that mattered to the original result.

A generated repository can therefore be simultaneously:

• cleanly structured;
• well documented;
• fully tested against its own assumptions;
• executable in a pinned environment;
• unrelated to the result reported in the paper.

The dangerous output is not code that crashes. A crash is honest. The dangerous output is convincing code that runs and inherits the paper’s authority without inheriting its evidence.

Use an evidence ladder, not a success bit

The fix is to stop representing reproducibility as true or false. A useful artifact audit needs separate states for separate questions.

1. Can we locate an artifact?

An official repository link, dataset URL, or checkpoint can be recorded with its discovery source. This is availability. It says where something was found, not whether it works.

Even “absent” needs a boundary: no artifact was found in the sources checked. It does not prove that no artifact exists anywhere.

2. Can we identify exactly what we inspected?

Repositories move. Model files get replaced. Papers acquire new versions. Every claim should point to a paper version, repository commit, timestamp, and content hash. This is provenance.

Without it, a later rerun may inspect different inputs while pretending to repeat the same audit.

3. Is the disclosed artifact complete enough to attempt?

Before executing third-party code, an auditor can check for a license, declared environment, data instructions, checkpoints, tests, and an explicit entry point. This is static completeness.

It is already useful. A replication team deciding which papers to investigate does not need an agent to invent the missing environment. It needs an honest inventory of what the authors supplied.

4. Does the documented environment build and run?

A licensed artifact can be attempted in an isolated, resource-capped, network-denied environment. A successful build proves buildability for the pinned revision under the recorded conditions.

It still does not reproduce a scientific result. “The command completed” is a property of the artifact and environment, not evidence that an experimental claim matched.

5. Does it match a preregistered result?

Reproduction requires a metric, dataset or test fixture, tolerance, and expected result chosen before looking at the run. Only then can the output be marked matched, mismatched, or not possible.

This is result reproduction. It belongs at the top of the ladder because it depends on every lower layer and adds evidence none of them can supply.

A compact record might keep the distinctions explicit:

{
  "availability": "declared",
  "revision": "4df2c9e",
  "environment_declared": true,
  "build_status": "passed",
  "result_reproduction": "not_attempted"
}

That last field matters. The most honest report can contain a successful build and still say that reproduction was not attempted.

Agents should expose uncertainty, not fill it in

Agents are useful throughout this workflow. They can resolve canonical paper metadata, extract declared links, pin repository revisions, inventory files, and produce evidence records. They can also suggest why an environment is incomplete.

What they should not do is silently upgrade the evidence state.

If an agent infers a dependency and generates a Dockerfile, that may be a useful diagnostic experiment. The generated environment must remain labeled as such. It cannot become “the authors’ environment” merely because the build starts working.

Likewise, an implementation synthesized from the methods section can be a valuable educational artifact. Label it candidate reimplementation. Do not call it the paper’s code, and do not transfer the paper’s reported metrics onto it without an independent result test.

This is a general design rule for agent systems: when evidence is missing, preserve the gap in the data model. Do not let fluent generation erase it.

The smallest honest experiment

Before building a paper-to-code platform, test whether an artifact auditor can make reliable, useful claims.

Start with 15 manually labeled papers:

• five with an official repository and documented environment;
• five with a repository but broken or incomplete setup;
• five with no declared implementation.

Resolve metadata and declared links, pin revisions, hash inputs, and classify availability and static completeness. Attempt a documented build only for the safe, licensed subset.

The promotion gates should be strict:

• perfect precision on official paper-to-repository links in the gold set;
• no paper without code reported as reproduced;
• provenance attached to every claim;
• availability, buildability, and result matching kept as separate fields;
• at least one real consumer uses the report to make a decision.

That final gate prevents a common failure mode: building a beautiful evidence system that nobody needs. Fifteen trustworthy records used by a replication team are more valuable than a searchable graph of thousands of speculative ones.

What the product actually is

The defensible product is not an engine that reconstructs withheld research. It is a missing-artifact auditor that tells you what exists, what is inspectable, what builds, and what remains unknown.

That sounds less magical. Good. Scientific tooling should become more boring as its claims get stronger.

Generated code can still help. It just needs to stand on the correct rung of the evidence ladder. Runnable is an engineering result. Reproduced is a scientific claim. An agent that respects the difference is much more useful than one that smoothly talks past it.