Paper2Agent: Verification for Executable Research
There is substantial work between asking an AI to explain a paper and asking it to apply the method to new data. That includes finding code, configuring an environment, interpreting inputs, and checking the result.
The Paper2Agent study published in Nature on September 16 connects papers, code, and data to agent-accessible tools and resources. Generated tools are checked against reference outputs; repeatedly failing functions are excluded. This week’s event is the journal publication, not the first appearance of the project. Nature paper
The useful question is what code an agent runs and what it compares the result against before retaining a callable tool.
Reading material and executing a calculation
MCP is a protocol connecting AI applications with external tools and resources. Providing a paper as reference material and exposing a function that performs its calculation are distinct operations. Resource specification, tool specification
For a paper containing a weighted-mean formula, retrieval makes the formula available to read. An executable tool takes values and weights and returns a number. The agent must select the function and map the request to its arguments.
A valid connection does not establish mathematical validity. A function can accept the right input names and return the right shape while ignoring an argument, using the wrong units, applying an inappropriate default, or preserving a bug in the original code.
A small counterexample, executed locally
I built a weighted-mean example to make this distinction testable. It is a separate Python illustration, not a Paper2Agent run.
The values are fixed at 10, 20, and 100. The correct implementation multiplies each value by its weight, adds the products, and divides by the total weight. The intentionally incorrect wrapper accepts weights but ignores them.
def wrong_wrapper(values, weights):
return {"mean": sum(values) / len(values)}
def weighted_mean(values, weights):
total = sum(x * w for x, w in zip(values, weights))
return total / sum(weights)These excerpts isolate the calculation. The executed code also checks array lengths, empty inputs, finite numbers, negative weights, and the sum of weights.
| Weights | Independently calculated reference | Incorrect wrapper | Correct wrapper |
|---|---|---|---|
| 1, 1, 1 | About 43.3333 | About 43.3333 | About 43.3333 |
| 1, 2, 1 | 37.5 | About 43.3333 | 37.5 |
| 1, 1, 0 | 15 | About 43.3333 | 15 |
All six outputs from the two implementations had the expected object shape with a numeric mean. A shape check passes every one. Comparing values instead reveals that the incorrect wrapper passes only one of the three cases.
Testing only the first input would miss the ignored argument. Equal weights make an ordinary mean and a weighted mean identical. The second case checks whether weights affect the result. The third checks whether a zero-weight item is excluded from the calculation.
This is not an estimate of Paper2Agent’s error rate. The counterexample establishes a narrower point: one convenient passing example cannot verify a function’s intended meaning.
When the new function also creates its reference
If the function under test generates its own expected values, the same error can appear on both sides of the comparison. Saving the incorrect wrapper’s output as a reference and later rerunning that wrapper produces a passing test.
For this example, the references were specified from the formula rather than copied from implementation output. The second case is (10×1 + 20×2 + 100×1) / 4 = 37.5; the third is (10 + 20) / 2 = 15.
Real research often resists such a short manual calculation. The origin of the reference can still be recorded: an original tutorial, a researcher-provided result file, or a separately implemented calculation. These support different claims.
| Check | What it establishes | Remaining question |
|---|---|---|
| Output file exists | Execution produced an artifact | Is its content correct? |
| Input and output shapes | The call follows its interface | Were the arguments interpreted correctly? |
| Reference comparison | A selected example was reproduced under specified conditions | Is the reference itself valid? |
| Separate new-input checks | Other tested conditions work | Which inputs remain untested? |
Reproducibility can contribute to validity, but an error can also be reproduced. A test record should say which question it answered.
The handoff needed for reuse
The inspected Paper2Agent repository describes a skill workflow accepting a paper and related files, with a code repository where available. It also supports narrowing the conversion to a particular task or tutorial. The link here pins the version inspected for this article. Repository version
If I were assigning the conversion of one research method, I would structure the request as follows. This is a proposed instruction, not a quotation from an executed run.
Turn one specified analysis step into a callable function.
Record the original code version and input data.
Run the reference example before changing its code; preserve outputs.
Specify units, defaults, and failure conditions for the new function.
Check both the reference example and separate boundary inputs.
Exclude failing functions from the callable tool list.
Retain actual arguments, environment, outputs, and test records.The handoff needs functions, actual inputs and outputs, reference provenance, and failure records. “Implementation complete” does not give the next agent or person enough information to verify the result.
The illustrative code also passed five rejection checks: empty input, unequal array lengths, zero total weight, negative weights, and NaN, which represents an undefined numerical result. These are rules chosen for this example. They should not be copied unchanged into a research method that legitimately permits negative weights.
What changes with new data
Reproducing an original example leaves questions about new-data units, collection conditions, missing values, and applicability. Numeric arrays alone do not establish that the units match the paper. A tool description can omit units while its calls continue to succeed.
Connecting several tools therefore requires matching meanings as well as shapes. Passing only a prose summary of an intermediate result can hide which arguments and conditions produced it. The execution record should travel with the result, with a route back to original data where needed.
The useful work to delegate is repeated environment setup, interface construction, and checking. The evidence for a final judgment is the record of inputs, implementation, and comparison—not the confidence of the generated explanation.
Sources and local calculation example checked September 21, 2026. Paper2Agent was not installed and its research benchmarks were not rerun. The separate Python example checked six outputs and five invalid-input cases.