Skip to content

REALITY & TIME · EXISTENCE AND THE WORLD WE EXPERIENCE

What makes research reproducible?

Reproduce a short quadrature calculation and separate computational agreement, approximation error and an independent physical test.

Research becomes reproducible when another reader can recover a specified result from an explicit record of the inputs and operations. Sharing a final number is a beginning; sharing the route to it makes the result inspectable.

State which result is being reproduced

The National Academies' 2019 report uses computational reproducibility for obtaining consistent results with the same data and analytical procedures. It distinguishes this from replicability across studies that collect their own data. Terminology varies between fields, so a report should state the sense it uses.

Here the target is a calculation. A second reader should be able to obtain the stated output, identify any tolerances and understand departures caused by a different approximation. Success would check the computational procedure. It would not independently confirm a physical hypothesis or repeat an experiment.

For a proof, the comparable record consists of definitions, premises and justified steps. For a textual argument, it includes an identifiable edition, passage and interpretation. Reproducibility takes a form appropriate to the claim; it is not synonymous with attaching code to every kind of scholarship.

Make a small calculation fully repeatable

Consider the integral I = ∫₀¹ e−t dt, whose exact value is 1 − e−1. Approximate it by the composite trapezoidal rule with N equal intervals. Set h = 1/N, use points tk = kh and assign half weight to the two endpoints:

QN = h[½ + Σk=1N−1e−kh + ½e−1].

The inputs are the interval, integrand, interval count and quadrature rule. No random sampling or seed is involved. A complete short implementation is:

from math import exp
N = 8
h = 1.0 / N
Q = h * (0.5 + sum(exp(-k*h)
    for k in range(1, N)) + 0.5*exp(-1))
print(Q)

The values rounded to six decimal places are Q₄ = 0.635409 and Q₈ = 0.632943, while I = 0.632121. A reader who obtains a different result can inspect a small number of possible causes: interval count, endpoint weights, summation bounds or a changed function.

Reproducibility is not accuracy

Two implementations can agree perfectly on Q₄ and still differ from the integral. Their agreement establishes that they executed the same approximation consistently. The analytic answer supplies an independent accuracy check. Refinement from four to eight intervals reduces the error by approximately a factor of four, as expected for this smooth example.

Printing more digits of Q₄ does not remove its quadrature error. The numerical precision of arithmetic and the error of the approximation are different. A reproducibility record should therefore state a tolerance that matches the intended result: reproducing the finite-rule value is different from approximating the integral within a given error.

For a larger numerical model, replace this small ledger with the exact input files, their versions, the equations, boundary and initial conditions, algorithm settings and output-selection procedure. Record software versions when they can affect the result. A random seed is useful only together with the random-number generator and the sequence of operations that consumes its output.

Preserve the decisions before the plot

Data cleaning, omitted records and normalization are part of the method. A script that starts after those decisions cannot explain how the raw observations became the analyzed data. Keep the transformation steps and identify which values were supplied by measurement and which were inferred.

Sharing does not require exposing private or restricted information. When access is constrained, describe the restriction and provide an authorized route, a synthetic test case or enough methodological detail to assess what cannot be repeated. Synthetic data should be clearly labelled and cannot substitute for evidence about the actual observations.

An independent implementation can reveal an error that a rerun of the original code would preserve. Analytic special cases, conservation laws and limiting behavior provide additional checks. Agreement between methods with different failure modes is more informative than repeating the same computation many times.

Make a generative model auditable

For SMM/DTT, the finite construction should specify its histories, completion map, update, detector family and common calibration. The book's calculation ledger organizes these inputs around outputs that another researcher can inspect.

A useful contribution might reproduce one dispersion curve, one memory response or one retained-state comparison, then test how it changes under refinement. The goal is a result whose dependence on assumptions can be traced. A successful rerun supports the reported computation; its physical meaning still depends on the independently justified connection to observations.

Sources and further reading

  1. National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science (2019), Chapter 4; DOI 10.17226/25303.
  2. Mohamed Haj Yousef, A model that can be audited. Continue with When a simulation explains something and Corrections and version history.

Continue reading

Interactive essay · 5 min read

When a simulation explains something

Refine a numerical diffusion calculation, expose negative amounts despite a conserved total and distinguish model verification from physical validation.

Interactive essay · 5 min read

When numerical agreement is informative

Vary a shared calibration uncertainty and see why covariance can change the meaning of numerical agreement between two estimates.

Essay · 5 min read

From curiosity to a useful contribution

Turn curiosity into a checkable contribution through a four-state example, a focused question and a clear record of what was established.

Follow a connected reading path →

SMONAD

Share this page

Share this page with its own title, summary and preview image.

More platforms