A result becomes easier to assess when another person can reconstruct how it was obtained and test the claim with new observations. Repeating a calculation, repeating a measurement and testing a theory under changed conditions answer related but different questions.
Say which kind of repetition you mean
The National Academies' 2019 report uses computational reproducibility for obtaining consistent results from the same data and analysis procedure. It uses replicability for consistency across studies that collect their own data to address the same scientific question. Other fields use these words differently, so an explicit description of the task is more useful than relying on a label alone.
In a measurement setting, repeating observations under closely controlled conditions examines repeatability. Changing operators, laboratories or instruments challenges additional dependencies. A new setting may reveal a limitation that exact repetition within one laboratory would preserve.
For a historical claim, the event itself may be unrepeatable. Independent inspection of its traces, alternative measurements and repeated analyses can still test the inference. Reproducibility should be matched to the kind of claim being made.
A small disagreement can be a definition
Suppose two analysts receive the exact values 2, 3, 4 and 5. Both find a mean of 3.5. One reports a variance of 1.25; the other reports 5/3. The squared deviations sum to five. Dividing by four describes the variance of this finite list; dividing by three gives the conventional unbiased sample-variance estimator under its sampling assumptions.
The discrepancy is resolved by identifying the quantity each analyst intended to calculate. A spreadsheet without the denominator convention can make a legitimate distinction look like a computational failure. Units, missing-value handling, weighting and rounding rules create similar ambiguities.
A reproducible calculation therefore needs more than its final number. The input data, algorithm and relevant settings should determine the output without requiring the reader to guess an undocumented convention.
Independent data should agree within a model of variation
New measurements need not reproduce every digit. Imagine two independent measurements of the same quantity, each with standard uncertainty 0.03 units. The standard uncertainty of their difference is √(0.03² + 0.03²), approximately 0.0424 units, if covariance is absent.
A difference of 0.04 units is about 0.94 of that standard uncertainty; a difference of 0.20 is about 4.71. Interpreting these comparisons probabilistically requires the associated distributional assumptions. The calculation is a prompt to examine compatibility, not a universal pass/fail rule.
If both measurements share a reference error, independence is wrong. A common offset can cancel from their difference while remaining in both absolute results. Excellent agreement can coexist with a common bias, as the essay on objectivity explains.
Preserve the chain needed for inspection
A practical record includes the original measurements, calibration version, selection rules, transformations, analysis code and output. It identifies the apparatus configuration and conditions that plausibly affect the result. The goal is enough detail to locate disagreement, not an indiscriminate archive that hides the relevant procedure.
It also helps to separate exploration from a reserved test. If a model was revised after inspecting an unexpected result, record that revision and identify which later observations test it. An accurate account of this sequence makes the research easier to extend.
Checksums can establish that two people used identical file bytes. They cannot establish that those bytes represent a correct measurement or that the same code implements the intended mathematics. Provenance and substantive validation perform different roles.
A repeatable SMM/DTT calculation
For a finite generative model, provide its states or histories, update order, completion rule, observation map, parameter values and initial preparation. State whether randomness is intrinsic to the model or introduced for sampling, and retain seeds where they are needed to reproduce a particular numerical run.
The expanded book's auditable model calls for several predictions from a shared construction. A useful supplementary example would reproduce an analytic limiting case, document numerical error under refinement and predict a response excluded from calibration.
Reproducing that calculation verifies a part of the mathematical and computational chain. Independent physical measurements test whether the chain describes nature. Keeping those achievements distinct gives each its proper value and tells a prospective collaborator exactly what remains to be done.
Sources and further reading
- National Academies of Sciences, Engineering, and Medicine, Understanding Reproducibility and Replicability, in Reproducibility and Replicability in Science (2019).
- W3C, PROV Data Model (2013), for representing derivation and responsibility. The numerical examples here are original illustrations.