A fair comparison gives competing models the same observational task, makes their calibration choices visible and evaluates consequences that were not adjusted after seeing the outcome. The aim is to discover which distinctions the data can support.
Fix the question before ranking the models
A model can be useful for interpolation, long-term prediction, causal explanation or compact representation. Performance on one task does not automatically establish performance on another.
Before comparing, specify the observable, preparation, input range, uncertainty model and available calibration data. Decide which parameters each model may adjust and which measurements will be reserved for assessment.
A more elaborate model may describe a wider range, but it may also use more freedom. A simpler model may work well in a narrow regime without representing every mechanism. A fair comparison makes those differences visible rather than treating one fitted curve as a verdict on an entire programme.
Two models can agree at the start
Consider two dimensionless relaxation predictions after a normalized preparation: A(t) = e−t and B(t) = 1/(1 + t). Both begin at one and have initial slope minus one.
Near zero, their expansions are A = 1 − t + t²/2 + … and B = 1 − t + t² + …. Very early observations can therefore make the models look similar. Agreement on the initial value and slope does not establish the same later dynamics.
At t = 1, A predicts about 0.3679 and B predicts 0.5. At t = 4, they predict about 0.0183 and 0.2. The equations supply concrete candidate comparisons, but the instrument's sensitivity and noise determine which is informative.
Include the measurement model
Suppose, solely for a teaching calculation, that a reserved measurement at t = 1 reports z = 0.40 with independent Gaussian error of known standard deviation σ = 0.05. Neither model is refitted to this observation.
The log likelihood ratio in favour of A over B is [(z − B)² − (z − A)²]/(2σ²), approximately 1.79. The likelihood ratio is therefore about six. Under the stipulated noise model, this one observation favours A over B.
This is not a posterior probability that A is true. That would require prior model probabilities and a specified set of alternatives. Nor is six a universal evidential threshold. It is the result of one invented observation and one explicitly chosen error model.
If both models were allowed a fresh amplitude chosen from this one observation, each could fit it exactly. The comparison would no longer test the normalized predictions just stated. Parameter freedom changes the inferential task.
Choose the next measurement for a reason
The largest difference between model curves is not always the best measurement point. At late times, detector offsets, a reporting floor or uncertain background can dominate a small signal. At early times, finite response time can distort the initial slope.
A good design compares the predicted separation with the actual uncertainty and checks that both models use the same detector description. Otherwise a difference attributed to dynamics may really come from incompatible assumptions about measurement.
The exposure-window example shows why this matters: the same pulse can be partly captured, fully captured but diluted, or missed altogether. A model of the source must be passed through the measurement map before being compared with recorded values.
Make asymmetric freedom visible
It is unfair to hold one model to a fixed parameter while repeatedly tuning the other after each discrepancy, then describe their final agreement as equally predictive. It is equally unhelpful to deny a model a calibration that the physical question legitimately requires.
Report the calibration protocol, the number and role of adjusted parameters and the observations reserved for testing. If the comparison led to a model revision, label the revised result and evaluate it on another appropriate test.
Alternative models can also share an observational limit. If their predictions coincide over the accessible regime, a null distinction is a useful result. It identifies where the present experiment cannot choose between them and which new observable might help.
Apply the standard to a generative proposal
The book's tests with explicit alternatives asks the SMM/DTT programme to make precisely this kind of comparison. A generative account should calculate an observable with a fixed physical dictionary and state where it differs from a conventional or phenomenological alternative.
If the new account reproduces a relation already used as an input, that is a consistency check. If it fixes another relation that was previously adjustable and the relation survives a reserved test, the explanatory gain is more substantial.
A fair comparison can therefore strengthen an ambitious proposal by making its independent content easier to see. It turns a broad claim of superiority into a calculation with an observable consequence.
Sources and further reading
- The exponential/rational response comparison and Gaussian likelihood calculation are explicit teaching constructions, not experimental results.
- Mohamed Haj Yousef, Tests with explicit alternatives, A model that can be audited and Resolution sampling and missed events.

