A person can have an experience that nobody else directly shares. Scientific inquiry begins when some aspect of that experience is reported, compared or linked with an observation procedure. The report is evidence with its own conditions of production.
Distinguish an experience from a claim about it
Suppose someone reports hearing a brief tone. Several questions are possible: was a tone physically presented, did the person experience one, what feature did they identify, and when did they decide to report it? An answer to one question does not automatically settle the others.
A microphone can help investigate the acoustic signal at its location. A button press can record a response. An interview can document the person's account. These observations contribute different information; none becomes a complete measurement of experience merely because it is stored numerically.
This distinction respects first-person reports while allowing their evidential role to be investigated. A report may be sincere and still involve uncertainty about timing, source or interpretation. Recognizing that uncertainty is compatible with taking the report seriously.
Turn a comparison into a shared procedure
Consider an invented task in which two readers independently classify one hundred brief descriptions as referring to a changing or an unchanging scene. Specify the category definitions, instructions and materials before comparing their reports. Preserve ambiguous cases rather than silently rewriting them after the results are known.
Suppose the readers both choose “changing” on forty descriptions and both choose “unchanging” on forty. They disagree in each direction on ten. Their observed agreement is 80%, and each reader uses each label half the time.
Under the independent-marginals comparison used by Cohen's kappa, expected agreement is 0.5 × 0.5 + 0.5 × 0.5 = 0.5. The coefficient is therefore (0.8 − 0.5)/(1 − 0.5) = 0.6. This summarizes a particular classification comparison; it does not establish that either reader's judgment is correct.
Agreement is not a universal measure of truth
The expected-agreement term depends on the readers' label frequencies. If both nearly always use one label, high raw agreement can carry a different interpretation. A coefficient should be reported with the underlying table and task, rather than treated as a context-free score of objectivity.
The readers might also share a misleading cue in the descriptions. They could agree reliably because the task unintentionally signals the expected response. Independent reporting removes some forms of direct influence between them; it does not remove every common cause of their judgments.
A better investigation can change the materials, vary the instructions or compare judgments with an independently specified feature of the scenes. These are new tests of the explanation for agreement. A single number cannot substitute for them.
Shared evidence need not erase perspective
Reports become publicly inspectable when their circumstances and transformations are documented. Other researchers can examine the same materials, repeat an analysis or ask a modified question. They may then discover that an apparent disagreement concerns different category meanings rather than different perceptions.
There can also be genuine differences in experience that remain scientifically interesting. The aim is not to require identical experience from every participant. It is to identify stable relationships among conditions, reports and other observations, while keeping the limits of each measurement clear.
For temporal experience, distinguish the stimulus sequence, the integration of that sequence, the remembered interval and the time of the report. The essays on the experienced present and consciousness and observation develop these separate questions.
Philosophical interpretation and physical evidence
The SMM writings connect temporal generation with broader questions about unity, appearance and experience. The expanded book's place of philosophy distinguishes that project from mathematical constructions intended to reproduce measurable physical structures.
A philosophical account can organize questions and suggest new comparisons. To support a distinct physical claim, it must specify an observable consequence that the inquiry can evaluate. Conversely, a successful measurement model need not exhaust the meaning of a person's experience or settle every metaphysical interpretation.
This distinction creates room for serious exchange. A report can be preserved as a report; an interpretation can be examined as an interpretation; and a quantitative prediction can face an appropriate test. Their connections become clearer when none is asked to stand in for all the others.
Sources and further reading
- Jacob Cohen, A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20 (1960), 37–46. The classification table here is invented for explanation.
- Reproducible observations and comparing worldviews fairly.