Running an experiment

Give the scientific test a stable identity

Start with the relevant question in QUESTIONS.md, current status, and any plan governing the work. Ask what decision this test will change. A new comparison or hypothesis may deserve an experiment record. Another seed, shard, or retry of the same design belongs in its Runs table. A backend job counts an execution attempt; it does not define the scientific unit of work.

Reserve the next free ID through the project's coordination mechanism before creating the record and index entry. Keep the ID in both filename and first heading. A pilot, reanalysis, or full-scale follow-up that is not a separate experiment can use a revision-letter suffix when it needs its own record. Retain the relationship to its parent; a new identifier does not make reused evidence independent.

Use proposed while the design is incomplete, then planned when method and implementation are ready. Agree when results will be reported: as they arrive, after each experiment, after a batch review, or at the session's end.

Separate the design, execution, and interpretation

Before execution, the record describes the test and what its possible outcomes would mean. During execution, it records attempts and their outputs. Afterward, it provides the evidence needed to assess or reproduce the result. Keeping these parts separate preserves the distinction between an earlier prediction and a later interpretation.

Metadata supports navigation and coordination; it does not certify scientific validity. A completed experiment can have a negative answer. A completed job can belong to an experiment whose results still need interpretation. Registration belongs to each estimand, the quantity the experiment aims to estimate. Lifecycle status belongs to the experiment. Neither substitutes for a recorded review decision.

An amendment should identify what changed and what prompted it. Preserve the design and review history so readers can distinguish choices made before execution from later revisions. The record reference lists the exact metadata syntax and section headings.

Connect the experiment to the evidence around it

Research questions identify what the test addresses; a plan explains why the work is being undertaken. Informed by records the earlier observations, analyses, or literature that shaped the design. That history matters even when the prior result was exploratory or arrived as an agent's summary.

Run and artifact links let a reader inspect the work behind the observations. Findings combine results from several experiments; links to them identify where an experiment is used. Claim citations identify the specific estimand offered as evidence for an assertion in a paper. The paper uses that evidence; it does not establish the experiment's result.

In a fictional retrieval project, EXP-102 investigates a pattern in answer accuracy by document length. It links back to the breakdown that motivated it, and a later claim points to its measured quantity. If that quantity is corrected, the experiment ID lets a reviewer find and update dependent records.

Give primary, secondary, and gate statistics different jobs

The primary statistic answers the main question. In the retrieval example, it is the paired difference in answer accuracy over the intended population. Secondary statistics explain other aspects of performance or test additional questions, such as how that difference varies with document length. State their roles before interpreting the run.

Primary versus secondary describes purpose; registered versus found describes how the analysis was selected. A secondary comparison may have been specified in advance. An unexpected breakdown selected after seeing outcomes is found, even when it is more interesting than the primary result. Record both dimensions. A planned experiment can produce numbers that were not preregistered.

A scientific gate has a third job: establish whether a measurement is interpretable. If the scorer cannot handle known-answer cases, an accuracy comparison may be void. Passing that check enables interpretation; it is not evidence that the revised retrieval method improves accuracy. Report each statistic with its checks, uncertainty, and scope, and keep an inconclusive primary result visible beside any promising secondary pattern.

The evidence guide follows that pattern into reuse and confirmation. The estimand reference supplies the registration grammar, which does not encode primary or secondary role by itself.

Specify the decision rule before collecting data

Link the research question and state a falsifiable hypothesis. Under Method, name the instrument, data and sampling, conditions and controls, metrics, script, revision, and replication command. Give each estimand a stable E# heading and its own registration value.

## Estimands

### E1 Overall answer-accuracy difference
**Registration**: registered. Revised minus baseline accuracy;
positive direction and decision rule fixed before collection.

### E2 Scoring validity
**Registration**: gate. Interpret E1 only if the predeclared
scoring checks pass.

## Informed by

- [[RQ1]]

## Decision rule (a priori)

- Scoring checks fail: E1 is uninterpretable; stop and diagnose.
- The document-clustered 95% interval lies above zero:
  support an improvement for the specified population.
- The interval lies below zero: report worse performance.
- The interval includes zero: improvement is not established.

The complete design must specify those scoring checks and their acceptance criteria. Write predictions with expected ranges, reasoning, and null conditions. Cover outcomes that leave the question unanswered. Under Informed by, link everything read or measured that shaped the design, including unregistered observations from earlier runs. Update that history at each amendment without rewriting the original predictions.

Review the design and test the instrument at small scale

Pause for human design review before preregistration. Then present the predictions and decision rule for human preregistration review before execution and outcome inspection. Record the decision and approved scope in the experiment. An agent's prepared analysis or approval recommendation cannot stand in for the human decision.

Validate the manipulation or scoring on the cheapest sufficient test. For retrieval, known-answer cases can expose a scorer that always returns the same value. For an intervention, check that changing its key parameter changes the intended phenomenon. A successful pipeline check alone does not establish scientific validity.

Use a bounded pilot when scale, cost, or failure risk warrants it. Define its workload and stop condition before launch. Record what it establishes and where it may not generalize. At pilot-complete, apply the gate and seek approval for a gated follow-up. If the pilot changes the design, preserve the amendment and its information history; do not retroactively register observations already seen.

Freeze the approved work, then submit through the runner

Pin the code revision or content identity, data version, parameters, seed, and relevant environment. Preserve raw outputs and assign each attempt a distinct artifact location. A command copied into Method is useful only if its code and inputs remain recoverable. Retain their identities alongside output URIs, hashes, and retrieval dates.

Read the project's RUNNER.md for submission, status, logs, artifact retrieval, cancellation, and processed-state procedures. Tag every job with the experiment ID and, for a shared runner, the project ID. Research approval defines the scientific scope; the runner command implements it. Neither bypasses authorization for spending, remote execution, or unattended work.

Add every attempt to Runs using Backend, Job ID, Description, Status, and Artifacts. Move the experiment through queued and running as appropriate. A retry receives a new job ID and links to the original. Keep the failure row, logs, and useful partial outputs; never replace them with the successful attempt.

Process every terminal outcome in order

A backend's completion status says computation ended. Processed state says its outputs were checked and incorporated into durable notebook evidence. Succeeded, failed, and canceled jobs all need a disposition.

1. Validate artifacts and instrument. Authenticate the output and producing revision. Check expected cells, missingness, plausible values, and scientific gates. Check both whether the file is readable and whether the measurement is interpretable.
2. Record the result. Analyze primary outcomes before exploration. Compare every prediction with observations, including missing or void results. Record uncertainty, exclusions, anomalies, and the conclusion at the tested scope.
3. Apply the decision rule. Use its failed and null branches too. A disappointing answer does not authorize an unregistered rescue campaign.
4. Pause for interpretation review. Give the human reviewer the preregistration, checks, exclusions, estimates, exploration, and proposed follow-up. Record corrections, then update affected question, status, index, priority, claim, and publication pointers.
5. Secure the evidence. With retained backend artifacts, retrieve and verify outputs, validate the notebook, and commit within its version-control boundary. Write the processed mark last, linking the record and notebook revision where supported. For a manually reached persistent host, retrieve outputs or record their location and unsynced state. For local runs, record numbers before another run can overwrite the only copy; the experiment status serves in place of a processed flag.
6. Report. Use the interaction mode agreed at the session's start.

Reprocessing starts by checking the existing evidence links, not adding duplicate entries. The evidence workflow covers statistical interpretation and the distinctions needed before publication use.

Review follow-ups before authorizing jobs

An experiment can include conditional follow-ups before it runs: if scoring fails, diagnose the instrument; if the pilot meets its criterion, consider a larger measurement. The decision rule makes those branches visible. After processing, Follow-ups records which branches the evidence actually makes relevant and any new questions it raises.

A gated follow-up names the triggering evidence, the proposed work and its bounds, and the review decision required before execution. It differs from a scientific gate: “this measurement is interpretable” and “this next experiment is authorized” are separate judgments. A checked control, a completed run, or an unchecked task entry cannot grant approval.

Suppose the overall retrieval comparison is inconclusive but its length breakdown suggests a benefit. Record the pattern as found and propose a scoped reanalysis or independent confirmation. Keep the overall conclusion unchanged. Link an adopted follow-up to its parent evidence through Informed by, and put larger multi-stage work into a plan.

A reviewer may choose to stop, defer, repair the method, or approve work within stated limits. Keep that decision alongside the proposal so another agent cannot mistake an interesting idea for an instruction. An experiment can be completed with unexecuted follow-ups. See the follow-up record pattern.

Leave a decision another session can act on

Use completed when runs finished and results are documented, including a negative result. Use abandoned when work stops early, with the reason. Use blocked when continuation awaits a named dependency. Avoid closed, which hides that distinction.

For a handoff, leave the current status, artifact locations, review decision, and concrete next action in the record. Resume by checking these against runner state and processed evidence before submitting anything. Feed the disposition into the governing plan. A single experiment's conclusion stays here; cross-experiment synthesis and reviewed claims proceed through the paper workflow.