field lab

the field lab evaluates coding-agent systems through repeatable engineering work. it measures what the operator must understand, supervise, recover, and verify. its output is comparable run evidence rather than a single winner score.

protocol

each primary harness is evaluated against the same repository state and task specification. a run should exercise:

1. architecture inspection with file-level evidence.
2. planning before a multi-file change.
3. implementation with repository-native verification.
4. interruption, denial, failure, and recovery.
5. session continuity and durable-state recovery.
6. delegated analysis with a bounded result.
7. isolated work in a separate worktree.
8. final diff review and completion evidence.

the task specification defines pass conditions before the agent begins. a run records failed scenarios and operator interventions rather than editing them out.

launch baseline

the first baseline records codex implementing the v4 publication from commit 60aa685. the public task specification is this repository’s accepted launch plan. the codex publication baseline publishes completed scenarios, skipped scenarios, artifacts, redactions, and limitations together.

the equivalent current claude code run is pending. current claude code guidance remains source-verified until that run is performed from the same base commit and task specification.

published data

run records conform to the repository run schema. public artifacts may include commits, pull requests, test logs, screenshots, and concise design-review notes.

exclude these from every public run:

• raw chat or agent transcripts.
• credentials, environment values, or account identifiers.
• private repository names or private absolute paths.
• personal data unrelated to the engineering result.
• inferred token cost when the product leaves it unavailable.
• a success claim without evidence from the layer it describes.

interpretation

elapsed time is useful only with operator interventions and review time beside it. tool-call count can describe a run. quality requires review evidence from the resulting work. resource measurements must identify what ran locally and what remained provider-hosted.

comparative conclusions require comparable runs. a missing run remains visible as pending until comparable hands-on evidence exists.