Skip to main content
A case describes one coding task:
Keep name short. Write goal as the task, not the full agent instruction. Put shared instructions in the run prompt.

Source

Omit source for an empty workspace, or set it explicitly:
Use a repository for larger tasks. Pin ref to a commit when runs must start from the same files.
Use inline files for small fixtures:

Setup and verification

setup runs after the source is prepared. A nonzero exit prevents that trial’s harness from starting and fails the run. Omit it when no preparation is needed. verify runs after the harness finishes. Exit code 0 passes. A nonzero exit fails when the command produced usable evidence. A verifier that cannot execute, or a test runner that reports no tests, produces a void trial. Use set -o pipefail in shell pipelines. Set verify: null to record the trajectory without scoring the trial. Void trials do not affect pass rate or create a baseline.