Import a case file

Most teams write their cases down long before they own a way to run them. Usually it is one file of JSON, or a directory of YAML with a file per case, each carrying a prompt and a list of things the answer ought to say. anpord eval import reads what they wrote and writes a *.eval.ts suite from it.

anpord eval import --format evals-json ./evals.json --out ./release-notes.eval.ts

Leave --out off and the suite goes to stdout, so you can read it before it lands anywhere.

The formats

--format names the shape of the file, not the team that wrote it. Anpord reads two today:

`--format`	Shape
`evals-json`	One JSON file holding every case
`yaml`	One YAML file per case, or a directory of them

The path can be a single file or, for a format that keeps one case per file, a directory.

evals-json

evals-json is one file holding every case:

{
  "skill_name": "release notes",
  "evals": [
    {
      "id": 1,
      "name": "Lists the breaking change",
      "prompt": "Read CHANGELOG.md and write the release notes.",
      "expected_output": "A short note naming the breaking change.",
      "files": ["fixtures/changelog"],
      "assertions": []
    }
  ]
}

Only id, prompt and assertions are required. name labels the case, expected_output describes the answer in prose, and files names the fixture directories the case starts from.

Two dialects

An assertion comes in two shapes, and both turn up in the wild, sometimes in sibling files. The first is a plain string:

"assertions": ["The summary reads as though a maintainer wrote it."]

The second is an object naming a check and the strings it looks for:

"assertions": [
  { "text": "names the removed flag", "kind": "content_contains_all", "needles": ["--legacy", "removed"] }
]

The import reads both from the same file. What it does with them is not the same.

What converts

A structured assertion is mechanical. It says which strings to look for and what rule to apply to them, so it converts exactly:

`kind`	Becomes
`content_contains_any`	`containsAny(answer, needles)`
`content_contains_all`	`containsAll(answer, needles)`
`content_contains_none`	`containsNone(answer, needles)`

The case gets a validate function that reads the agent’s final answer and returns whether every check held. Those three helpers are written into the file rather than imported, so an imported suite is one module a reader can follow and edit without learning a scorer library first. Matching is case-insensitive, which is what somebody writing “mentions the dashboard” means. The assertion’s text stays above the call as a comment, so the generated line still says what the author meant by it.

What does not

A prose assertion cannot be converted, and Anpord does not try. “Reads as though a maintainer wrote it” has no mechanical reading; any conversion would be a guess at what the author meant. A guess that guesses wrong produces a suite that passes while measuring the wrong thing, which is worse than no suite at all, because it is a green check nobody has reason to doubt. So a prose assertion is written into the generated file as the author’s own words, verbatim, next to a placeholder that is always false:

validate: async (context) => {
  const answer = await context.answer();

  return [
    /* Write this check, then delete the line under it: */
    /* The summary reads as though a maintainer wrote it. */
    unwritten("The summary reads as though a maintainer wrote it."),
  ].every(Boolean);
},

The file compiles and runs. The case fails until somebody replaces the placeholder with a real check. That is the point: an unwritten check reads as a failure, never as a pass. The command says so on stderr when it finishes:

Read 3 cases, converted 4 assertions. 3 assertions could not be converted and
need a human: each is written as prose, and the suite fails until you write the
check it describes.

expected_output is prose too, so it becomes a comment on the case rather than a check. It describes the answer; it does not decide one.

Fixtures do not travel

files names fixture directories by convention. The JSON carries the names, never the contents, so the import cannot fill them in. Every case gets an empty files({}) source and a comment naming what the entry asked for:

/* This case named fixtures/changelog. The JSON carries the directory names,
   not their contents, so add the files here. */
source: files({}),

Fill each one from the directory it names, or point the case at a repository instead.

The workflow

Import. Run the command and read the suite before writing it anywhere.
Fill the fixtures. Replace each files({}) with the files the case starts from.
Pick a task. The suite ships with one harness, model and sandbox. Edit it, or add variants to compare several.
Write the checks. Work through every placeholder. Each one carries the sentence that specifies it.
Run it. anpord eval ./release-notes.eval.ts.

Steps two and four are the work. Nothing that comes out of step one is worth trusting until they are done, and the suite stays red until they are.

When an evals-json file is refused

An import fails rather than writing a suite it is not sure of.

Refusal	Meaning
`CaseFileUnreadable`	The path does not exist or cannot be read
`CaseFileNotJson`	The file is not JSON
`CaseFileNotEvalsJson`	A case does not match the format, named by its `id` and the field
`CaseFileEmpty`	`evals` is empty, so there is nothing to import

A shape mismatch names the case the way you would search for it:

evals.json is not an evals-json file: case 47, prompt: Expected string, actual 12

yaml

yaml is one case per file. A directory imports as one suite, a file at a time, sorted by name:

anpord eval import --format yaml ./cases --out ./browsing.eval.ts

Point it at a single file instead and you get a suite of one. A directory is read for *.yaml and *.yml; anything else in it is left alone. Each file holds four fields:

name: Checkout flow
task: Open the demo storefront, add the sample widget and pay with the test card
judge_context:
  - The agent must reach the basket page before paying
  - The agent must use the test card, never a real one
max_steps: 12

name and task are required. judge_context lists what a person told a model judge to look for, and max_steps caps how long the case may run.

What converts

task becomes the case prompt, verbatim. name becomes the case name, slugged. That is the whole of it.

What does not

Nothing else does, and that is the point. Every judge_context entry is prose one person wrote for a model judge to read. “The agent must use the test card, never a real one” has no mechanical reading: it is not a string to look for, it is a description of correct behaviour that a judge weighs. Pattern-matching it into a contains check would produce a suite that goes green while measuring something the author never asked for. So each entry is written into the file as the author’s own words, next to the same always-false placeholder evals-json uses:

validate: async (context) => {
  const answer = await context.answer();

  return [
    /* Write this check, then delete the line under it: */
    /* The agent must use the test card, never a real one */
    unwritten("The agent must use the test card, never a real one"),
  ].every(Boolean);
},

A file that names no judge_context gets one placeholder anyway. Nothing in it says what a good answer is, so the case fails until somebody writes that down. max_steps becomes a comment on the case rather than a setting. Anpord has no step budget, so there is nothing to set it to, but the number says how much room the case had when its author last ran it, which is worth keeping:

/* The file allowed 12 steps. Anpord does not cap steps, so this is a note, not a limit. */

Every case also gets an empty files({}) source. The YAML names nothing the case starts from, so there is nothing to fill in from.

The workflow

Import. Run the command and read the suite before writing it anywhere.
Fill the sources. Replace each files({}) with what the case works on, or point it at a repository.
Pick a task. Edit the one harness, model and sandbox the suite ships with, or add variants.
Write the checks. Work through every placeholder. Each carries the sentence its author wrote for the judge, which is the specification.
Run it. anpord eval ./browsing.eval.ts.

Step four is the work. The suite stays red until it is done.

When a yaml file is refused

Refusal	Meaning
`CaseFileUnreadable`	The path does not exist or cannot be read
`CaseDirectoryUnreadable`	The directory exists but cannot be listed
`CaseFileNotYaml`	A file is not YAML
`CaseFileNotYamlCase`	A file is YAML but not a case, named by the field
`CaseDirectoryEmpty`	The directory holds no `.yaml` or `.yml` files

A file that fails to decode stops the import by name rather than being skipped quietly, so a directory that half-imported is never mistaken for one that imported:

cases/checkout.yaml is not a yaml case: task: Expected string, actual 12

Getting started

Evals

Guides

Prompts

Tools

Help

Import a case file

The formats

evals-json

Two dialects

What converts

What does not

Fixtures do not travel

The workflow

When an evals-json file is refused

yaml

What converts

What does not

The workflow

When a yaml file is refused

​The formats

​evals-json

​Two dialects

​What converts

​What does not

​Fixtures do not travel

​The workflow

​When an evals-json file is refused

​yaml

​What converts

​What does not

​The workflow

​When a yaml file is refused

The formats

evals-json

Two dialects

What converts

What does not

Fixtures do not travel

The workflow

When an evals-json file is refused

yaml

What converts

What does not

The workflow

When a yaml file is refused