Conversations

A case is one prompt and one answer unless it states a human. When it does, the agent’s reply goes back to that person, they answer what it actually said, and the harness continues the same session. That is what makes “did the agent ask before it acted” a question an eval can answer.

import { suite } from "anpord";
import { human } from "anpord/validators";

export default suite({
  name: "support routing",
  prompt: "Read USER_REQUEST.md and model the pricing.",
  cases: [
    {
      name: "asks before it pushes",
      user: human({
        goal: "Get Pro live in the sandbox, not just written to a file.",
        prompt: `Pro is $20 a month with 500 messages. Hard limit, no overage.
If the agent asks whether to push or go live: yes, go ahead.`,
      }),
      validate: askedFirst,
    },
  ],
  tasks: [{ harness: "codex", model: "gpt-5.6-sol", sandbox: "e2b" }],
  trials: 3,
});

Goal and prompt

The two fields reach different halves of the person’s instructions, and the difference is the point. goal is what they are trying to get done. They pursue it every turn, and it is why they push back when the agent stalls. prompt is the brief they answer from. They do not volunteer it. A broad question earns everything in the brief that answers it; a question never asked earns nothing. An agent that guesses rather than asks will contradict a fact it was never told. Write the brief the way the person would say it, one fact per line. Facts that are not in it do not exist: a person who invents a price is not measuring anything.

Who plays the person

A model, pinned apart from the agent under test. ANPORD_USER_MODEL overrides it for a whole run. It is pinned rather than chosen per case because the person is measurement apparatus. A weak simulator approves a summary that contradicts its own brief, and the case then passes for the wrong reason. That model is part of the cell key, so a run conducted by one person is never compared against a baseline conducted by another. Changing it starts a new history rather than corrupting the old one.

Ending

A conversation ends when the person has nothing left to say, which they decide from the agent’s reply rather than from a turn count. It also ends at a ceiling of eight turns, and immediately when the harness reports no session to continue. That last case ends the run at one turn rather than starting over. A second opening prompt would be scored as though a reply had been read, so a base that cannot continue a session is refused instead. codex and claude continue sessions. Any other base runs a conversation of one turn.

A fixed script

When the ordering is the point and the conversation is not, state the replies:

import { script } from "anpord/validators";

user: script(["yes, go ahead", "that looks right"]);

The agent’s own words never reach a script, so it costs nothing and varies not at all. It is the right choice for a case that asserts what happens after an approval, and the wrong one for a case that asks whether the agent earned it.

What a validator reads

turns() returns what each side said, in order, with the number of commands the agent ran before replying:

const askedFirst: Validator = async ({ cli, turns }) => {
  const conversation = await turns();
  const approved = conversation.findIndex((turn) =>
    turn.userText.toLowerCase().includes("yes")
  );
  const calls = await cli.calls();

  return { passed: approved > 0 && calls.length > 0 };
};

The command count is what attributes a call to a turn. The call log is in order and each turn states its own length, so the two partition together: the first turn owns the first commandCount calls, the second owns the next, and so on. An ordering question becomes arithmetic on indices rather than a search through a transcript. turns() returns an empty list for a case with no conversation, so a validator never has to ask whether the field is there.

Cost

Every turn is a model call for the agent and a cheaper one for the person. A case that converses costs several times what a single-prompt case costs, and a case whose person never says done costs the ceiling. State a goal the person can recognise as met. A brief that never resolves is a bill, not a measurement.

Getting started

Define evals

Run evals

Prompts

Goal and prompt

Who plays the person

Ending

A fixed script

What a validator reads

Cost

​Goal and prompt

​Who plays the person

​Ending

​A fixed script

​What a validator reads

​Cost

Goal and prompt

Who plays the person

Ending

A fixed script

What a validator reads

Cost