# Benchmark an agent

Give an agent a set of tasks in your app, and Squiggle scores each one by checking the app afterwards: did the cart end up with two items? Use it to compare agents, models or prompts on your own journeys.

> **Before you start**
>
> A benchmark needs a suite: `benchmark/suite.yaml` in your project, describing the tasks. Your app must be running on a simulator through `squiggle up`.

## Write a suite

```yaml
suite: checkout-tasks
version: 1.0.0
apps:
  - id: com.acme.shop
    label: Acme Shop (iOS simulator)
tasks:
  - id: add-two-items
    app: com.acme.shop
    prompt: >
      Put exactly two items in the cart and leave the cart showing them.
    goal:
      route: Cart
      visible: { text: "2 items" }
    prerequisite:
      reset: relaunch
      route: Home
    reference: programs/add-two-items.reference.yaml
    wrong: programs/add-two-items.wrong.yaml
```

| Key | What it is |
| --- | --- |
| `prompt` | What the agent is asked to do, in plain words. |
| `goal` | What must be true at the end, written as flow assertions. This is the score. |
| `prerequisite` | Where each attempt starts, so every agent starts from the same screen. |
| `reference` | An intent program that does the task correctly, in benchmark/programs/. |
| `wrong` | A program that gets close but doesn’t finish the task. It must fail the goal. |

See [Intent programs](https://squiggle.sh/docs/agents/programs) for how to write the two programs.

## Run it

```shell
squiggle benchmark list                  # the tasks in the suite
squiggle benchmark run --reference       # check the suite itself
squiggle benchmark run --agent 'claude -p "$SQUIGGLE_BENCHMARK_PROMPT"'
```

Run `--reference` first. It runs each task’s reference program, which should pass, and its wrong program, which should fail. If both pass, the goal is too loose to tell them apart, and the task is reported as `indiscriminate` and scores nothing until you tighten it.

`--agent` runs your command once per task, with the prompt in `$SQUIGGLE_BENCHMARK_PROMPT` and the task id in `$SQUIGGLE_BENCHMARK_TASK`. The agent can drive the app any way it likes; only the end state is scored. Use `--task <id>` to run one task.

## Read the score

- **The task’s goal is the one checked**, whatever goal the agent’s own program declares, so an agent can’t pass by aiming lower.
- **Unproven is neither a pass nor a fail.** A task with no simulator, no app, or a busy simulator is left out of the score and counted beside it, so three tasks out of fifteen never reads as a score over fifteen.
- **Compare like with like.** A score belongs to one suite version. Change the tasks, and bump the version.

```shell
squiggle benchmark history 3
```

Output:

```text
benchmark history last 3
when              agent      suite  score  pass  fail  turns
2026-09-09 18:16  reference  1.0.0   100%     5     0      0
2026-09-09 18:17  reference  1.0.0   100%     5     0      0
2026-09-09 18:18  reference  1.0.0   100%     5     0      0
```