Benchmark an agent

For AI agents: this page is Markdown at https://squiggle.sh/docs/agents/benchmark.md. Every docs page is listed at https://squiggle.sh/llms.txt, and all of them are one file at https://squiggle.sh/llms-full.txt.

Give an agent a set of tasks in your app, and Squiggle scores each one by checking the app afterwards: did the cart end up with two items? Use it to compare agents, models or prompts on your own journeys.

Write a suite

benchmark/suite.yaml

suite: checkout-tasksversion: 1.0.0apps:  - id: com.acme.shop    label: Acme Shop (iOS simulator)tasks:  - id: add-two-items    app: com.acme.shop    prompt: >      Put exactly two items in the cart and leave the cart showing them.    goal:      route: Cart      visible: { text: "2 items" }    prerequisite:      reset: relaunch      route: Home    reference: programs/add-two-items.reference.yaml    wrong: programs/add-two-items.wrong.yaml

Key	What it is
`prompt`	What the agent is asked to do, in plain words.
`goal`	What must be true at the end, written as flow assertions. This is the score.
`prerequisite`	Where each attempt starts, so every agent starts from the same screen.
`reference`	An intent program that does the task correctly, in benchmark/programs/.
`wrong`	A program that gets close but doesn’t finish the task. It must fail the goal.

See Intent programs for how to write the two programs.

Run it

Terminal

$ squiggle benchmark list                  # the tasks in the suite$ squiggle benchmark run --reference       # check the suite itself$ squiggle benchmark run --agent 'claude -p "$SQUIGGLE_BENCHMARK_PROMPT"'

Run --reference first. It runs each task’s reference program, which should pass, and its wrong program, which should fail. If both pass, the goal is too loose to tell them apart, and the task is reported as indiscriminate and scores nothing until you tighten it.

--agent runs your command once per task, with the prompt in $SQUIGGLE_BENCHMARK_PROMPT and the task id in $SQUIGGLE_BENCHMARK_TASK. The agent can drive the app any way it likes; only the end state is scored. Use --task <id> to run one task.

Read the score

The task’s goal is the one checked, whatever goal the agent’s own program declares, so an agent can’t pass by aiming lower.
Unproven is neither a pass nor a fail. A task with no simulator, no app, or a busy simulator is left out of the score and counted beside it, so three tasks out of fifteen never reads as a score over fifteen.
Compare like with like. A score belongs to one suite version. Change the tasks, and bump the version.

Terminal

$ squiggle benchmark history 3benchmark history last 3when              agent      suite  score  pass  fail  turns2026-09-09 18:16  reference  1.0.0   100%     5     0      02026-09-09 18:17  reference  1.0.0   100%     5     0      02026-09-09 18:18  reference  1.0.0   100%     5     0      0