Agents & MCP

Why the loop is the product

A conventional agent driving an app does this: screenshot, infer, tap, screenshot, infer, tap. Every action costs an inference, every screenshot costs thousands of tokens, and any of them can hallucinate a button that is not there. Twenty steps is twenty chances to go wrong and twenty times the bill.

Squiggle makes the model a compiler. It emits one goal-conditioned intent program — a plan with branches, declared escalation points, and a goal that gets verified afterwards — and the runtime executes it against the live semantic graph. Turns become O(escalations) rather than O(actions).

intent: check out with a saved cardgoal:  route: OrderConfirmation  visible: { text: "Order placed" }plan:  - tap: { testID: cart }  - scrollUntilVisible: { element: { testID: checkout }, direction: down }  - tap: { testID: checkout }  - branch:      if: { visible: { text: "Add a card" } }      then:        - escalate: { reason: "no saved card on this account", want: decision }      else:        - tap: { testID: pay-now }  - waitFor: { route: OrderConfirmation, timeout: auto }

On the reference journey, run from a dirty app, that is 3/3 passes at zero model turns, and 20/20 on repeat. The model was called once, to write the program.

What happens before anything is tapped

A program is gated offline before it is allowed to touch the device, through a pyramid that degrades honestly rather than silently:

Static

Does every selector name something the route graph has actually seen? A program aiming at a screen that does not exist is stopped here, with zero live actions.

Replay

Do the program’s assertions hold against the newest relevant recording? This checks the plan against territory we have already been through.

No coverage

A program with no covering recording degrades to static-only with a note saying so. It never passes silently — an ungated run that claims to be gated is worse than no gate.

alternatives turns this into best-of-N: emit several candidate programs in the same turn, let the gates score them all, run the winner, and keep the runners-up as pre-validated fallbacks. Zero extra model turns.

The deny rail

An actuation blocklist enforced at delivery time — below the model, in the interpreter, not in a prompt. A model cannot talk its way past it because it is not being asked.

Your app can declare what a control is, and a declaration always beats an inference from copy:

<Pressable  testID="delete-account"  {...agentHint({ destructive: true, role: 'button', note: 'permanent, no undo' })}/>

A rail hit that the run genuinely needs escalates as needs_approval. The grant is scoped, single-use and logged — and approvals are never cached. It is answered with approve_action, a separate tool marked as requiring user interaction, so a client puts it in front of a human rather than letting an agent grant its own override by answering an escalation.

Escalation packets

When the runtime hits a real decision point it freezes and builds a packet: the program counter, the reason, delta narration since the program started, ranked heal candidates, the log tail, and an evidence coordinate. Big payloads stay behind resource references rather than being pasted in.

A seeded rename — someone changed a testID — is repaired by exactly one escalation in a 1,143-byte packet. The answer is one constrained arm: pick, patch, retry, skip, abort, or learnInterrupt. A deny-rail escalation is the exception: resume_intent refuses it and names approve_action.

An agentic flow's goal step arrives the same way, with the reason agent_step and the want drive. The packet carries the goal, the assertions that will verify it and, from screen memory, the known taps toward the screens it names. The answer is a patch of ordinary steps, and when they verify, they become the recording.

Why it gets cheaper the more you use it

This is the part that compounds, and it is worth reading as a list because each item removes a future model call permanently:

Delta narration — cheaper perception

After the first look, sinceLastLook returns what changed rather than a second full dump, while reconstructing ground truth exactly. How much it saves depends on the screen: from 14% on a tiny one to 97% on a busy one across 75 measured runs. It is always smaller than the full dump, which is the part the acceptance gate holds.

Learned interrupts — one escalation per popup, ever

An unhandled interstitial escalates once. learnInterrupt persists the handler, and interstitial autopilot runs it below every step boundary from then on. The tenth time that cookie banner appears it costs nothing.

The flow library — flows are skills

A successful program can be saved, generalised into a parameterised version, and replayed. Replaying a library flow costs zero model turns. Your agent’s capability grows as a library rather than as a longer prompt.

The route graph — knowledge instead of exploration

Every run records observed transitions with counts and p95 timings. predict answers “what has tapping this historically led to” with no live actions; route gives the shortest known path to a screen; and timeout: auto takes its number from what the app has actually done rather than from the model guessing milliseconds. Unknown transitions return “no history” — the graph never guesses.

The decision cache — a repeat escalation costs nothing

Answers are memoised by screen signature, failure kind, goal shape and app build. A hit costs zero model calls and still flows through the rail and the gates, so a signature collision produces a failed gate rather than a wrong action.

Inferred contracts — tests nobody wrote

What held across every green recording of a screen becomes a contract, with its confidence attached. Violations report as contract_drift beside the goal, never as a hard failure, because inferred things can be wrong.

The tools

Eighty-three tools over one persistent warm session, held across calls, so each verb is a ~3 ms socket round trip rather than a process boot. Registering the server never boots a simulator.

Which app: target

Several injected apps can be up at once, and a session attaches to the most recently launched one, which is often not the one you meant. target lists them and, with to, switches the session and its screen memory. Several matches are refused with the list rather than resolved by guessing. Call it first whenever the user names an app.

Read this first: status

The capability matrix — transport, session, injected?, dylib freshness, captures, and a map of which capabilities are unusable right now and why. An unknown dylib version is treated as stale, never as trustworthy.

Perception

describe (interactive-first, token-budgeted, stable short refs like n42, or delta narration), find (a confidence verdict — unique / ambiguous / none — with ranked alternatives up front, so you disambiguate before acting rather than after failing), query, assert, wait_for, and screenshot as the escape hatch, never the default.

Code ↔ screen

In a development build every element of your code carries callSite — the line that writes it, like src/screens/Login.tsx:42:7 — so an agent opens that line instead of grepping for a label. It is a selector too: after editing line 42, find with { callSite: "Login.tsx:42" } shows what it renders now. Live runs only: library_save rewrites it to a stable selector, because a saved flow cannot keep a line number. inspect looks inside one component — its live props, its hooks and state by the names in your source, the contexts it reads and who provides them, its style and whether React Compiler memoized it. It is read-only: the desktop app can edit props and state, an agent cannot.

Frames

filmstrip, for the defects the graph cannot see — because the graph is correct the whole time and only the rendering is briefly wrong. An overlay painting in the corner for one frame, a row drawn with new content in its old position, a janky drag: describe reports a correct screen at every point and assert passes. It renders a labelled contact sheet from a recording — each frame stamped with its timestamp and the screen route it was on, and old→new on the frames inside a navigation, because a transition shows two screens at once and either name alone would be half wrong. A second mode returns per-frame brightness over a crop, which is the only way to prove a single-frame artifact: a sheet samples every ~45 ms at best, so a clean strip is not proof that nothing flashed. Needs ffmpeg; optional, and nothing else depends on it.

Actuation

tap, input_text, swipe, scroll, back, launch_app, run_flow. Each takes a ref from your last look as well as a selector. A success means dispatched, not effected — which is why programs are the fast path. Each of them settles and returns the screen delta as observed in the same reply, so an action does not have to be followed by a look; observe: false gets the bare ack.

Reaching what is off screen

scroll_until_visible loops server-side and costs one round trip where scrolling and looking by hand costs four. Read its found — ok only says the loop ran.

Batching

batch runs several of those tools in one round trip — same names, same arguments, nothing new to learn — stopping at the first failure and observing only at the end. It is the rung between a single poke and a program, and deliberately weaker than one: no offline gate-check, no escalation, no loops, no goal:. Those absences are the reason to graduate.

Programs

run_intent’s program argument is typed inline with the common subset, so a first program needs no schema fetch; the shape is permissive, and unlisted step kinds and selector fields pass through rather than being stripped. intent_schema remains the full grammar. intent_wait (long-poll to the next boundary), resume_intent, intent_stop, and steer — a hint injected into a live run, enforced at the guard seam rather than offered as advice, because a hint the program can ignore is a suggestion. intent_continue resumes a run that died (a server restart, a crash): it re-drives the journey from the checkpoint journal and answers every escalation it already paid for from the record, so the recovered part costs zero model turns.

Memory and planning

library_list/library_save, predict, route, interface (the route graph compiled into a typed per-screen SDK), diff_runs, affected_by_diff (which library flows an interface change puts at risk — expanded through the import graph, so a change to a shared hook marks the screens it reaches, not nothing).

Recording

record_start and record_stop capture a black-box session from the live app into a .sqrec, and replay_recording replays one in-process with no app, optionally asserting a flow against it.

Where to start: briefs

The investigations the app opened on its own — a crash, a friction cluster, a perf finding — each with the repro recording, the causal chain, the suspected source line and the flows that cover that screen. Start here when asked what is wrong with the app. It costs no model turns and drives nothing.

Evidence

explain walks the deterministic why-chain behind a coordinate — blame is a query, not an inference — and verify_citations re-resolves every evidence link in a narrative and flags the ones that do not hold. An uncited report is reported as uncheckable. publish_evidence attaches a recording, screenshot or filmstrip contact sheet to the pull request, editing its previous comment in place. It needs prEvidence: true in the project config, because an agent posting video of your app where collaborators can see it needs your say-so.

Data

The whole perf and product surface — the MCP catalogue is a superset of those CLI families, wrapping the same in-process functions rather than shelling out.

Serving fewer of them

The full catalogue is 83 tools — roughly 30,000 tokens of schema in every conversation, whether the session drives an app or profiles one. Set SQUIGGLE_MCP_MODE on the server and it registers fewer: drive drops the perf and product families (52 tools, about 8,000 tokens back), analyze drops the device half instead, and qa / perf / a11y / explore serve exactly the 14–18 tools their subagent runs under. Unset serves everything.

Prompts and resources

Two prompts, squiggle-drive-mode and squiggle-perf-mode, carry the same working agreements as the drive_mode_briefing and perf_mode_briefing tools, so a client can pin them instead of an agent fetching them. Resources: squiggle://screen (the current screen, with change notifications), squiggle://insights/latest, squiggle://license, the ten skill documents, and the templates squiggle://run/{runId}, squiggle://recording/{name} and squiggle://skill/{+id}.

A scoped-out tool is absent, never disabled. That distinction is load-bearing: a disabled tool is refused by the protocol layer before any of our code runs, so the agent gets a generic error instead of an explanation. An unrecognised mode value serves everything and says so — hiding tools because someone mistyped an env var looks exactly like a broken product.

What a client shows you

Every tool carries a derived display title, stamped from the tool name in one place so none of the eighty-two can forget — a client’s UI and its permission prompts read “Launch app” where the wire carries launch_app.

Tool icons and in-flight progress were built here and removed, which is worth recording because they are the obvious next reach. Icons are priced by transport, not by preference: an HTTP server may serve images from its own authority, a stdio server may use file:///, and any transport may use data:. Squiggle is stdio, so the cheap https:// option is nobody’s authority and renders nothing, and the schemes that do work cost 2,900–5,100 tokens across the catalogue — against the 7,800 that mode scoping reclaims. Decoration at half the saving is the wrong trade.

Progress is narrower than it sounds. A server can emit notifications/progress while a call runs, and only if the client sent a progressToken. There is no completion notification and no checkmark primitive: the tool result is the completion signal, and what a client draws when a call finishes is its own rendering of that result. intent_wait does stream progress, because there the payload is a real program counter rather than an elapsed-time label.

No call waits forever. A tool that has not answered within 90 seconds (15 minutes for the ones whose job is to take long, such as run_flow, crawl or intent_wait) is answered with an error saying so, and the server drops its app session, so the next call reconnects rather than queueing behind the one that hung. Connecting to the app gives up after 20 seconds with the reason: the app is not running, it is not injected, or another process such as Studio holds it.

What each tool costs you

Every tool carries a licence posture, and the four classes are the whole taxonomy: 1 free, 25 drive, 33 analyze, 24 agent.

free — status

Exactly one, and it is the one that tells you why the others are refusing. A capability report that needed a licence to read would be the worst possible thing to gate.

drive

Anything that touches the device: perception, actuation, recording, the flow library, branch, crawl.

analyze

The perf and product families, plus query, explain, diff_runs, affected_by_diff and filmstrip — everything that reads the store or a recording you already captured. Two perf-named tools are the exception: perf_mode_briefing and perf_narrative_context serve the intent runtime, so they are agent.

agent

The intent runtime itself: programs, escalations, approvals, briefings, probe_read, interface, invariants, legibility, and propose_fix, which turns a stored perf finding into an edit plan an agent applies itself.

The mapping is not a comment — it is a table three test suites enumerate, one of them against a live server, so a newly added tool cannot quietly inherit a default. It fails closed: unclassified means paid, and a fail-closed default is safe but is still not a decision, which is what the suite exists to say.

Read code-mode

probe_read collapses roughly twenty-five flat read tools into one verb. You write an async function body against a typed read capability surface and return only what you need — filtering and aggregating server-side:

// probe_read — write a program, return only what you needconst slow = await perf.sql(`  select name, avg(dur_us) avg from spans  where kind = 1 group by name order by avg desc limit 5`);const insights = await perf.insights();return { slow, blocking: insights.filter(i => i.severity === 'error').length };

Returning 20 MB so you can grep it in context is the exact failure this exists to prevent, and there is a hard result cap. Reads are low-consequence, so it trades “zero parse failures” for a tight runtime-error → edit → rerun loop.

Acts get no such treatment. They stay the governed intent DSL, because a tap can log out, delete, or pay. Security is layered and the sandbox is the last layer, not the guarantee: actuation is absent rather than gated (there is no drive verb to reach for), the authorization proxy runs in the parent, the database rejects writes, there is no network or filesystem, and every run is traced in a killable child process.

Scoring how well it drives

Everything above is a claim about a loop. squiggle benchmark is where the claim gets a number: a versioned suite of mobile-UI journeys, each one a natural-language objective handed to an agent, and the score is whether the task’s assertion held afterwards. That is the only reason this can exist at all — grading a mobile-UI agent is normally a human reading screenshots, and Squiggle grades from the runtime graph instead, so a result is auditable and re-running a recorded one gives the same number.

$ squiggle benchmark list                     # the task set$ squiggle benchmark run --reference          # the oracle, 0 model turns$ squiggle benchmark run --agent 'claude -p "$SQUIGGLE_BENCHMARK_PROMPT"'$ squiggle benchmark history 20               # the recorded trend

--reference runs each task’s own oracle program at zero model turns — the CI-shaped form, and a regression gate on the suite rather than on any agent. --agent <cmd> runs your command once per task with the prompt in an environment variable, waits for it to exit, and scores the end state through the same path. Nothing supervises how it drives, because the point is to score any MCP-speaking agent and not only the ones that use Squiggle the way we would.

Three rules do the work, and each of them refuses something:

The task owns the goal

Whatever goal: a submitted program declares is discarded and the task’s substituted. Otherwise a program passes by asking for something easier than it was set, and the leaderboard measures ambition.

Every task carries a counter-example that must fail

Half of a gate is proving it is not always-green. A goal that both the oracle and the deliberately-wrong program satisfy is reported indiscriminate and scores nothing — its passes mean nothing either.

unproven is neither a pass nor a fail

No device, no app, or a gate refusal before actuation all mean no data. Those tasks leave the denominator and are counted beside it, so a score over three of fifteen tasks can never be quoted as a score over fifteen.

Setting it up

$ squiggle init --mcp                # writes .mcp.json for you (merges, never overwrites)# or by hand, either way:$ claude mcp add squiggle -- squiggle mcp# .mcp.json at your project root:$ { "mcpServers": { "squiggle": { "command": "squiggle", "args": ["mcp"] } } }

squiggle init --mcp is a merge, not a write: every server already in your .mcp.json stays, along with any keys your client puts beside them. If squiggle is already registered it changes nothing — a command you pinned yourself is a command you meant. It is opt-in for the same reason: that file configures your agent client, not us.

Then install the methodology that travels with the tools:

$ squiggle agent kit --install          # install skills, subagents and commands into .claude/$ squiggle agent kit -o ./kit        # or stage all of it, hooks included, to read first$ squiggle agent rules               # an AGENTS.md fragment, generated from the catalogue

The kit is a build artifact, compiled from the DSL schema, the capability library and the mode briefings. It ships four skills (squiggle-setup, squiggle-drive, squiggle-perf, squiggle-triage), four subagents (qa, perf, a11y, explore), slash commands, and optional hooks. Every skill is also readable over the wire as a squiggle://skill/… resource, so a client that is not Claude Code still gets the guidance rather than eighty-three tools and no instructions.

--install writes the seventeen files a harness loads and withholds the hooks, because merging those edits your settings.json — not something an install should do behind your back. -o <dir> stages all twenty so you can read the hooks before arming them. A destination that is something else — a symlinked .claude/agents, say — is reported and skipped rather than overwritten, and everything else still installs.

The kit tells you when it is behind. Each skill carries the kit’s own content digest in its frontmatter, and both squiggle doctor and the MCP server compare it against the build you are running — the server by attaching a one-time note to a tool result, which is the only channel your agent actually reads. It is a hash of the generated content rather than the product version, so an upgrade that did not change the kit asks you to do nothing. When it does say something, run squiggle agent kit --install again.

A mode is a prompt, a goal template and a hard tool allowlist — not emphasis. The allowlist becomes the generated subagent’s frontmatter, so a perf agent cannot reach a drive verb by accident.

Governance

One live run per device

Agent and human can never actuate the same simulator at once.

Budgets

Escalations, actions and wall-clock, enforced by the runtime and reported in the run. A ceiling you hit at speed is a wall, so the runtime can advise “finish fast” and let the answerer drop optional work instead of losing the run.

Every acting call is logged

Each acting tool call appends to the project’s shared history tagged source: mcp, the same file the CLI and the desktop write. A human reading the log sees both halves of the work.

Every intent run auto-arms a recording

A failure already has its reproduction.

Licensing refuses inside the protocol

The server always starts — a non-zero exit reads to a client as a broken install, so the customer files a bug instead of buying a licence. It serves, and the refusal travels in the result envelope for the agent to relay. There is deliberately no activate tool: that would mean a licence key typed into a conversation, and into whatever that transcript syncs to.

Agentic flows in Studio

Agentic flows live on the desktop app’s Flows screen, in their own tab. A flow with an agent: step replays its recording, or stops at NEEDS AGENT with a prompt to paste into Claude Code or Copilot — Studio never calls a model itself. See Agentic flows.

A human can answer instead

An intent program run from the Flows screen shows its escalation packets in the run log, and you answer them by hand: pick a ranked candidate, retry, skip, abort, or approve a step the deny rail held. Once a packet is structured, a human is a perfectly good model — and free. The same seam serves Claude Code over MCP, a scripted answerer in a test, or you.