Skip to main content

​
Evals

Evals let you define conversation test cases and re-run them after changes, so important chat workflows do not quietly break. Find them under App Blueprint → Evals. Use them for critical paths (invite, core intake, “where is my order”), not for every small tweak. Skip evals for one-off experiments and cosmetic page edits that never touch chat behavior. After a Usage-Driven fix, an eval can lock the original pain point so it does not regress (see Usage-Driven Development).

​
How to verify

  1. Create or open a set under App Blueprint → Evals. The list shows each set’s last run status and date so you can see what is still green without opening every set.
  2. Run before a risky Build and again after the version lands.
  3. Open the run detail; fix failures in Build; re-run until green enough to ship.
  4. Still Preview as for interactive pages; evals are not a full QA suite.

​
What you work with

Exact assertion UI can grow over time; treat a case as “this conversation should still succeed for this role and intent,” and read run results for pass/fail and failures.

​
Example case

Runs start from an empty throwaway database. Put fixture rows in the set’s seed code (for example an insert for job 1042) so the prompt can ask about that job.

​
Example run readout

After you change how clients learn status, re-run the same set. If the agent starts inventing status, the run fails; fix in Build, then run again before you treat the version as safe. Pair this with Preview as for the interactive check; evals catch regressions when you are not clicking every path by hand.

​
How to run them in practice

  1. Create an eval set under App Blueprint → Evals.
  2. Add cases for the few flows that would embarrass you if they broke.
  3. Run before a risky Build, and again after the version lands.
  4. Open the run detail; fix failures in Build; re-run until green enough to ship.
  5. Keep the set short so you actually run it.

​
What a run actually does

Each run gets a throwaway database: empty except for your app migrations, isolated from live project data. Optional seed code on the eval set runs once at the start of the run and can insert fixture rows. Cases in that run share the same throwaway database, so later cases can see rows earlier cases wrote. Ask Building to clear fixtures at the start of seed code if you want a clean slate before inserting rows. Notifications, Signals, invites, and hook notify are not delivered. The run records those outbound calls on the result (kind, name, and a short snapshot of arguments) so you can inspect what would have been sent. Open the run detail and expand a case to see Outbound calls (not sent). This lock does not mock outbound HTTP or web search. Preview as still sends those for real. Trying a page from Building uses the same lock (those outbound calls are not sent and are not stored on an eval run).

​
Limits (be honest with yourself)

  • Model variability: the same case can flake if the prompt is vague. Prefer concrete user messages and clear success criteria.
  • Not a full QA suite: pages, schedules, and inbound webhooks need Preview, Executions, and manual checks too.
  • Credits: runs consume AI usage. Keep cases focused; do not mirror your entire chat history.