Test an agent with evals

Use evals to check whether an agent still meets your expectations after changing instructions, files or tools. Each case contains a task briefing and criteria for a good result.

Before you begin

Owners and dashboard editors manage cases and run suites. Viewers can read results. Each suite consumes account Sparks; it tests the current agent configuration.

Save a case

1. Open Manage → Activity and select a good completed run.
2. Select Save as eval. From Chat, use Save last exchange as eval case instead.
3. Review Name and Briefing so the task can run independently.
4. Write Expectation with required sources, output checks and actions the agent must avoid.
5. Select Save case.

To author a case without a past run, open Manage → Evals and select New case. A source execution is optional; when provided, it must belong to the same agent.

Use durable criteria, such as required columns or “must not send email”. Avoid exact dates or figures that change between runs.

Prepare file-based cases

1. Find representative spreadsheets, CSVs, PDFs or other input files in Workspace, or upload examples.
2. Keep independent copies of inputs and reference outputs in agents/<agentId>/eval/<case-slug>/.
3. Put the verified copy paths in the briefing and expected checks in the expectation.
4. Treat replacement of a reference output as an explicit baseline change.

When the dashboard Assistant prepares these copies, complete its file-copy approvals before it uses their paths in a case. Without representative examples, treat the configuration as provisional.

Run and compare

1. Open Manage → Evals.
2. Select Run evals. Keep at most 10 cases active for the dashboard run; archive others or ask the Assistant to run a subset.
3. Read each pass/fail verdict and the judge's explanation.
4. Open failed runs to inspect the audit and identify missing data, configuration errors or runtime failures.
5. Make a targeted correction and rerun the affected cases.

A good run becomes a saved case. After a change, the suite repeats its briefing and compares the evidence with your expectation.

The judge examines outputs and simulated actions, including file structure and recipients. A skipped or unavailable test leaves behavior unverified.

The dashboard Assistant can create, edit, archive or delete cases and run suites without an approval prompt. Other operations keep their own approval requirements. Archive cases to exclude them while retaining history.

Before running evals, the Assistant reviews cases against the agent's current instructions and relevant skills, updates outdated briefings or expectations and checks required files. Valid regression criteria and reference outputs remain the baseline; they are not relaxed just to obtain a pass.

After configuring an agent, the Assistant may run relevant cases once and report the results and Sparks spent. It does not automatically repeat paid runs after a failure or an approved fix.

If you explicitly ask it to ensure the eval passes, it may fix root causes and rerun affected cases, up to three suite executions per agent for that request, including earlier attempts. It stops on success, a blocker, no meaningful progress or the limit and reports what remains. You can request a lower budget, another run or no testing.

Related

• Agent instructions
• Workspace access
• Usage and costs