Evaluation Planner
Turn an architecture choice into a measurable validation protocol.
Evaluation Planner
Set the evaluation context
Your starting protocol
Agent evaluation protocol
Use these as a starting protocol. Do not compare scores across benchmark suites without matching task, version, and evaluation conditions.
Measure
- Task completion on a held-out task set
- Unauthorized or unsafe tool attempts
- Human interventions per successful task
- Latency and cost per completed task
Test failure paths
- Ambiguous instructions
- Missing or stale tool data
- Permission boundary violations
- Interrupted or failed tool calls
Launch criteria
- Define a representative held-out test set
- Set a target success rate before implementation
- Measure cost and latency alongside quality
- Compare one alternative under the same conditions