Overview
Evaluations answer "did this workflow (or this change to it) actually get better or worse?" — by running a workflow against a fixed set of test cases and grading every output automatically. This is distinct from session feedback, which captures what real end users thought of a real conversation; evaluations are for regression-testing a workflow before it reaches users, against inputs you control.
Evaluator Types
An evaluator is { name, type, config }, created on /evaluators and attached to one or more evaluations. Six types are available — five deterministic/rule-based, one LLM-as-judge. All of them grade a single dataset row and return Pass, Fail, or Error (with an optional 0-1 score and reasoning).
Promotion Gates
Workflows carry an env (dev / staging / prod); POST /api/v1/workflows//promote moves a workflow to a new one, snapshotting the current version first so the promotion is reversible. Promoting to prod also sets status to pending_approval, so an admin/owner must explicitly approve before its webhook accepts real traffic.