Evaluations & Testing

Evaluations answer "did this workflow (or this change to it) actually get better or worse?" — by running a workflow against a fixed set of test cases and grading every output automatically. This is distinct from session feedback, which captures what real end users thought of a real conversation; evaluations are for regression-testing a workflow before it reaches users, against inputs you control.

Unlike tools that grade a sampled trace or a mocked call, an OrcFlows evaluation executes the real workflow through Temporal, once per dataset row — the same durable execution engine used in production, with real branching and real AI reasoning. Nothing about the row's execution is faked; only what happens after it runs (the grading) is configurable.

The building blocks

Concept	What it is	Where
Dataset	A reusable list of test cases — each row is an input (trigger payload) and an optional expected_output.	/datasets
Evaluator	A reusable check that grades one output — rule-based (text/numeric/JSON/similarity/trajectory) or LLM-as-judge.	/evaluators
Evaluation	Ties one workflow + one dataset + a list of evaluators + an output expression together, and runs them.	/evaluations
Run	The result of grading one dataset row within one evaluation — status, latency, metrics, and every evaluator's verdict.	inside an evaluation's detail page

Evaluators and datasets are both standalone and reusable: the same "mentions the refund policy" text evaluator can grade five different evaluations across different workflows, and the same dataset of support tickets can be reused as your product changes.

Creating an evaluation

1. Build a dataset of test cases (or capture real ones — see below).
2. Create one or more evaluators — the checks that will grade each row's output.
3. On /evaluations, pick a workflow, a dataset, the evaluators to apply, and an output expression — the path into the workflow's step outputs that holds the gradable answer, e.g. {{steps.respond.text}}.
4. Creating an evaluation runs it immediately: one Temporal workflow execution per dataset row, each with a 5-minute timeout so one hung row can't stall the whole run.

Every row gets:

• actual_output — the resolved output expression.
• metrics — responseLength, latencyMs, promptTokens, completionTokens, totalTokens, costUsd (summed across every step's usage, when the underlying nodes report it).
• evaluator_results — one verdict (Pass / Fail / Error) per evaluator, each with optional reasoning and, for LLM-judge and similarity evaluators, a continuous score between 0 and 1.

A row's overall result is Error if the workflow itself errored or any evaluator errored, Fail if any evaluator failed, otherwise Pass — and the evaluation's aggregate pass_pct is the share of rows that passed overall.

Datasets

A dataset is just rows of { input, expected_output }. Build them by hand on /datasets, or:

Turning a real execution into a test case

Rather than hand-typing rows, open any past execution's detail page and use Save as test case: it copies that execution's actual trigger payload into a dataset row (creating a new dataset inline if you want), with an editable expected_output. This is the fast path for turning a bug report or a flagged bad response into a permanent regression test — fix the workflow, then re-run the evaluation to confirm the fix and that nothing else broke.

Versions and comparing runs

Running an evaluation again — via Run again on the evaluation detail page, or by creating a new evaluation with the same name — creates a new version rather than overwriting the old result. Every evaluation detail page shows v{n} of {total} and a Versions drawer listing every past run with its pass rate and date.

From any version, use Compare against this version to diff its per-row results against another version's, matched by row position (valid as long as the dataset's row order hasn't changed). The comparison banner reports how many rows regressed (passed in the baseline, no longer pass) and how many improved (the reverse), and each changed row gets a small up/down indicator next to its status icon — so you can see exactly which test cases a change broke, not just that the aggregate pass rate moved.

Aggregate metrics

The evaluation detail page's stat row shows pass_pct, total runs (pass/fail/error breakdown), avg_latency_ms, total cost and tokens, and — when at least one row produced a continuous score (LLM judge or similarity evaluators) — avg_score. Per-row latency and cost are also charted so an outlier row is easy to spot.

Next

• Evaluator types — the full reference for every evaluator type and its config.
• Promotion gates — block a workflow from reaching staging/prod unless an evaluation passes.