Evaluator Types

An evaluator is { name, type, config }, created on /evaluators and attached to one or more evaluations. Six types are available — five deterministic/rule-based, one LLM-as-judge. All of them grade a single dataset row and return Pass, Fail, or Error (with an optional 0-1 score and reasoning).

Deterministic checks (text/numeric/JSON/similarity/trajectory) don't call a model, so they're free, instant, and 100% reproducible — prefer them whenever the thing you're checking has a clear right answer. Reach for the LLM judge only for genuinely open-ended quality dimensions (helpfulness, tone, correctness of a free-form answer).

Text match

Checks the resolved output string against a value.

{
  "name": "Mentions refund policy",
  "type": "text",
  "config": {
    "operator": "ContainsAny",
    "value": "refund, return policy"
  }
}

Operator	Passes when
StartsWith / NotStartsWith	The output starts / doesn't start with value.
ContainsAny	The output contains any of value's comma-separated terms.
ContainsAll	The output contains all of value's comma-separated terms.
DoesNotContainAny	The output contains none of value's terms.
DoesNotContainAll	The output is missing at least one of value's terms.

Matching is case-insensitive.

Numeric

Compares a measured metric against a threshold — for cost/latency/token budgets as much as correctness.

{
  "name": "Under 2s",
  "type": "numeric",
  "config": {
    "measure": "latencyMs",
    "operator": "lt",
    "value": 2000
  }
}

measure is one of responseLength, latencyMs, promptTokens, completionTokens, totalTokens, costUsd. operator is one of equals, notEquals, gt, lt, gte, lte.

JSON

Checks whether the output parses as JSON — useful when a step is supposed to emit structured data.

{
  "name": "Valid JSON output",
  "type": "json",
  "config": { "operator": "IsValidJSON" }
}

operator is IsValidJSON or IsNotValidJSON.

Similarity

Scores how close the output is to an expected answer using Dice-coefficient bigram overlap — a cheap, deterministic stand-in for "close enough" grading on open-ended text, without the cost or nondeterminism of an LLM call.

{
  "name": "Matches template",
  "type": "similarity",
  "config": {
    "value": "Your refund has been approved and will arrive in 3 business days.",
    "threshold": 0.85
  }
}

threshold defaults to 0.7 if omitted. The result carries a score (the raw 0-1 similarity), so you can trend "how close" over time even on rows that already pass.

Tool trajectory

Grades the path an agent or orchestrator step took — which tools it called, and in what order — instead of its final text. This reads the same tool_uses data already shown on the execution trace view: agent steps expose it directly, orchestrator steps expose it per role inside spans, and both are read automatically.

{
  "name": "Escalates via Slack",
  "type": "trajectory",
  "config": {
    "operator": "ContainsAll",
    "value": "search, slack-send",
    "step_id": "classify"
  }
}

Operator	Checks
ContainsAll	All of value's comma-separated tool names were called.
ContainsAny	At least one was called.
DoesNotContainAny	None of them were called.
ExactSequence	Tools were called in exactly this order — requires step_id, since call order across multiple steps isn't well-defined.
MaxToolCalls	At most value tool calls were made in total (an efficiency check — is the agent taking an unnecessarily roundabout path?).

step_id is optional for every operator except ExactSequence: leave it blank to check tool calls across every agent/orchestrator step in the workflow, or set it to grade one step specifically.

LLM judge

Uses a model to grade the output as Pass/Fail with a 0-1 score and a short written reasoning — for quality dimensions too subjective for a rule.

{
  "name": "Helpfulness",
  "type": "llm",
  "config": {
    "provider": "anthropic",
    "model": "claude-sonnet-4-6",
    "api_key": "{{ secret.ANTHROPIC_KEY }}",
    "prompt": "You are evaluating an AI assistant response for helpfulness.\n\nUser question: {{input}}\nAssistant response: {{actualOutput}}\n\nIs this response helpful?"
  }
}

The prompt template has access to {{input}}, {{expectedOutput}}, and {{actualOutput}}; leave prompt blank to use a sensible default ("is the actual output correct and acceptable given the input and expected output"). Whatever prompt is used, the judge is always instructed to respond with a result, a score, and reasoning — if a custom prompt's own instructions don't ask for a score, one is derived from the verdict (1 for Pass, 0 for Fail) so the field is never missing.

The Evaluators UI ships six ready-made prompt templates so you don't have to write a judge prompt from scratch: Correctness, Relevance, Conciseness, Helpfulness, Groundedness, and Safety (checking for harmful content, PII leakage, and jailbreak compliance). Pick one as a starting point and adjust it — they're plain text, not a separate mechanism.

Next

• Overview — datasets, evaluations, versions, and comparing runs.
• Promotion gates — require an evaluation to pass before a workflow reaches staging/prod.