Eval

Evaluator kinds and their configuration, claims and behaviour bindings, eval sets, live trace scoring, and eval runs — with the endpoints behind them.

Eval is where you define what “good” means for a capability and measure it — in batch runs that compare models and prompts side by side, and continuously on production traces as they arrive. The same evaluators serve both.

The Evaluations area of the Console has three surfaces: Runs, Eval sets, and the Eval library.

Evaluators

Generated evaluations preserve provider finish reasons. A response stopped by its output token limit remains inspectable, but the sample is marked degraded and its judgments are skipped rather than assigned quality zeros. Degraded samples are excluded from trusted score aggregates and counted in run health. Check this coverage before comparing model quality: a high average over the few complete samples is not a successful overall run. Training-linked evaluations with degraded evidence report failure and no headline comparison score.

Rubric judges default to generative grading. Qualified bounded checklist, claim-support and per-turn decisions can opt into Jev with generative fallback. Authoring and holistic reasoning remain generative. Evaluator policies and score provenance identify which path ran.

An evaluator is one reusable, versioned scorer — editing one writes a new version row, so past runs keep meaning what they meant. Five kinds exist:

LLM judge (llm_judge)

A model grades the response against a rubric you write in plain language. The rubric is compiled into a checklist of atomic, weighted yes/no questions; on a boolean evaluator an item can be a gate that fails the verdict regardless of the rest. Judges return reasoning and per-item sub-scores. The managed Correctness and Faithfulness judges use a proportional mode instead: they enumerate the output’s claims and score the share that holds. Judge responses are cached content-addressed, so re-scoring identical inputs is free.

Deterministic (deterministic)

A fixed rule, no model involved. Checks: exact_match, contains, regex, json_schema_valid, schema_field_conformance, field_present, reference_field_compare, canonical_fields, tool_selection (precision/recall/F1 over called tools), tool_args_match, goal_state_match, card_constraints, latency, cost, and turn_count.

Trajectory (trajectory)

Judges the path the agent took, not just the output. Two modes: match compares the tool-call sequence against a reference with strict, unordered, subset, superset, or subsequence semantics (with per-tool argument-match overrides), or judge hands the trajectory to an LLM.

Agentic (agentic)

An LLM judge for long, multi-step runs; on live traces the summarising cascade keeps the judge under its context budget.

Statistical (statistical)

Scores a whole dataset at once rather than one example at a time: accuracy, precision, recall, f1 (macro/micro/weighted), calibration (expected calibration error with a bootstrap interval), exact_match, bleu, rouge_l, embedding_cosine, chrf, and duplicate_rate, with output_normalize set to strip (default), first_word, last_word, first_line, last_line, or a regex:<pattern> extractor. When the references are labels, the run also reports per-class precision, recall, F1 and a confusion matrix.

Every evaluator declares a scope (final_output, turn, step, trajectory, sample, or dataset), a score type (numeric, categorical, or boolean) with a pass_threshold, an evidence requirement (model_output, harness_artifact, trajectory, or reference), and a surface — model, harness, or any — that says where it may run: a model grader never runs on live trace scoring, a harness grader never runs on a generate variant. If a run cannot supply the required evidence, the evaluator abstains or is marked not-applicable instead of producing a number.

Claims

Every evaluator also carries a claim: what it asserts and at which grain. The claim is what trace scoring uses to compose one execution score out of many verdicts.

Claim type	Asserts	Composes as
conformance	The output has the required shape (a schema, a field, a format)	A scoring leaf whose own passed flips on failure
verification	A fact in the output checks out against evidence	A scoring leaf whose own passed flips on failure
progress	A step of the behaviour was taken	A weighted leaf
quality	The delivered result is good	A weighted leaf
grounding	The deliverable rests on the evidence gathered	A weighted leaf
safety	Nothing harmful or forbidden happened	A cap on the score

The grain is the slice a claim is judged on: unit (one turn or step), trajectory (the path through the behaviour), terminal (the deliverable), or session (the whole conversation). Grains land in three weighted phases — steps, trajectory, output — and no single evaluator vetoes the composite.

Behaviour bindings

An evaluator can be bound to one of the capability’s behaviours with a role: an outcome evaluator judges the behaviour’s deliverable, a step evaluator judges one anchor segment of its route. The Eval library groups evaluators by behaviour (the Console labels it “Task”), shows each behaviour’s coverage (outcome plus every step), and lists the rest under Unassigned. GET /api/behaviours/coverage/?capability={id} returns the same rollup.

Judges are authored behaviour-scoped from the behaviour’s group in the Eval library or through the upsert_evaluator MCP tool: POST /api/evaluators/author/ takes behaviour, behaviour_role (outcome or step), and — for a step judge — the anchor_segment, validated against the behaviour’s contract. GET /api/behaviours/{id}/authoring-context/ grounds the authoring with the behaviour’s latest contract: anchor sequence, steps, tool set, and terminal. A judge whose applicable_roles include trace_scoring must be bound to a behaviour, and eval_set plus eval_set_role attach the new judge to an eval set in the same call. PUT /api/evaluators/{id}/author/ re-authors an existing judge.

Authoring

Author from the Eval library by hand, or describe what you want to test and let Overmind generate the evaluator (POST /api/evaluators/generate-prompt/). The generated prompt template references the agent’s output as {{output.<path>}} and dataset fields as {{reference.<path>}}, and everything stays editable before saving.

overmind sync refreshes the Default eval set of every current capability, in tiers: Tier 0, rule-based checks compiled from the capability’s card (its output contract, fields, and constraints — no model call); Tier 1, generative LLM judges grounded in the card, the prompt, and the behaviour map; and the behaviour outcome and step judges for trace scoring. Tier 1 authors once per capability; when it authors nothing for the generative role, one floor judge fills the role until a later push covers it. Authoring reads the capability’s evidence profile — what its telemetry carries (declared behaviours, explicit units, real tool spans, provenance tags, a declared deliverable) — and refuses a spec whose evidence the traces cannot supply.

The Eval library grouped by behaviour, with each behaviour's outcome and step coverage.

Evaluators are platform-managed templates, project-wide, or scoped to a single capability.

Eval sets

An eval set is a named group of evaluators, optionally mapped to a capability. On Evaluations → Eval sets, choose New eval set, enter a name, and search the whole evaluator library by name, type, or mapped capability. Select multiple evaluators; selected rows show their name, type, and capability and can be removed before creation. Leave capability set to None for a reusable project set.

POST /api/eval-sets/ accepts evaluator_ids and an optional nullable capability; the set and members are created together. MCP provides create_eval_set and the read-only overmind://eval-sets/{eval_set} resource. Neither creation path starts an evaluation or activates the set. Each member receives its applicable roles:

• generative members score outputs produced during eval runs and optimisation runs.
• trace_scoring members run on every production trace, per task execution, writing verdicts and the execution score into Observability. The mechanics are on Trace scoring.

The same evaluator can hold both roles. One set per capability is active (POST /api/eval-sets/{id}/activate/); it is what trace scoring, optimisation runs and training runs default to. POST /api/eval-sets/{id}/members/ adds members.

A set without a capability can be selected for project evaluation and training, but cannot be activated for live trace scoring until a capability is assigned.

Eval runs

Context warnings

Training setup shows advisory context estimates for the model being evaluated and its generative judges. These include prompt, tools and reserved output tokens. Judge preflight uses reference-answer size as a proxy for the answer that has not been generated yet. Unknown model limits are shown as unverified. These are estimates, not exact tokenizer counts; warnings never prevent launch.

The Judge model selector below Eval set starts with the set’s saved judge or Mixed models when its evaluators use different judges. A selection applies to this training run’s generative judges and fallbacks, not the saved set or its Jev policy. The dropdown marks estimated context fit and shows the budget change across all judges for one dataset pass per evaluated model. Undersized options are greyed out but remain selectable. Estimated limit overflows turn the affected judge or benchmark selector orange. Start also turns orange with a warning icon; clicking it opens a short risk summary with Start anyway. Unverified, unavailable and pending checks stay neutral and do not open this warning. Changing the model is optional.

Before launch, context warnings suggest alternatives with verified context and output capacity. Judge suggestions preserve configured model order, exclude the model family under test and require structured-output support. Generation suggestions require tool support when the input declares tools. These checks do not predict model quality. No suggestion changes the selection automatically. A judge change should be applied to both sides of a comparison.

Evaluation cost comparisons use published token rates, all checked rows and the full reserved output for one pass per model or judge. They are not a total-run quote: retries, extra tool turns, caching, hosting and training are excluded. Missing prices remain unavailable. In training setup, training alternatives use the existing dataset ranking and show training-cost and duration estimates for their recommended configurations separately from judge evaluation costs. Benchmark deployment costs are not inferred from the base model’s token prices.

API clients can preview a selected version with POST /api/eval-runs/context-check/ using project, dataset, optional cell, eval_set, capability, variants and a run-specific judge_model. The response includes judge_models with fit and budget comparisons. Training launch accepts the selected eval_judge_model; omitting it preserves the eval set’s per-evaluator models. Each preview variant can supply output_tokens for a planned absolute output reservation; otherwise the registered model parameters or provider-call default apply. The request reads data without launching a run or calling a model. MCP check_evaluation_readiness accepts proposed variants and judge_model and returns advisory context_checks and judge_models; these do not change ready. MCP start_finetune accepts eval_judge_model. Standalone REST creation and MCP run_evaluation accept judge_model, matching the preview. Omit it or pass blank to preserve each evaluator’s saved judge. An explicit choice applies only to that run’s generative judges and fallbacks, including replay judges attached after launch; it does not edit the library, deterministic checks or Jev policy. Judge selection is frozen once created: start a new run to use another judge. Run detail shows the snapshot judges; the MCP run resource exposes judge_model, run_evaluators and the context checks.

Runtime checks use the actual model messages and assembled judge prompt. The judge funnel preserves that prompt instead of clipping it to a fixed character limit. If a judge exhausts its output budget, it can retry once with a larger budget only when published context and output limits permit it. It does not switch judges or shorten the evidence. Permanent context rejection and exhausted grading attempts are technical errors, not quality zeros. Score _judge metadata records attempt budgets, finish reasons, usage, request IDs and costs where the provider supplies them.

Configuration

An eval run is one experiment: one data source × N variants × M evaluators.

• Data source — dataset (the version is used and frozen, so the run stays reproducible as the dataset evolves) or trace_filter, drawing directly from captured traffic. max_items caps how many rows are drawn.
• Variants — the columns of the experiment. A variant’s mode is generate (re-run a model over each row) or existing (grade the recorded output as-is). Each pins a model (a catalogue model, a registered endpoint, or one of your trained models) and optionally a prompt; one variant is the baseline (is_baseline) every other is measured against.
• Evaluators — attached as frozen snapshots at creation, so later library edits cannot change what a completed run measured.

Creation refuses a run whose eval set has no live generative LLM judge, and a run over a closed-form labelled dataset whose eval set carries no comparator against the gold labels. In the Console, Evaluations → New evaluation selects an eval dataset, eval set, model to test, maximum samples and optional run-only judge. Model selectors show the provider logo and model name, matching training setup. Judge options also show estimated context fit and cost comparisons; context warnings remain advisory. Estimates cover the full dataset, while maximum samples limits the run. Eval runs can also be created through the API and the run_evaluation MCP tool. An eval dataset’s cell also offers Run the optimiser and Use in a training job.

# Create (which also launches), follow, and compare a run
curl -X POST -H "X-Api-Key: $OVERMIND_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "project": "<project-uuid>",
    "name": "Triage — prompt v3",
    "data_source": "dataset",
    "dataset": "<dataset-uuid>",
    "variants_input": [
      {"label": "production", "mode": "existing", "is_baseline": true},
      {"label": "prompt v3", "mode": "generate", "model_name": "<model-ref>"}
    ],
    "eval_set": "<eval-set-uuid>"
  }' \
  "https://api.overmindlab.ai/api/eval-runs/"

curl -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/eval-runs/{id}/comparison/"
curl -X POST -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/eval-runs/{id}/run/"     # re-launch: deletes samples and scores first
curl -X POST -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/eval-runs/{id}/cancel/"  # revokes in-flight tasks
curl -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/eval-runs/datasets/"             # datasets that have runs

Execution fans out server-side: sample preparation (generation), then one scoring task per sample × evaluator, then aggregation into the run summary. A run’s status moves pending → running → completed, or ends failed or cancelled.

For recorded conversations, a generated variant can set params.generation_strategy to per_assistant_turn. The model produces one decision for each recorded assistant turn, using the conversation recorded before that turn as context. Tool calls are evaluated as decisions without executing their tools. Each subsequent turn uses the recorded history, keeping its context independent of the candidate’s previous decisions. Such a run attaches a per-turn judge automatically, and params.replay_dimensions names what it grades. This also applies to conversations containing a single recorded assistant turn; a tool-only decision does not require a final answer.

A sample that exceeds the generation worker’s time limit records an error and is excluded from scoring. Scoring starts after sample preparation finishes, so a run can be generating while its score count is still zero.

Reading the report

A finished run: per-evaluator pass rates and mean scores per variant, the score distributions, and cost and latency per model.

The report shows per-variant rollups (mean score, pass rate) per evaluator, the score distribution per evaluator and variant, cost and latency per model — generation cost and eval cost separated — and a per-datapoint table. Opening a datapoint shows each evaluator’s score with its reasoning; the Compare runs tab lays one run’s results next to another’s.

Each dataset row has its own entry in the breakdown, with responses from different model variants grouped under that row. Search or page through the entries, then open one to inspect its input and each available model output. A model with no sample for that row does not display another model’s response.

The datapoint popup shows Input beside Output and evaluation. Select a model to switch both panels together. Generated output is the model’s response, not a label extracted by a statistical evaluator; the Reference answer is shown separately as grading data. Expand Generated messages to inspect recorded assistant and tool messages.

Generated samples capture the initial model-runner input, including the system prompt and tool definitions. This is not the provider’s rendered chat template or every subsequent replay request. A sample without a captured request shows Dataset input from the run’s pinned version. A missing pinned source is shown as unavailable, never reconstructed from a mixed input/output transcript.

The MCP overmind://eval-runs/{eval_run} resource exposes up to five bounded sample inspections with separate input, output, reference, and evaluator reasoning. input_source distinguishes captured requests, pinned dataset rows, and unavailable input; samples_truncated indicates additional samples.

Datapoint drill-down: each evaluator's score with reasoning. A check that could not be applied is not-applicable and does not enter the averages.

• Outcomes are typed. Every score’s outcome is scored, abstained, not_applicable, skipped, or error — and only scored results feed averages and pass rates.
• Samples can be degraded. Each trace is normalised into a canonical transcript before scoring; if it cannot be reconstructed, the sample is marked degraded with a degraded_reason and excluded from aggregates. Failure scores also carry a failure_role — root_cause or propagated — so one upstream failure does not read as five independent ones.

Scores accumulate into a per-capability history (GET /api/evaluators/score-history/) that powers the trend chart on the capability page.

Calibrating judges

Samples accept annotations — human labels with an optional note, written through the annotate_evaluation_sample MCP tool — giving you ground truth to compare judge behaviour against. When a judge and your annotations disagree consistently, tighten the rubric; the compiled checklist shows which criterion is drifting.