Capability
Evaluation
Score agent output against expected results before release, and keep scoring it while the process runs in production.
Evaluation
Evaluation in ÆTERION BRAIAN turns agent quality into a measurement. Test suites hold representative cases with expected outcomes; every candidate change — a new prompt, model or tool — is scored against them. Release gates block a change that regresses, and the same checks continue to sample production runs, so quality drift is visible before a business owner reports it.
What gets measured?
- Task success against a labelled expected outcome.
- Faithfulness to source documents, for retrieval-based steps.
- Policy compliance: did the run stay inside its permissions and limits.
- Cost and latency per run, per process and per model version.
- Human override rate — how often a reviewer changes the proposed action.
Release gates
A change reaches production only when the suite passes the thresholds you set. The comparison is version to version, so you can see exactly which cases improved and which broke.
From incident to test case
- Capture
- A wrong output in production can be promoted into the test suite with one action, together with its inputs.
- Fix
- The candidate change is scored against the whole suite, not just the failing case.
- Prove
- The release record shows which version fixed the case and who approved the promotion.
FAQ
Do we need labelled data to start?
A small set is enough. Most teams start with twenty to fifty real cases per process and grow the suite from production incidents.
Who owns the thresholds?
The process owner. Thresholds are policy, set per process and versioned like every other rule.
Measure before you deploy
We build a first evaluation suite on one of your processes during the demo.