How it works
From scenario to actionable evaluation report.
ASHE is AI reliability infrastructure for production systems. It automates scenario generation, execution, judging, and failure diagnosis with evidence-backed reports.
01
Connect your system
Provide the endpoint ASHE will evaluate. This can be a knowledge-backed assistant API or an autonomous agent. Bearer authentication is supported via environment configuration.
02
Define evaluation scenarios
ASHE generates questions from a knowledge base or runs structured tasks from a task set.
03
Configure the run
Choose your target, scenario source, case count, and concurrency from 1 to 8.
04
Start a run
The API returns 202 Accepted and a background worker executes the evaluation asynchronously.
05
ASHE executes scenarios
Each case is sent to your live system. Retrieved documents and execution traces are captured when available.
06
ASHE judges outcomes
Frontier intelligence evaluates correctness, grounding, and task completion. Advanced Reasoning escalates low-confidence cases.
07
ASHE produces the report
Overall score, component breakdown, confidence distribution, and case-level results.
08
Inspect failed cases
Drill into individual cases with evidence, traces, failure classification, and recommendations.
09
Improve and run again
Fix the underlying issue, re-evaluate, and track improvement over time.