The Agent Development Lifecycle
One system for your agent’s whole life, from spec to production and back.
Your agent starts as a signed spec, not a guess.
Build · before production
Your agent starts as a signed spec, not a guess.
Mutagent reads your goal and your uploaded cases, then produces a full agent spec: flow, tools, a suggested architecture and eval criteria, signed off before a line of code.
Plan Agent
Review Agent Spec
Generate in any Framework
Spec Agent
What outcome should this agent own?
Resolve refund + reply requests, end to end.
Great. Upload your cases, SOPs, policies and traces.
Attach sources historic dataSOPspoliciestraces
Working1,284 cases
- Analyzing historic data
- Organizing context
- Reading traces
Mutagent · produced from your dataBrief
Resolve customer refund + reply requests, end to end.
Context
historic dataSOPspoliciestraces
Flow
- classify intent
- fetch order + policy
- check refund eligibility
- draft customer reply
- route to human approval
Goal metric
reply approval → resolution rate
Triggers
support inboxSlackwebhook
Tools
CRMpayments
Guardrails
refund cap $500PII redactionhuman approval
Data access
orderspaymentspolicy docs
Suggested pattern agentic workflow, generated in LangChain
Signed off DSBK
Signed
Agent framework
LangChaingeneratedLangGraphCrewAIMastra
Eval framework
BraintrustwiredPromptfooDeepEval
Tracing
LangfusewiredLangSmithArizeHelicone
- Produced from your dataYour chat plus uploaded cases, SOPs, policies and traces become one signed spec.
- Eval criteria baked inJudged on resolution rate, escalation accuracy, tone, policy compliance and refund correctness.
- Suggested, then generatedMutagent proposes the node flow, then scaffolds it into Claude, Codex, LangGraph or Mastra.
Evaluate · step 1 · derive
Derive the scenarios and evals your agent is judged on, from your real cases.
Mutagent samples a subset of your historic cases, clusters them into named scenarios, extracts each scenario and its evals for you to review, and assembles it all into one reusable eval framework.
Analyze Historic Data
Derive Datasets
Review Baseline Eval Framework
Scanning + clustering
refund dispute ×12
wrong order ×11
late delivery ×10
policy question ×9
escalation ×8
50 of 1,284 sampled
Scenarios
nameexpected outcome
refund disputerefund $40, 3-5 days
wrong orderreship, no charge
late deliveryescalate to human
policy questionquote policy, no refund
escalationroute to specialist
Evals
nametype
refund correctnesscode-based
resolutionLLM judge
escalation accuracycode-based
toneLLM judge
policy complianceLLM judge
Eval framework assembling
Dataset50 cases
Scenarios
refund disputewrong orderlate deliverypolicy questionescalation
Evals
resolutionescalationtonepolicyrefund
- Clustered into scenariosMutagent samples your real cases and clusters them into the scenarios your agent must handle.
- Scenarios and evals you reviewEach scenario comes with an expected output and the evals that grade it, laid out for you to approve.
- One eval frameworkDataset, scenarios, evals and metrics, assembled into a single reusable framework.
Evaluate · step 2
Run the evals, get your approval, then calibrate the judge to your expert.
Mutagent takes the eval framework, runs the evals and annotates each verdict on the trace for you to approve, then calibrates the judge to match your approved labels and locks it into a versioned eval suite.
Load Eval Framework
Rate Traces + Dataset
Calibrate the Judge
Eval framework draft
Dataset50 cases
Scenariosexpected outputs
Evals5-metric rubric
Trace #312 refund disputeyour rating
UserWhere’s my $40 refund?
AgentYour $40 refund is on its way, 3-5 business days.
Your verdicts
refund correctness
resolution
tone
Annotates the dataset50 cases
Judge vs your rating
case Human Judge
refund disputepass··
wrong orderpass··
late deliveryfail··
policy questionpass··
escalationfail··
agreement0%
Eval suite v1 · locked
- Evals annotated on the traceEach eval runs and its verdict is marked on the trace, in context.
- You approve the labelsThe ground truth is yours: approve or correct the labels before anything locks.
- Judge calibrated, then lockedThe judge is tuned to match your approved labels, then locked into a versioned eval suite.
Deploy · the release gate
The scorecard ships it. Not hope, not you.
Every change replays 100 cases through the eval suite before it merges. The scorecard decides, versions are pinned, and it all runs inside the pipeline you already have.
Open a Pull Request
Run the Eval Suite
Gate the Merge
Pull request
Merged
Update reply-agent prompt
fix/reply-agent → main
Eval suite
0 / 100 passed resolution rate
escalation accuracy
tone
policy compliance
refund correctness
Release gate
Checks running
- Regression gateBlocks the merge automatically if any of the five metrics regress. No shipping on hope.
- Versions pinnedEvery prompt, model, and config version tracked, so you can roll back.
- Runs in your CILands as a check in your existing pipeline, not another dashboard.
Monitor · in production
Catch the slip in production, before your users do.
Your online eval suite scores every live trace as it passes. Traces that fail an eval turn up flagged, grouped by which eval broke, and the moment they cross your threshold, Mutagent raises the incident.
Score Live Traces
Group Failures by Eval
Raise the Incident
prod · live
Online eval suiterunning
resolutionescalationtonepolicyrefund
scoring every trace · from Evaluate
eval gate
Failing traces
refund correctness
tone
policy compliance
Alert
Mutagent#incidentsnow
Evals failing above threshold
refund correctness, tone and policy · 12 traces
Automatically starting Mutagent diagnosis
- Your online eval suiteThe evals you built in Evaluate score every production trace as it passes the gate.
- Failing, grouped by evalTraces turn up flagged and grouped by which eval broke, so you see the pattern.
- Threshold alertsCross the threshold and the incident routes to Slack with the failing traces attached.
Diagnose · in production
Every failure you fix, you fix for good.
Mutagent scans every failing trace, clusters them into named root causes, then works cluster by cluster, walking each trace to the failing span and down a 5-whys chain to the true root cause. The fix ships as a PR and lands in your eval suite as a permanent check.
Scan Failing Traces
Find the Root Cause
Grow the Eval Suite
Scanning
Wrong refund amount ×18
Missed escalation ×9
Off-policy reply ×6
Wrong refund amount 18 traces
Trace analysis
#8842
#8817
#8756
+15
Root cause analysis
the refund amount didn’t match the order
because
it was sent without being checked
because
the flow has no validation step
Root causeMissing validation
Proposed fixguard the refund amount Proposed evalamount-validation check
Fix opened · PR #142
Eval suite
resolution rate
escalation accuracy
policy compliance
refund correctness
amount-validation check Proposed
- Scan & clusterEvery failing trace scanned, then grouped into a short list of named root causes.
- Walk, then 5 whysMutagent walks each trace to the failing span, then a 5-whys chain to the true root cause.
- Every fix becomes an evalThe fix opens as a PR and lands in the suite as a permanent check.