Showcase: In-Car Voice Assistant, 0.39 to 0.61 by Deleting Half the Agent

0.388. That was the first working version of our in-car voice assistant, CARlo V1, scored on CAR-bench’s Pass^3 metric across a hundred and twenty-nine tasks. V0, the bare model with no scaffolding at all, scored 0.566. The first attempt at making it better made it worse.

Then Helix went through the failing traces, found that the scaffolding itself was doing the damage, and rebuilt the agent around a handful of guardrails instead. That’s V2. It scored 0.605. Above V0, on the full task set, not a lucky sample. Our part was the goal, the first playbook, and saying yes.

What CARlo has to handle

CARlo sits behind an in-car voice interface. A request comes in, climate, navigation, the sunroof, and the agent has three ways to fail that have nothing to do with understanding the sentence:

• It can hallucinate a capability that isn’t there. CAR-bench deliberately removes tools mid-task to test this. A cheap model under pressure will confidently promise something it can no longer do.
• It can mishandle ambiguity in either direction. Guess when it should ask, or ask when the context already answered the question. Both lose points.
• It can break a policy. Check the weather before opening the sunroof. Open the sunshade before the sunroof. Small ordering rules that a fast model skips when it’s busy sounding helpful.

The backbone is gemini-3.5-flash, and we picked it to have company. On the challenge’s Track 1 leaderboard, the two entries right behind the winner run on it, so a Flash harness has a known ceiling on this benchmark and known peers to be measured against. It’s also one of the cheapest models on that board, which matters because Pass^3, the headline metric, wants the same task solved three times out of three, and that is exactly the consistency a cheap model loses under pressure. The bet was that a harness could buy the consistency back without switching to a frontier model.

The first version, and why it lost to doing nothing

The first version wrapped the model in a full pipeline: a feasibility gate, an ambiguity gate, a policy pre-check, an argument self-check, a risk-triggered verification step, all running before the model was allowed to answer. It sounded like the right design. That was V1. It scored 0.388, worse than V0, the bare model with none of it.

Instead of guessing, Helix’s diagnose stage went through the failing trials. It read the traces and named the mechanism: the gates added to stop the model from hallucinating and from acting on bad assumptions were themselves in the way, refusing requests the model could have handled and hesitating exactly where V0 would have answered. We had built scaffolding to fix a reliability problem, and the scaffolding was the new reliability problem.

What Helix cut, and what it kept

Helix removed everything its diagnosis had convicted: the standalone feasibility gate, the ambiguity gate, the always-on verification step, the policy pre-check as a separate stage. What stayed: schema validation, a small retry budget for malformed output, a check that grounds every capability claim against the tool inventory the model was given, and one added lever, thinking effort turned from off to high.

It tested that on a thirty-task sample first. V2 beat V0 there, 0.667 against 0.567, and beat V1 by a wide margin. Thirty tasks is a sample, so it ran the full hundred and twenty-nine task train split before believing it.

Pass^3	V0 (bare model)	V1 (six gates)	V2 (minimal)
base	0.720	0.44	0.700
hallucination	0.458	0.417	0.542
disambiguation	0.484	0.26	0.548
overall	0.566	0.388	0.605

The sample prediction held at full scale, in direction and roughly in size. Ahead on hallucination and disambiguation, about tied on base, clear of V0 overall. The move from 0.388 to 0.605 came from removing code, not adding it, plus one config flip on how hard the model thinks before it answers.

What this run is, and what it isn’t

A method, shown once end to end: build fast, measure against your own V0, find out which piece of the design is hurting you, cut it, and check again at full scale before believing the win. The agent went from a design that looked reasonable and lost to doing nothing, to a design simpler than the one it started with that beats the bare model it’s built on, in one diagnose-and-optimize cycle, with Helix doing all of the work.

CARlo is also one data point in a larger question, whether Helix builds a better agent than doing the same work by hand in a coding harness, and for what cost. That question has its own post.

The challenge had already closed by the time we ran this, so there is no leaderboard entry. Everything is traceable instead: spec, evaluator verdicts, traces, and the full pipeline at github.com/mutagent-io/examples/tree/main/showcase/carlo.