Why a Stronger LLM Isn't Enough to Orchestrate a Reliable Conversational Agent

The gist

• On well-defined procedural tasks, a single model given the full procedure in context can genuinely outperform an explicit LangGraph-style orchestrator: a 2026 study measures exactly that.

• But that improvement is about average execution quality, not about guaranteeing that a specific case, already hit once in production, will never happen again.

• In Gamaro's own engineering history, every guardrail that matters today (anti-repetition, honoring an explicit refusal, business-rule consistency, GDPR consent) was added in code, after a real incident, precisely because a prompt instruction alone had proven bypassable.

• The moat isn't the model's raw capability. It's the accumulation of deterministic guardrails, individually trivial, collectively hard to reconstruct on short notice.

• For a commercial qualification agent, an uncovered edge case isn't a minor stumble: it's a lost qualified lead or a GDPR exposure, not just an awkward reply.

The bet on a stronger model

The idea has obvious appeal: if the underlying model gets good enough, why keep maintaining a state machine, hundreds of lines of guardrails, and a hand-built orchestration engine? Just put the whole procedure in the system prompt, wire up a few MCP servers for data access, and let the model steer itself turn after turn.

That bet isn't wishful thinking. A study published in May 2026 by Dennis, Diamond, Patil, Shabahang, and Guo compares, using the same model placed in both configurations, in-context execution against explicit orchestration frameworks like LangGraph, across procedural tasks (travel booking, technical support, insurance claims).[1] The measured result: the in-context approach scores 4.53 to 5.00 out of 5 on quality, versus 4.17 to 4.84 for explicit orchestration, with a markedly lower failure rate (0.5% to 11.5% versus 9% to 24%): the sharpest gap is on technical support, where the in-context approach drops to a 0.5% failure rate against 9% for the orchestrator. The authors are explicit about scope: their conclusion holds for well-defined procedural tasks, not beyond.

That's a serious result. The paper actually measures more than an average: it also reports full, domain-by-domain failure rates. But what it doesn't measure is something more precise than "the average": the guaranteed non-recurrence of an incident class already identified and explicitly forbidden by a guardrail. That's the property a commercial qualification agent has to prove.

What an average doesn't guarantee

An average quality score says this: across a large number of conversations, the model follows the procedure correctly most of the time. It says nothing about one particular subset of conversations: the ones where a user unexpectedly repeats a question, where two business rules contradict each other depending on which city is involved, where a prospect explicitly declines to share their contact details. These aren't hypothetical: in Gamaro's own history, they are real incidents, each one discovered in production and then fixed with a dedicated constraint.

This is where the distinction between a workflow and an agent, proposed by Anthropic in late 2024, stops being theoretical: "Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents, on the other hand, are systems where LLMs dynamically direct their own processes."[2] A full-fledged agent gets better as the model gets better. A workflow encodes something else: Anthropic itself frames it as "predictability and consistency" for well-defined tasks, not a raw capability score, and predictability doesn't follow from an average, however high.

One nuance is owed to Dennis et al. here, to stay fair to their result: their protocol pits a full procedure held in context against a LangGraph graph that fragments everything into routed nodes. That isn't exactly the architecture defended here. This isn't an argument for forcing every conversation through an external graph: the model keeps autonomy over whatever can stay probabilistic, and only specific invariants, the ones whose violation has already shown a real cost, get hardened into code. The paper makes a case against blanket orchestration; Gamaro's own history makes a case for selective orchestration.

A code constraint beats a prompt instruction

The first lesson from Gamaro's engineering history fits in one sentence: when a prompt instruction alone controls a behavior, the model can work around it again, even a model that otherwise follows that same instruction the vast majority of the time. This bypass was measured directly, with a dedicated metric (crush_rate, the rate at which the orchestrator lets an off-topic digression through despite an active conversation objective). Disabling a tool through instruction alone (forceToolUse: false) remained bypassable: the model would call the tool again regardless of the instruction. The fix that held was a hard lever: literally removing the tool from the set of functions available for that turn, a constraint the model cannot override because it simply no longer exists in its choice space.

That same lesson resurfaced a few weeks later, while evaluating a candidate model for a provider switch. Under that specific model, crush_rate froze at 1.00: proof that prompt-only arbitration failed systematically, reproducibly across multiple runs and temperatures. This time the fix wasn't a new hard lever (one already existed); it was the upstream classifier responsible for triggering that lever, whose recall was measured at 5/8 on the diagnostic corpus. Once its labels were enriched and recall brought to 8/8, the metric returned to its expected level. Both episodes tell the same lesson from two angles: arbitration left entirely to the model's judgment is, by construction, model-dependent. A behavior that holds with one LLM can silently stop holding with the next, with no code change at all.

One guardrail per incident, never anticipated in the abstract

The second lesson is more unsettling for anyone hoping to one day finish the guardrail list: the deterministic guardrails that matter most in Gamaro's history today were added after a specific incident, rarely anticipated in the abstract.

A diagnostic session, for instance, uncovered an agent repeating a question already asked two turns earlier, word for word, despite a different answer from the user: a bounded pattern, not an infinite loop, but a genuine repetition issue whose real fix came from an architectural rework of the orchestrator rather than a local patch. A separate, seemingly unrelated incident froze a multi-city journey for hours: zero results shown despite dozens of internal calls, a genuine deadlock between two tool-activation conditions that mutually excluded each other in one specific configuration, resolved not by a quick fix but by a rework of the location data model. A third case surfaced prompt labels that no longer matched the values a tool actually accepted, fixed both in the prompt and by a dedicated lint rule to prevent recurrence automatically.

None of these three fixes was guessable in advance from a general spec for "a good conversational agent." Each encodes a lesson pulled from one real, specific case. A competitor starting today with the best available model and a standard set of MCP servers inherits none of these three lessons: they will likely have to experience most of these incidents the hard way, in their own production system, with their own users as the detectors.

Explicit refusal as an edge case

One last case shows why this isn't only about conversation quality, but also about compliance. An explicit refusal to share contact details, expressed in natural language rather than by clicking a dedicated button, wasn't always honored reliably: the exact-match logic used until then stopped detecting the refusal the moment it was phrased differently.

The fix took the form of a dedicated classifier, triggered as a fallback whenever the deterministic quick-reply wasn't used. Its confidence threshold was deliberately set high (0.85, versus 0.70 for a less critical sibling classifier), on an explicit business trade-off: the state it triggers is terminal, so a false positive silently costs a valid lead. Measured on a small, dedicated internal evaluation corpus (12 cases, 5 of them refusals): precision of 1.00, recall of 0.60, with zero false positives observed. The honest trade-off in that choice: some free-text refusals slip past this specific filter, a point that calls for ongoing vigilance, not a settled conclusion about the compliance of the mechanism as a whole.

Consent as a state, not a judgment call

The same principle applies to GDPR compliance, with a nuance worth stating precisely rather than claiming broadly. Nothing makes an LLM-driven conversation automatically compliant just because it contains the right words: a model can very well say it logged a consent without that fact being structurally verifiable afterward.

Here too, the story starts with a specific incident: asking for consent used to be entirely delegated to a prompt instruction, and one day, the model closed the conversation without ever asking, even though the corresponding tool remained available. The fix wasn't to rewrite the instruction more firmly. It became a systematic server-side rewrite that turns any conversation exit into an explicit proposal the moment a dedicated state of the conversational state machine opens (email collected, conversation objective reached, consent not yet requested). Concretely, two deterministic quick-replies are always shown at that point, whether or not the model mentions them in its own reply.

One honest nuance is due here: displaying the question is deterministic and guaranteed, but recording the answer still goes through a tool call the model triggers on the next turn. The difference from a plain model judgment call is that this particular turn no longer involves free interpretation: the click can only take one of two fixed values, not a free-form phrase to interpret. This isn't proof of full regulatory compliance: that would be a claim nothing here supports. It's a more modest argument: making the question itself impossible to silently skip already, verifiably, reduces the number of steps where the model's interpretation can fail.

Software capital as the only defensible moat

None of this refutes the study cited earlier. It's right on its own turf: for well-known, tightly bounded procedures, a sufficiently capable model steering directly can outperform a poorly designed explicit orchestration. But a commercial qualification agent isn't judged on its average quality: it's judged on the worst case a real user eventually triggers, one day, in a way nobody anticipated. That's precisely the axis the average doesn't measure, and precisely the axis where an architecture that accumulates deterministic guardrails, incident after incident, builds a lead no model swap catches up to overnight.

The competitive moat, then, isn't the model: anyone can call the same API tomorrow. It's the history of incidents already crossed and already encoded in code, plus the structure that makes every guardrail inspectable after the fact rather than merely hoped for. A thesis already argued from the infrastructure and inference-cost side in Sovereignty and Inference: Local Models.[3] This one is its counterpart on the conversational-orchestration side.

[1] Dennis, S., Diamond, M., Patil, R., Shabahang, K., Guo, H. (2026). In-Context Prompting Obsoletes Agent Orchestration for Procedural Tasks. arXiv:2604.27891.

[2] Anthropic (December 2024). Building Effective Agents.

[3] Gamaro (2026). Souveraineté et inférence : modèles locaux (French original).