Start here
What happens when the agent gets it wrong
Most professional-services teams that build an AI agent focus on what it can do. Far fewer plan for what happens when it gets things wrong. That gap is where trust breaks down, and where an agent goes from useful to dangerous.
This is the practical work of making an agent reliable: grounding its answers in real data, forcing it to calculate rather than guess, and building the audit trails that let you trust what it produces. These are not theoretical safeguards. They are the controls that separate a clever demo from a production tool.
The villain
Confident-sounding nonsense
There is a specific failure mode that causes more damage than obvious errors. It is called hallucination, and it is why many pilots that look promising in a demo quietly get shelved after six weeks in the real world.
A language model does not look up facts. It predicts the most likely next word based on patterns in its training data. When it has strong evidence, it performs well. When it does not — when the question is specific, numerical, or depends on your firm’s data — it guesses. And it delivers that guess with the same confident tone it uses when it is right.
Think of a multiple-choice exam where you have no idea which answer is correct. You pick something, because leaving it blank feels worse. Models do the same thing. The difference is they never visibly hesitate.
Numbers make this worse. Models are good at spotting patterns in text and unreliable at arithmetic, tabular logic, and anything involving unit conversion or multi-step calculation. Asked for a grand total across a dataset, a model may return the right shape of answer — a plausible-looking figure — with the wrong value. The fix is not a smarter model. It is a workflow where the model never does the maths itself.
The three controls
Design choices that make an agent reliable
These are not complex engineering decisions. They are choices you make when building the agent’s prompt and workflow, and each one closes a specific failure mode.
- Ground answers in your data, not the model’s memory. The model’s memory is a statistical average of the internet, not your firm. Give the agent read-only tools — connections to your spreadsheets, CRM, or databases — and instruct it to retrieve before it responds. An agent that pulls live data from a Google Sheet and cites the row it read from is fundamentally more trustworthy than one generating answers from its own weights.
- Provide worked examples so it copies the right behaviour. Showing beats telling. Short, complete examples in the prompt reduce the surface area for the model to improvise and establish a pattern it can recognise and replicate. In testing, the difference is visible: with examples, the model uses valid IDs, calls tools in the right order, and formats output predictably. Without them, it drifts — sometimes subtly, sometimes in ways that break downstream processes.
- Validate before sending, and stop if things do not match. After the agent produces an answer, a second check compares the calculated result to the source data. If they match, the answer goes out. If they do not, the agent stops and asks for clarification. In production this often means two agents working the same problem, each checking the other. When they disagree, neither answer is sent, the discrepancy surfaces, and a human decides.
Match your controls to the stakes. For internal advice — a summary, a status update, a report draft — light checks are enough. For client-facing outputs or anything involving financial figures, add a human review step and keep an audit log. The higher the impact, the more explicit the verification.
Grounding answers in centralised, searchable, AI-accessible sources of truth is the Knowledge layer of the KWA framework — Knowledge, Workflow, Agents. If that layer is not in place, everything built on top of it is unreliable.
The guardrails
Instructions you can implement today
Beyond the three core controls, a handful of specific instructions cut error rates in production. These come straight from how we configure agents for client deployments.
- Refuse to answer when there is no data. If a tool call returns zero rows, the agent should say so and ask the user to narrow the query — not fabricate a response. A single line in the prompt eliminates one of the most common failure modes.
- Limit the surface for mistakes. Cap row counts and date ranges. If a query would return more than 5,000 rows, the agent asks the user to narrow the period first. More data does not make answers more accurate; it makes errors harder to spot.
- Return units explicitly. Working across multiple currencies, the agent should state the unit in the output. Never assume the reader will infer it.
- One tool call per turn. This sounds minor. In practice it significantly reduces compounding errors, because a mistake in the first of several chained calls propagates through the rest. Slowing to one call per step gives you cleaner audit trails and simpler debugging.
- Round only at the final step. Intermediate rounding introduces errors that accumulate. Carry full precision through the calculation and round only in the output.
The paper trail
Audit trails you can actually trust
Reliable answers are not enough on their own. You also need to know, after the fact, what the agent was asked and what it said.
Most workflow automation tools include basic execution logs — a record of what ran, when, and what it produced. These are useful for debugging but may not be retained long enough for compliance. A permanent audit trail — a database or log file capturing every question and answer — is worth the small additional effort, particularly for client-facing or regulated work.
The second mechanism is a test suite: known inputs with expected outputs that you run every time you change the agent’s configuration. As you swap models, update prompts, or adjust tool connections, the suite tells you whether the agent still behaves the way it did when you signed it off. Without one, you are testing changes in production — a fair description of how most teams currently operate.
The shift
From chatbot to reliable data assistant
The move from a chatbot that sounds useful to an agent that is genuinely reliable comes down to a few decisions made at the design stage. Ground the agent in your data. Show it how to behave with worked examples. Force it to calculate with a tool rather than guess. Validate outputs before they leave the system. Log what it does so you can audit it later.
These are the Workflow and Agent layers of the KWA framework working as they should — standardised processes that can be automated, with appropriate human oversight. The Govern step of the 5 Steps for AI Leadership framework exists precisely for this: approved tools, data-handling rules, and a risk register are not bureaucracy. They are what let you scale with confidence rather than hope.
An agent you can trust is one you will actually use — and one your team will use without being asked. Reliability is not a feature you bolt on at the end. It is a set of decisions about where the risk sits and what controls match that risk, made before the first line of the prompt.