What If the AI Gets It Wrong?

AI does not need to know everything

A common misconception is that an AI system becomes more useful if it has access to as much information as possible.

In insurance, the opposite can often be true.

Suppose a voice assistant is being used to answer inbound calls for an insurance broker.

Its job might be to:

• answer the telephone;
• identify the caller;
• understand why they are calling;
• answer straightforward questions about the business;
• collect relevant information;
• route the caller;
• arrange a callback;
• or book an appointment.

It does not necessarily need to know how to interpret every policy wording, determine coverage or advise the caller about what insurance they should buy.

Giving it that broader role could introduce risk without providing much additional operational benefit.

The first guardrail is therefore:

Define the job narrowly and clearly.

Use approved information

Customer-facing AI systems should not necessarily be encouraged to answer questions from unrestricted general knowledge.

Instead, they can be grounded in information approved by the insurance business.

This might include:

• company information;
• product descriptions;
• frequently asked questions;
• opening hours;
• contact information;
• claims notification procedures;
• departmental responsibilities;
• approved explanations of services;
• and defined operational processes.

The organisation determines what information the system is permitted to use.

That creates a significant difference between:

“Answer this question however you think best.”

and:

“Answer this type of question using the information we have approved.”

The second is a much more controlled proposition for many insurance workflows.

Define what the AI must not do

Good guardrails are not only about defining what the system can do.

They also define what it must not do.

For example, depending on the workflow, an AI receptionist might be explicitly prohibited from:

• interpreting policy wording;
• confirming whether something is insured;
• giving a view on whether a claim will be paid;
• advising someone which insurance product they should purchase;
• making an underwriting decision;
• accepting liability;
• agreeing a claims settlement;
• handling a complaint beyond capturing and escalating it;
• or inventing an answer when information is unavailable.

These boundaries should be deliberate.

If the AI encounters one of these situations, the correct behaviour is not to improvise.

It is to stop and escalate.

Escalation is a feature, not a failure

There can be a temptation to judge an AI system by how many interactions it can complete without human involvement.

That is not always the right measure.

In a controlled insurance workflow, escalation can be exactly what the system was designed to do.

Consider a caller who asks:

“Am I definitely covered for this?”

A poorly controlled system might attempt to answer.

A better system might say that it cannot make a coverage determination and arrange for the appropriate insurance professional to assist.

The AI has not failed.

It has followed the workflow correctly.

The same principle could apply to:

• a complaint;
• a distressed caller;
• a vulnerable customer;
• an emergency;
• an unusual claim circumstance;
• an unclear policy question;
• conflicting information;
• or a request outside the system's approved scope.

A good automation should know its limits.

Confidence should not replace verification

Generative AI can sometimes give an incorrect answer in a very convincing way.

That makes verification particularly important.

Critical information can be confirmed before it enters the next stage of a workflow.

For example, a voice assistant collecting an email address might read it back:

“I have that as [email protected]. Is that correct?”

The same approach can be used for:

• names;
• telephone numbers;
• policy references;
• claim references;
• incident dates;
• addresses;
• appointment times;
• or other important information.

This is especially valuable in voice systems where accents, background noise and similar-sounding words can create transcription errors.

A few seconds of confirmation can prevent incorrect information from being carried through the entire process.

Structure the output

Another useful guardrail is to avoid relying solely on unrestricted narrative output.

Imagine an FNOL call.

At the end of the conversation, the AI could simply produce several paragraphs describing what happened.

But it may be much more useful to extract specific fields such as:

• Policyholder:
• Policy reference:
• Date of loss:
• Location:
• Description of incident:
• Third parties involved:
• Injuries reported:
• Emergency services involved:
• Supporting evidence available:
• Urgent escalation:

Structured outputs make it easier to identify missing information and apply further workflow rules.

They can also reduce the amount of interpretation required by the next person or system receiving the information.

Separate conversation from decision-making

One of the most important design principles is separating the ability to have a natural conversation from the authority to make a decision.

Modern voice AI can sound remarkably natural.

That is useful because customers do not want to interact with a rigid telephone menu.

But conversational ability does not mean the AI should automatically have decision-making authority.

An AI system might have an excellent conversation with someone reporting a claim.

It could:

• show empathy;
• ask appropriate questions;
• gather the details;
• clarify unclear answers;
• structure the information;
• and send everything to the claims team.

None of that requires it to decide whether the claim is covered.

The conversational layer and the decision layer can remain separate.

Use deterministic rules where appropriate

Not every part of an AI workflow needs to be controlled by AI.

Traditional software rules can still be extremely useful.

For example:

• If the caller says there is an immediate danger → provide the agreed emergency instruction.
• If the caller wants to make a complaint → escalate to the complaints process.
• If the enquiry is outside business scope → do not attempt to answer.
• If required information is missing → request it before completing the workflow.
• If a meeting has been requested → check the approved calendar process.

These rules can sit around the language model.

The AI handles the conversation.

The surrounding workflow determines what actions are permitted.

This can make the overall system much more predictable.

Limit what tools the AI can use

The same principle applies when an AI system is connected to other applications.

An assistant might be able to:

• transfer a call;
• send a message;
• book a meeting;
• create an enquiry record;
• or trigger an internal workflow.

But it does not follow that it should have unrestricted access to every function within those systems.

Permissions can be deliberately limited.

For example, the assistant may be allowed to create a new enquiry but not delete an existing customer record.

It may be allowed to view available meeting times but not access unrelated calendar information.

It may be allowed to send information to a defined mailbox but not send arbitrary emails to anyone.

The authority given to the AI should match the task.

Design for uncertainty

A particularly important guardrail is telling the system what to do when it does not know.

The safest answer will sometimes be:

“I don't have enough information to answer that.”

or:

“I will need to pass that to a member of the team.”

That is preferable to an answer created simply because the AI has been instructed to always be helpful.

Systems should therefore have explicit behaviour for:

• uncertain answers;
• incomplete information;
• conflicting information;
• questions outside the knowledge base;
• unexpected requests;
• and situations that do not match the normal workflow.

Uncertainty should trigger caution, not creativity.

Test the difficult conversations

Testing an AI system should involve more than checking that the ideal conversation works.

The difficult situations often reveal the weaknesses.

Before deployment, tests might include callers who:

• interrupt;
• change their mind;
• provide contradictory information;
• speak quickly;
• have a strong accent;
• refuse to answer a question;
• ask something completely unrelated;
• demand insurance advice;
• ask whether a claim will be paid;
• complain;
• become upset;
• try to manipulate the system;
• or repeatedly ask the AI to ignore its instructions.

The objective is not just to test whether the AI can answer.

It is to test whether it behaves safely when it should not answer.

Monitor real interactions

Pre-launch testing is important, but it cannot anticipate every real customer conversation.

Once an AI system is live, its performance should continue to be monitored.

Depending on the workflow, this might involve reviewing:

• transcripts;
• structured outputs;
• escalations;
• failed interactions;
• unusual questions;
• incorrect classifications;
• customer feedback;
• and cases where staff had to correct information.

Patterns can then be identified.

Perhaps customers repeatedly ask a question that is missing from the approved knowledge base.

Perhaps a particular phrase is being misunderstood.

Perhaps the escalation rule for one type of enquiry needs to be improved.

Automation should not be treated as a piece of software that is configured once and forgotten.

It is an operational process that can be reviewed and refined.

Keep an audit trail

Where appropriate, it should be possible to understand what happened during an automated interaction.

For example:

• what information was provided;
• what information was captured;
• what workflow was triggered;
• whether the interaction was escalated;
• and what output was passed to the business.

This can be particularly useful when investigating an error.

Without an audit trail, it may be difficult to understand whether the problem occurred because:

the customer provided incorrect information;

the AI misunderstood the customer;

the knowledge base contained incorrect information;

a workflow rule was wrong;

or an integration failed.

Visibility helps organisations improve their systems.

Someone still needs to own the process

Human oversight does not simply mean having someone available to take over a telephone call.

Someone within the organisation should understand and own the automation.

That person or team should know:

• what the system is intended to do;
• where its boundaries sit;
• what information it uses;
• what actions it can take;
• when it escalates;
• how its performance is reviewed;
• and how changes are approved.

Without ownership, an automation can gradually drift away from the business process it was originally designed to support.

What does a controlled insurance AI workflow look like?

A useful way to think about guardrails is as a series of boundaries around the automation.

• 1. Defined purpose — The AI has a specific job rather than an unlimited remit.
• 2. Approved knowledge — It answers defined questions using information authorised by the business.
• 3. Restricted actions — It can only access the systems and functions required for the workflow.
• 4. Verification — Important information is confirmed before being relied upon.
• 5. Structured outputs — Information is captured in a predictable format.
• 6. Escalation — Situations outside the approved scope are passed to a person.
• 7. Monitoring — Real interactions are reviewed and the workflow is improved.
• 8. Accountability — Someone remains responsible for how the system operates.

None of these controls requires the AI to stop being useful.

They make it more useful because the business can understand what it is expected to do.

What if the AI still gets something wrong?

It probably will at some point.

So will people.

The objective of good automation design is not to claim that errors are impossible.

It is to make errors:

• less likely;
• easier to identify;
• less consequential;
• and easier to correct.

That means avoiding workflows where one unchecked AI response can automatically produce a serious outcome.

Instead, important decisions can remain with the people who are qualified and authorised to make them.

Automate within boundaries

The question should not be:

“Can we trust AI never to make a mistake?”

No responsible technology strategy should depend on that assumption.

A better question is:

“Can we design the process so that the AI performs a useful, clearly defined role and mistakes are contained?”

In many insurance workflows, the answer is yes.

Limit the scope.

Control the information.

Restrict the actions.

Verify important details.

Escalate uncertainty.

Monitor performance.

Keep people responsible for the decisions that matter.

That is what makes AI automation practical.

And those guardrails do not prevent automation.

They are what make it possible to use it responsibly.