The Accuracy Problem: When the AI Assistant Gets It Wrong

Why trust, verification, and human oversight matter most as AI moves closer to the job

Last month, we looked at how glazing businesses can capture their best people’s knowledge and build AI tools from it. These assistants can answer site questions, support new starters, and hold on to what experience looks like in practice. The case was compelling. The tools exist, they’re getting easier to use, and the problem of knowledge walking out the door is real.

This month, the part of the conversation that often gets skipped. What happens when the AI assistant gives a wrong answer?

It will. That’s not a criticism of the technology. It’s a feature of how these systems work. Large language models generate plausible responses based on patterns in data. Most of the time, those responses are accurate and useful. But sometimes they’re confident and wrong. And in glazing, where a wrong spec, a missed lintel detail, or an incorrect u-value can have real consequences, “confident and wrong” is the failure mode that matters most.

The problem isn’t that AI is unreliable in a general sense. The problem is that it doesn’t signal uncertainty the way an experienced person does. A senior surveyor who isn’t sure about something will usually say so. An AI tool, unless it’s specifically designed to flag uncertainty, will give you an answer that reads the same whether it’s right or wrong.

This is sometimes called hallucination, when an AI generates something that sounds authoritative but is factually incorrect. It’s more common with specific technical details than with general guidance. Ask an AI to draft a follow-up email and the worst case is a clumsy sentence. Ask it to confirm the correct sightline measurement for a specific profile and the worst case is a job that doesn’t fit.

So how do you use AI confidently without getting caught out?

The answer isn’t to stop using it. It’s to match the level of verification to the level of risk. Think of it as a simple two-category approach.

Low-risk outputs such as drafts, summaries, administrative tasks, communication templates, can leave the AI with minimal checking. The worst case is usually that you tweak something before it goes out. These are exactly the tasks AI handles well, and the tasks you should be automating.

High-risk outputs would be anything that informs a technical decision, a product specification, a survey checklist, or a contractual position, and need a human to check the AI’s work against a reliable source. Not because AI is usually wrong, but because the cost of it being wrong once is high enough to justify the few minutes it takes to verify.

The practical implication for businesses building internal AI tools is to be explicit about this. If you’re building an assistant based on captured knowledge, document which questions it’s a reliable source for and which it isn’t. If your AI assistant is trained on surveying notes and installation guides, it’s probably a good source for site procedure questions. It probably isn’t a reliable source for regulatory compliance or current building standards unless those have been explicitly included and kept up to date.

There’s a broader point worth making. The businesses building trust in AI right now, not just enthusiasm, are the ones building verification into the workflow from the start. That means defining who checks what, under what circumstances, and what happens when an AI output looks off.

This doesn’t require a policy document. It requires a conversation. Decide which uses of AI in your business need a human to review the output before it affects the job. Write that down. Remind new starters. Check it still makes sense in six months.

AI accuracy is improving. The gap between what it gets right and what it gets wrong is narrowing. But the right response to that isn’t to remove oversight. It’s to stay informed about where the gaps are and make sure the humans in the loop are the ones who know the job well enough to spot them.

The most valuable thing AI can do in glazing isn’t replace expert judgement, It’s extend the reach of expert judgement across a wider team. That only works if the humans at the centre of it are paying attention to what comes back.

The industry needs more of that conversation. Not AI anxiety, and not uncritical enthusiasm either. Calibrated trust. That’s where the real return is.