The real risk with AI hallucinations starts when a wrong answer makes its way into a business decision. By then, the challenge is figuring out where the information came from, whether anyone verified it, and who is responsible for the outcome.
That gets harder when employees use AI tools outside approved systems.
Andy Fielder, CTO of MetaCompliance, argues that organizations need a clear view of where AI is being used and stronger verification when mistakes could have serious consequences.
Why It Matters: AI governance has to account for the consequence of an incorrect output. Applying heavy verification to low-risk work can erode efficiency, while weak controls around consequential uses can leave organizations exposed. Fielder advocates verification requirements tied to the risk of the specific use case.
- AI Errors Become More Serious Once They Enter Business Processes: In 2023, US lawyers were sanctioned after filing documents containing fictitious citations generated by ChatGPT. Fielder uses the case to show how an unchecked AI output can move from a useful shortcut to a costly mistake.
- Unsanctioned AI Leaves Organizations With Blind Spots: Employees may use AI without their organizations knowing what information is being shared or how much trust is being placed in the output. MetaCompliance research found that 40% of CISOs fear employees are sharing sensitive information with generative AI platforms. This lack of visibility makes it harder to catch inaccurate information before someone acts on it.
- Verification Should Match the Risk: Fielder argues against applying the same review process to every AI-assisted task. Low-risk work may only need a light check, while AI used in consequential decisions requires stronger approval and human review. This keeps verification focused where mistakes carry greater consequences.
- Governance Starts With Knowing Where AI Is Used: Organizations need a clearer picture of where unapproved AI has entered workplace processes. Providing approved alternatives can give security teams more visibility while creating clearer boundaries around acceptable use and accountability.
- AI Training Needs to Change Behavior: A generic instruction to “double-check” AI gives employees little guidance on what verification actually requires. Fielder calls for defined verification routines and training on automation bias so employees know when an AI output deserves more scrutiny. The EU AI Act adds regulatory pressure through requirements covering AI literacy and transparency.
Go Deeper -> What happens when AI is confidently wrong in the workplace? – TechRadar
