Reducing AI Hallucination with a Multi-Level Agent System

Feb 3, 2025

3 min read

Reducing AI Hallucination with a Multi-Level Agent System — Source: Freepik - kovalovds

Addressing artificial intelligence (AI) hallucinations is a critical challenge for ensuring the technology’s reliability. A recent study suggests that multi-level agent systems, combined with natural language processing (NLP)-based frameworks, could significantly mitigate this issue.

In the study "Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks," Gosmar and Dahl developed a three-tier agent system, testing its effectiveness in reducing hallucinations using 310 specially designed prompts. The results indicate that incorporating a second-tier agent reduced the hallucination score (THS) from -1.52 to -14.12, while implementing the third-tier agent further lowered it to -43.27 compared to the first level.

Source: https://arxiv.org/abs/2501.13946

The authors evaluated the system’s effectiveness using four key performance indicators (KPIs): Factual Claim Density (FCD), measuring the concentration of factual statements; Factual Grounding References (FGR), tracking the number of fact-based citations; Fictional Disclaimer Frequency (FDF), assessing how often the system flags potentially fabricated content; and Explicit Contextualization Score (ECS), quantifying the extent of explicit contextualisation in responses.

The findings highlight a strong correlation between reducing hallucinations and the structured transfer of information between agents. The Open Voice Network (OVON) framework and structured data transmission formats, such as JSON messages, play a pivotal role in this process, enabling agents to identify and flag potential hallucinations effectively. This approach has not only proven successful in lowering hallucination scores but has also contributed to enhancing the transparency and reliability of responses.

Sources:

Hallucination Mitigation using Agentic AI Natural Language-Based Frameworks

Hallucinations remain a significant challenge in current Generative AI models, undermining trust in AI systems and their reliability. This study investigates how orchestrating multiple specialized Artificial Intelligent Agents can help mitigate such hallucinations, with a focus on systems leveraging Natural Language Processing (NLP) to facilitate seamless agent interactions. To achieve this, we design a pipeline that introduces over three hundred prompts, purposefully crafted to induce hallucinations, into a front-end agent. The outputs are then systematically reviewed and refined by second- and third-level agents, each employing distinct large language models and tailored strategies to detect unverified claims, incorporate explicit disclaimers, and clarify speculative content. Additionally, we introduce a set of novel Key Performance Indicators (KPIs) specifically designed to evaluate hallucination score levels. A dedicated fourth-level AI agent is employed to evaluate these KPIs, providing detailed assessments and ensuring accurate quantification of shifts in hallucination-related behaviors. A core component of this investigation is the use of the OVON (Open Voice Network) framework, which relies on universal NLP-based interfaces to transfer contextual information among agents. Through structured JSON messages, each agent communicates its assessment of the hallucination likelihood and the reasons underlying questionable content, thereby enabling the subsequent stage to refine the text without losing context. The results demonstrate that employing multiple specialized agents capable of interoperating with each other through NLP-based agentic frameworks can yield promising outcomes in hallucination mitigation, ultimately bolstering trust within the AI community.