RAG Hallucination
Detects any response that carries a high risk of hallucinations due to inability to deduce the answer from the provided context. Useful for maintaining the integrity and factual correctness of the information when you only want to use knowledge from your RAG.
Background
Retrieval-augmented generation (RAG) applications are usually based on semantic search—you turn chunks of text from a knowledge base into embedding vectors (numerical representations). When a user asks a question, it’s also converted into an embedding vector. The system then finds text chunks from the knowledge base that are closest to the question’s vector, often using measures like cosine similarity. These close text chunks are used as context to generate an answer.
However, a challenge arises when the retrieved context does not accurately match the question, leading to potential inaccuracies or ‘hallucinations’ in responses.
Overview
This policy aims to assess the relevance among the question, context, and answer. A low relevance score indicates a higher likelihood of hallucinations in the model’s response.
Policy details
The policy utilizes fine-tuned specialized small language models to evaluate relevance between the question, context, and answer. When it’s triggered, the following relevance checks run:
- Is the context relevant to the question?
- This check assesses how closely the context retrieved from the knowledge base aligns with the user’s question.
- It ensures that the context is not just similar in embedding space but actually relevant to the question’s subject matter.
- Answer Derivation from Context:
- This step evaluates whether the model’s answer is based on the context provided.
- The goal is to confirm that the answer isn’t just generated from the model’s internal knowledge but is directly influenced by the relevant context.
- Answer’s Addressing of the Question:
- The final check determines if the answer directly addresses the user’s question.
- It verifies that the response is not only derived from the context but also adequately and accurately answers the specific question posed by the user.
The policy uses the <question> and <context> tags to differentiate between the question and context parts of the prompt. This is currently not customizable.