Dify RAG Evaluation for Retrieval, Answers, Citations, and Regression

This article provides a structured framework for evaluating Dify RAG deployments, covering fixed evaluation sets, retrieval quality scoring, and answer correctness assessment.

Why a Fixed Evaluation Set Matters

A fixed evaluation set is the foundation of any reproducible acceptance test for a Dify RAG deployment. Without a stable set of queries and expected outputs, you cannot compare results across model updates, prompt changes, or knowledge base modifications.

The goal is to create a benchmark that is representative of real business usage and includes edge cases that stress the system’s boundaries.

Start by collecting queries from actual user sessions, support tickets, and domain experts. Aim for a set of 50 to 100 queries that cover the typical scenarios your RAG system will handle.

For each query, define the expected answer based on the authoritative knowledge base. This expected answer should be a concise summary of the facts that the system should retrieve and synthesize.

Also, record the exact source chunks (document and section) that contain the supporting information. This becomes your ground truth for retrieval and citation checks.

Edge cases are critical. Include ambiguous queries that could have multiple interpretations, out-of-scope questions that the knowledge base does not address, and permission-restricted content that certain users should not see.

For each edge case, define the expected behavior: for ambiguous queries, the system should ask for clarification; for out-of-scope questions, it should refuse politely; for permission-restricted content, it should not retrieve or cite that content.

Store the evaluation set in a version-controlled file (e. g. , CSV or JSON) so that changes are tracked. Each query should have a unique ID, the query text, the expected answer, the list of expected source chunks, and any special handling notes.

This fixed set becomes your regression suite. When you update your Dify configuration, you run the same queries and compare results against the baseline.

A fixed evaluation set also enables you to localize defects. If a query fails, you can trace the issue to a specific retrieval step, a prompt version, or a model behavior.

Without a fixed set, you are left with anecdotal observations that are hard to reproduce.

Scoring Retrieval Quality: Hit Rate and Ranking

Retrieval quality determines whether the RAG system can find the right information to answer a query. Two key metrics are retrieval hit rate and ranking quality. Hit rate measures whether the relevant chunks appear in the top-k results.

For each query, check if any of the expected source chunks are present in the top 5 or top 10 retrieved chunks.

Ranking quality goes beyond hit rate by considering the order of results. Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG) are standard metrics. MRR is the average of the reciprocal rank of the first relevant chunk.

For example, if the first relevant chunk is at position 2, the reciprocal rank is 1/2 = 0. 5. NDCG accounts for the graded relevance of multiple relevant chunks.

These metrics give you a more nuanced view of how well your retrieval system prioritizes the most useful information.

To score retrieval, you need to log the retrieval results for each query. Dify provides API access to the retrieval step, so you can capture the list of chunk IDs and their scores. Record the retrieval latency and the number of chunks returned.

This data helps you identify performance bottlenecks.

When a query fails retrieval, inspect the chunk IDs returned. If the expected chunk is missing, the issue may be in the embedding model, the chunking strategy, or the search algorithm.

For example, if your knowledge base uses a chunk size of 500 tokens, but the relevant information is split across two chunks, the retrieval might miss one. Adjusting the chunk size or overlap can improve hit rate.

Another common failure is when the query uses synonyms or paraphrases that the embedding model does not map well. In such cases, you might need to add query expansion or use a hybrid search that combines keyword and vector search.

Dify supports hybrid search, which can improve recall.

Document your retrieval scoring process in a table. For each query ID, record whether the hit was successful, the MRR score, and any notes on the failure. This table becomes part of your evaluation scorecard.

Scoring Answer Correctness: Factual Consistency and Completeness

Answer correctness assesses whether the generated answer is factually consistent with the retrieved chunks and complete relative to the expected answer. Factual consistency means that every statement in the answer is supported by the retrieved chunks.

Hallucinations occur when the model generates information not present in the chunks. To check for hallucinations, compare each sentence in the answer against the retrieved chunks.

If a sentence contains a fact that is not in any chunk, flag it as a hallucination.

Completeness means that the answer covers all key points that the expected answer includes. For each query, list the key facts that must be present. Then check whether the answer includes them.

For example, if the expected answer mentions three benefits of a product, the generated answer should mention all three. Missing one is a completeness failure.

Use a scoring rubric from 1 to 5 for both factual consistency and completeness. A score of 5 means the answer is fully consistent and complete. A score of 1 means the answer is mostly hallucinated or missing critical information.

Define the rubric clearly so that different evaluators can apply it consistently.

For factual consistency, a score of 5 means every statement is directly supported by the retrieved chunks. A score of 4 means there is one minor unsupported detail that does not change the meaning.

A score of 3 means there is a moderate hallucination that could mislead the user. A score of 2 means the answer contains several unsupported statements. A score of 1 means the answer is largely fabricated.

For completeness, a score of 5 means all key points from the expected answer are covered. A score of 4 means one key point is missing. A score of 3 means two key points are missing. A score of 2 means more than half of the key points are missing.

A score of 1 means the answer does not address the query at all.

When an answer fails, localize the defect. Determine whether the issue is in retrieval (the relevant chunk was not retrieved), in the prompt (the model was not instructed to use the chunk), or in the model itself (the model ignored the chunk).

Record the query ID, the chunk IDs that were retrieved, the model version, and the prompt version. This information helps you identify patterns.

For example, if many queries fail because the model does not cite sources, you might need to adjust the prompt to require citations. If the model hallucinates when the retrieved chunks are contradictory, you might need to add a conflict resolution instruction.

Use the evaluation scorecard template to record your findings.

The template includes fields for query ID, query text, expected answer, retrieval hit, answer correctness score, citation validity score, permission test result, refusal behavior, defect description, and suggested fix.

This structured approach ensures that you capture all relevant information for each test case.

By systematically scoring retrieval and answer correctness, you can identify weaknesses in your Dify RAG deployment and make targeted improvements.

Citation Validity: Traceability and Accuracy

Citation validity is not about whether a citation exists; it is about whether each citation points to a real chunk that actually supports the claim it accompanies.

A Dify RAG answer may cite a chunk that is semantically related but does not contain the specific fact stated. To test this, you need a fixed evaluation set where each expected answer is paired with the exact chunk IDs that should be cited.

Start by defining a set of queries where you know the ground truth. For each query, record the expected answer and the list of chunk IDs that contain the supporting information.

When you run the query through Dify, capture the generated answer and the citation list.

Then, for each citation, verify two things: first, that the cited chunk ID exists in your knowledge base and was actually retrieved (not fabricated); second, that the chunk text contains the specific claim made in the answer.

A common failure is that the model cites a chunk that is topically related but does not contain the precise number, date, or name used in the answer.

To make this test repeatable, define a scoring rubric. For each query, assign a citation validity score from 1 to 5. A score of 5 means every citation is correct and directly supports the claim.

A score of 3 means some citations are correct but others are irrelevant or only partially support the claim. A score of 1 means no citation is valid, or the answer cites chunks that do not exist. Use this rubric consistently across your evaluation set.

Also check citation placement. In Dify, citations are typically attached to specific sentences or paragraphs. Verify that the citation appears next to the sentence it supports, not at the end of a long paragraph where it could be ambiguous.

For example, if the answer states a fact in the first sentence but the citation is placed after the third sentence, a reader cannot tell which claim the citation supports.

This is a defect that should be logged with the query ID and the specific sentence index.

Finally, ensure that citations are not fabricated. Dify may generate a citation that looks plausible but does not correspond to any chunk in your knowledge base.

To catch this, compare the citation list against the actual chunk IDs returned by the retrieval step. If a citation is not in the retrieved set, it is a fabrication. Log this as a critical defect.

Permission Isolation and Refusal Behavior Testing

Permission isolation ensures that a user only sees information they are authorized to access.

In a Dify RAG system, this often means that the knowledge base contains documents with different access levels, and the retrieval step must filter out chunks that the user is not allowed to see.

To test this, you need a set of queries that should be blocked for certain user roles.

Define a test matrix with user roles (e. g. , admin, manager, employee) and query types. For each role, identify queries that should return no results because the relevant chunks are restricted.

Run these queries and check that the system does not leak any content from restricted chunks. A common failure is that the retrieval step returns a restricted chunk, but the answer generation step omits it from the final answer.

This is still a defect because the chunk was exposed in the retrieval logs or could be inferred from the answer.

Also test that the system refuses to answer out-of-scope or sensitive questions. For example, if your Dify RAG is deployed for internal HR policies, it should refuse to answer questions about trade secrets or personal data of other employees.

Define a set of queries that are clearly out of scope and verify that the system responds with a refusal message rather than attempting an answer. The refusal message should be clear and should not leak internal information about why the query was blocked.

For instance, it should not say "This query is blocked because it references document ID 1234."

To score refusal behavior, use a pass/fail criterion. A pass means the system refused the query with a generic message and did not include any restricted content. A fail means the system either answered the query or leaked internal details in the refusal.

Record the exact refusal message for each test case so you can compare across versions.

Permission isolation and refusal behavior are closely related but distinct. A query may be in-scope but require a higher permission level, in which case the system should refuse or return an empty result.

A query may be out-of-scope entirely, in which case the system should refuse regardless of permission. Your test set should cover both scenarios.

Regression Testing: Impact of Changes

Regression testing is the practice of running the same evaluation set before and after any change to your Dify RAG configuration.

Changes can include updating the underlying LLM model, modifying the prompt template, adding or removing documents from the knowledge base, or changing retrieval parameters like top-k or similarity threshold.

The goal is to detect whether any dimension of performance has degraded.

To implement regression testing, you need a fixed evaluation set that covers all dimensions: retrieval hit rate, answer correctness, citation validity, permission isolation, and refusal behavior.

Run this set against your current baseline and record the scores. After a change, run the same set again and compare the scores. Any drop in score is a regression that needs investigation.

For example, if you update the LLM from version A to version B, you might see that answer correctness improves but citation validity drops. This is a regression in citation validity, even if the overall answer quality seems better.

You need to log this and decide whether to roll back the change or adjust the prompt.

To make regression testing traceable, log the version numbers and configuration changes for each run. This includes the model version, prompt version, knowledge base version, and any retrieval parameters.

Store these logs alongside the evaluation results so you can correlate a regression with a specific change.

For instance, if you add a new batch of documents and retrieval hit rate drops, you can check whether the new documents are poorly indexed or contain conflicting information.

A practical approach is to maintain a regression test suite that runs automatically on a schedule or after each deployment. The suite should output a comparison report showing the scores for each dimension before and after the change.

This report should highlight any dimension that dropped by more than a predefined threshold, such as 0. 5 on a 1-5 scale. If a drop is detected, the report should include the query IDs that failed so you can investigate the root cause.

Remember that regression testing is not about achieving perfect scores; it is about detecting unintended changes. A change that improves one dimension but degrades another is a trade-off that you need to evaluate consciously.

The evaluation set gives you the data to make that decision.

Defect Localization Fields and Reporting Template

When a test case fails, you need a structured way to report the defect so that developers can quickly identify the root cause. A defect report should include fields that localize the issue to a specific query, chunk, model, or prompt version.

Without these fields, you will waste time trying to reproduce the issue.

The following fields are essential for any defect report:

– **Query ID**: A unique identifier for the test case.
– **Query text**: The exact query that was run.
– **Expected answer**: The ground truth answer that the system should have produced.
– **Actual answer**: The answer that the system actually produced.
– **Retrieved chunk IDs**: The list of chunk IDs that the retrieval step returned.
– **Citation list**: The list of citations that the answer generation step attached.
– **Model version**: The version of the LLM used.
– **Prompt version**: The version of the prompt template used.
– **Knowledge base version**: The version of the knowledge base (e.g., date of last update).
– **Permission test result**: Pass or fail for permission isolation tests.
– **Refusal behavior**: Pass or fail for refusal tests.
– **Defect description**: A clear description of what went wrong.
– **Suggested fix**: A recommendation for how to fix the issue.

To make this actionable, use a standardized template. Below is an example of a defect report entry (with placeholder values):

This template can be used as a basis for your own defect tracking. The key is to include enough context so that a developer can reproduce the issue without having to guess.

For example, if the defect is a permission leak, you need to include the user role that was simulated and the expected access level.

To make the template even more useful, add a severity level and a category for the defect. Severity levels could be critical (e. g. , data leak), major (e. g. , wrong answer), minor (e. g. , citation placement issue).

Categories could include retrieval, generation, citation, permission, or refusal. This allows you to prioritize fixes and track trends over time.

By using a consistent reporting template, you can build a defect database that helps you identify systemic issues. For instance, if you see many defects where the citation is correct but the answer is wrong, you might need to improve the prompt.

If you see many defects where the retrieval misses relevant chunks, you might need to adjust the embedding model or chunking strategy.

Remember that the goal of defect reporting is not to assign blame but to improve the system. A well-structured report with localization fields makes it easier to fix issues and prevent them from recurring.

Next step

Download the evaluation scorecard template to start building your Dify RAG acceptance test set.