How to Evaluate Dify RAG Quality

Direct answer：A verifiable method to assess retrieval-augmented generation performance using question sets, component-level metrics, and regression checks.

Implementation framework

Evaluating Dify’s RAG (Retrieval-Augmented Generation) quality requires separating system components and verifying each against domain-specific expectations. This guide uses question sets with pre-defined evidence requirements.

Required inputs

Question set: 20-50 questions covering:

Evidence mapping: For each question:

Expected answer format
Required source documents
Acceptable citation boundaries

Performance thresholds:

Evaluation steps

Retrieval validation:

Run each question through the retrieval component only
Verify returned documents match the evidence map
Record precision and recall per question type

Generation assessment:

Check answer faithfulness to retrieved documents
Verify citations reference correct passages
Measure hallucination rates for unsupported claims

System behaviors:

Test abstention responses for out-of-scope queries
Monitor latency across query complexity levels
Compare versions using the same question set

Verification matrix

Metric：Measurement Method；Acceptance Criteria；Exception Handling

Latency consistency：Time-to-first-token measurement；<3s for 95th percentile；Profile slow queries

Exception cases

Queries requiring real-time data updates
Multi-hop reasoning scenarios
Domain-specific terminology variations

Next steps

Download the question set template and evidence mapping worksheet to begin baseline testing.

Implementation Framework

Build a verification pipeline that isolates each quality dimension before testing the integrated system. This prevents error masking and identifies improvement priorities.

Input Preparation

Question Set:

Field: question_id (unique identifier)
Field: question_text (exact phrasing for consistency)
Field: expected_answer (ground truth response)
Field: supporting_evidence (URLs or document excerpts)
Field: acceptable_variants (semantically equivalent answers)
Field: difficulty_class (fact lookup vs synthesis)

Acceptance: At least 50 questions covering your domain’s key entities, relationships, and edge cases.

Test Environment:

Field: dify_version (commit hash or release tag)
Field: index_fingerprint (content hash of ingested documents)
Field: hardware_profile (GPU model and memory allocation)
Field: concurrency_level (parallel request count)

Evaluation Steps

Retrieval Audit:

Run each question with retrieval-only mode enabled
Record retrieved_document_ids and retrieval_score
Check if expected evidence appears in top 5 results

Response Validation:

Field: faithfulness_score (0-3 scale for hallucination)
Field: citation_coverage (% of claims backed by evidence)
Field: abstention_correct (when "I don’t know" is expected)
Field: latency_ms (p95 response time)

Version Comparison:

Re-run identical question sets across versions

Exclude questions with modified evidence base

Working Record

question_id：retrieval_score；faithfulness_score；citation_coverage；latency_ms；version_compare

Verification items:

Confirm test index matches production content snapshot
Validate scoring rubrics against human judgments
Document known limitations of automated metrics

Establishing Evidence Standards

Documenting Source Hierarchy

Create a verification table with these fields:

Field：Purpose；Acceptance Criteria

Primary Source Type：Distinguish first-party docs, vetted research, and user manuals；Must have ≥2 independent citations per fact claim

Evidence Freshness：Date of last update for each source；Technical specs ≤12 months old; research ≤24 months

Contradiction Log：Track conflicting claims across sources；Flag unresolved conflicts for human review

Abstention Threshold：Minimum confidence score for answer generation；Default 0.7; adjust per risk profile

Exception Handling: When primary sources disagree, implement a fallback protocol:

Log the conflict in the Contradiction Log
Weight newer peer-reviewed sources higher
If unresolved, trigger abstention with explanation

Defining Answer Boundaries

Boundary Type：Verification Method；Example Failure Mode

Fact vs Opinion：Source type validation；Recommending untested architectures as fact

Known Unknowns：Confidence scoring；Asserting precision without error margins

Temporal Scope：Date range checks；Applying 2021 benchmarks to 2024 models

Quality Gate: Reject any response that:

Cites unpublished or retracted work without disclaimer
Extrapolates beyond study parameters
Fails to disclose conflicting evidence

Implementing Inspection Workflows

Retrieval Evaluation Protocol

Query Generation: Create 50 test questions spanning:

Ground Truth Mapping: For each question:

Field：Requirement

Expected Answer Type：Fact, procedure, or abstention

Supporting Evidence：Exact source passages with timestamps

Permitted Variance：Allowable paraphrasing threshold

Scoring Matrix:

Metric：Measurement；Pass Threshold

Latency Baseline: Measure retrieval time across:

Cold cache (no prior queries)
Warm cache (repeated queries)
Peak load (parallel requests)

Version Regression Checks

Test Case：Comparison Method；Delta Threshold

Answer Consistency：Semantic similarity score；≥0.85 F1

Verification Item: Document any performance degradation exceeding thresholds with:

Environment details
Exact query samples
Proposed mitigation timeline

Implementing a Dify RAG Evaluation Framework

Step 1: Define Test Cases and Acceptance Criteria

Create a structured question set with:

Query Intent Classification: Label each as navigational, informational, or transactional
Expected Answer Format: Specify whether responses should be extracted text, synthesized summaries, or direct citations
Evidence Requirements: Document required source documents or passages for verification

{

"test_case_template": {

"query_id": "TC-2024-06-001",

"query_text": "What are the hardware requirements for Dify Enterprise Edition?",

"expected_response_type": "extracted_passage",

"required_sources": ["Dify EE Installation Guide v3.2, Section 2.1"],

"acceptance_criteria": {

"retrieval_precision": 1.0,

"response_latency": "<1500ms",

"citation_completeness": true

}

Step 2: Execute Multi-Dimensional Evaluation

Retrieval Evaluation

Measure recall@k for critical documents
Verify document freshness against source update timestamps
Check for forbidden content types (e.g., draft documents)

Response Quality

Use ternary scoring (0=incorrect, 0.5=partially correct, 1=fully correct) for:
Answer faithfulness to sources
Abstention appropriateness
Citation completeness

Performance Metrics

Track 95th percentile latency across query types
Monitor cold-start performance after model updates

Step 3: Analyze Version Regressions

Maintain a version comparison matrix:

Metric：v1.3 Baseline；v1.4 Candidate；Δ Threshold

Medical QA F1：0.82；0.79；≤0.03

API Latency：1200ms；1350ms；≤200ms

Exception Handling

Ambiguity Resolution: For queries with multiple valid interpretations:

Log alternative interpretations
Measure clarification rate

Novel Query Handling: When encountering unseen query patterns:

Flag for human review
Add to regression test suite

Source Conflicts: For contradictory evidence:

Verify document versioning
Apply recency weighting

Verification Methods

Automated Checks: Implement CI/CD pipeline with:
Retrieval assertion tests
Citation validation rules

Human Audit: Monthly sampling of:
High-stakes queries
Edge case responses

Business Impact: Correlate with:
Support ticket reduction
User satisfaction surveys

Evaluating the quality of a Dify Retrieval-Augmented Generation (RAG) system requires a structured approach to ensure comprehensive assessment across multiple dimensions. Below is a step-by-step guide to help you systematically evaluate Dify RAG quality.

Step 1: Build a Question Set with Expected Answers and Evidence

Start by creating a diverse set of questions that cover various topics relevant to your domain. For each question, define the expected answer and gather supporting evidence. This evidence will serve as a benchmark for evaluating the system’s performance.

Step 2: Evaluate Retrieval

Assess the system’s ability to retrieve relevant documents. Use the question set to query the system and compare the retrieved documents against the expected evidence. Measure retrieval accuracy by calculating the percentage of relevant documents retrieved.

Step 3: Evaluate Ranking

Next, evaluate how well the system ranks the retrieved documents. The most relevant documents should appear at the top of the list. Use metrics like Normalized Discounted Cumulative Gain (NDCG) to quantify ranking effectiveness.

Step 4: Evaluate Faithfulness

Check the faithfulness of the generated answers to the retrieved documents. Ensure that the answers are accurate and do not introduce any misinformation. Cross-reference the generated answers with the expected evidence to verify their correctness.

Step 5: Evaluate Citations

Assess the quality of citations provided by the system. Citations should accurately reference the retrieved documents and support the generated answers. Verify that all citations are relevant and correctly attributed.

Step 6: Evaluate Abstention

Evaluate the system’s ability to abstain from answering when it lacks sufficient information. The system should avoid generating answers for questions it cannot confidently address. Measure abstention accuracy by comparing the system’s responses to the expected answers.

Step 7: Evaluate Latency

Measure the system’s response time for each query. Latency should be within acceptable limits to ensure a smooth user experience. Use benchmarks to compare the system’s latency against industry standards.

Step 8: Evaluate Version Regressions

Finally, assess the system’s performance across different versions. Compare the current version’s performance against previous versions to identify any regressions. Ensure that improvements in one area do not negatively impact others.

Record Fields and Decision Criteria

To document your evaluation, use a record template with fields for each evaluation dimension. Include criteria for acceptable performance, exceptions, and methods for acceptance checks. This structured approach will help you systematically assess and improve Dify RAG quality.

Exceptions and Acceptance Methods

Identify any exceptions where the system’s performance deviates from expected standards. Develop acceptance methods to address these exceptions and ensure continuous improvement. Regularly review and update your evaluation criteria to keep pace with evolving requirements.

Implementing a Controlled Evaluation Rollout

Baseline Establishment

Create measurement baselines for:

Retrieval precision/recall against gold-standard evidence
Ranking quality via NDCG@5
Answer faithfulness using contradiction detection
Citation accuracy through source verification
Abstention rate on unanswerable queries
P95 latency under production load
Version comparison matrix

Document baseline thresholds in your evaluation record (see artifact).

Observation Protocol

For each evaluation batch:

Log question ID and type
Record retrieval document IDs
Flag ranking position errors
Note faithfulness violations
Verify citation matches
Mark incorrect abstentions
Capture latency measurements
Compare version metrics

Decision Framework

Continue Criteria

Proceed to full rollout when:

Retrieval precision ≥ baseline (p<0.05)
No critical faithfulness failures

Latency under SLA
No version regressions

Rework Requirements

When metrics fall below baseline but above failure thresholds:

Document specific failure patterns
Isolate component responsible
Implement targeted improvements
Re-evaluate with same test set

Stop Conditions

Halt rollout if:

Ranking NDCG declines >0.2

Latency exceeds 2x SLA
Critical version regression