Dify RAG Evaluation Sets: Samples, Versions, and Regression

Dify RAG Evaluation Sets: Samples, Versions, and Regression is not about keyword stuffing or page volume; it is about turning business boundaries, inputs, handoffs, acceptance states, and maintenance into an inspectable operating system.

Direct decision

The direct decision begins with concrete inputs: the current Dify RAG evaluation set version, the candidate sample batch, the baseline score from the last approved version, and the regression threshold defined in the service contract. The work output is a single decision record that states "Approved for release" or "Blocked for release," accompanied by the delta between candidate and baseline scores per sample group and an exported JSON of the exact version used. This review state is visible to the release engineer and product owner, and it requires no further interpretation because the decision rule is fixed: if the candidate set meets or exceeds the baseline on all required metrics, the release proceeds; otherwise, it is blocked.

If the decision fails, meaning the candidate evaluation set shows regression in any required metric, the service team must immediately revert the active RAG configuration to the last approved evaluation set version and reopen the sample review queue. The concrete failure input is the specific sample IDs and question-answer pairs that caused the score drop, and the work output is a regression report that pairs those failing samples with the prior version’s outputs. The report enters a failed review state and is routed to the knowledge engineer, who must either fix the sample annotations, adjust the evaluation set version, or update the regression threshold with client-approved justification. Only after a new candidate batch is tested against the same baseline and passes the direct decision rule can the release move forward.

Fit and exclusions

Our evaluation set service is designed for teams using Dify RAG pipelines that need structured samples, versioned datasets, and regression-ready baselines. Concrete inputs include raw query logs, expected answer snippets, metadata tags such as document source and chunk ID, and a target evaluation frequency. From these, we produce a labeled evaluation set with clear version labels, a manifest of sample-level changes across versions, and a readiness report that flags ambiguous or conflicting samples. Each version enters a review state where your internal subject matter experts confirm the correctness of expected answers and boundary cases before we mark it as production-ready. If the review fails, we automatically flag the affected samples, provide a discrepancy log, and re-run the curation step with revised instructions or additional source material until the set passes validation.

Regression testing is part of the service only when your evaluation set includes at least one prior version baseline. For every new version, we run a side-by-side comparison against the previous set, measuring sample overlap, answer drift, and retrieval consistency. The work output is a regression summary that pinpoints where performance changed, which samples caused the shift, and whether a rollback is advisable. The review state here is a pass/fail decision based on your predefined tolerance thresholds, not our own benchmarks. If the regression fails, we provide a diff report and a recommended remediation path, such as reverting to the last stable version or refining the failing samples. Excluded from this service are custom retraining of RAG models, infrastructure tuning, and any ongoing monitoring beyond the evaluation set lifecycle.

Inputs and evidence

Concrete inputs for a Dify RAG evaluation set start with a curated collection of sample queries and ground-truth answers, sourced from production logs and expert annotations. The work output is a versioned evaluation bundle that includes a complete sample manifest, expected answer set, and a checksum for integrity. Each bundle’s review state is tracked in a changelog with an explicit approval status, such as draft, in review, or approved. If the evaluation set fails validation—for example, when missing fields or conflicting ground truth are detected—the entire bundle is quarantined, the last approved version is restored, and the affected samples are returned to the annotation queue for corrective action.

For regression testing, the primary input combines the previously approved evaluation set with a delta of newly added edge-case samples that reflect recent production queries. The work output is a side-by-side comparison report that shows retrieval hit rates and answer relevance scores across the old and new versions. The review state of this regression report is recorded as part of the version control history, requiring an explicit sign-off from the evaluation owner. If the regression run fails against a defined threshold, the pipeline is halted, the underlying model or embedding configuration is reverted to the last known-good state, and a new evaluation cycle is launched with the failing samples isolated for root-cause analysis.

Implementation workflow

Before building evaluation sets, the team must complete four dependent workstreams. Diagnosis: inventory existing business questions, customer tickets, support logs, and chat transcripts to identify the highest-frequency intents; classify which intents require a reference answer, a source, and a refusal rule. Design: define the evaluation schema (question, answer, source, refusal flag, expected behavior) and decide versioning keys for knowledge, model, and prompt. Production: generate a seed set from real records, have a second reviewer confirm answers against approved sources, and store versioned artifacts in the project repository. Launch: run a baseline regression, compare model outputs against the seed set, and log differences per version. Each handoff must include owner, status, and evidence.

Checklist and handoff fields: (1) Scope: list of intents and their priority; (2) Sample: each question has a stable ID, source citation, and refusal rule; (3) Versions: record knowledge version, model identifier, prompt version, and timestamp; (4) Regression: diff report showing changed outputs and reason codes. Verify that every seed record passes the acceptance criteria: answer matches the approved source, refusal rules fire correctly, and the expected behavior field is populated. If any pass fails, the failure diagnosis should name the artifact (sample, version, or prompt) and route it back to design. This workflow deliberately avoids promising any specific outcome; it only defines the evidence needed to confirm that an evaluation set is ready for regression testing.

Team responsibilities and handoff

The owning team receives the approved sample set from the evaluation lead, including raw queries, reference answers, and metadata tags. Their work output is a versioned evaluation set stored in the Dify project with unique version IDs and a change log describing additions, removals, and label updates. This output enters a review state where a second reviewer from the quality team checks that all sample IDs match the source records and that no sensitive data is present; only then is it marked ready for regression. If the review fails, the owning team must produce a corrected version, increment the version number, and re-enter the queue with a note about what changed and why.

The regression team then takes the approved evaluation set and runs it against the current and candidate RAG pipelines. Their work output is a regression report that lists per-sample pass/fail status, error excerpts, and a comparison of retrieval relevance scores for the previous and current versions. This report enters a review state where the solution architect verifies that the test run used the exact version ID from handoff and that the evaluation set was not altered during execution. If the report fails verification, the regression team must rerun the evaluation using the original versioned set and document the rerun reason in the handoff log, while the owning team is alerted only if the sample set itself needs correction.

Readiness review

For each evaluation set, we review concrete inputs: the source documents, the query set, and the annotated ground truth. The work output is a versioned sample manifest that records the exact data split, any preprocessing steps, and the linked Dify knowledge base configuration. The review state is "pending" until every sample in the set maps to a verifiable source chunk and a human-approved label. If the review fails—missing source references, ambiguous labels, or outdated embeddings—we reject the version, document the gap in the change log, and route it back to the evaluation engineer with a specific remediation checklist.

On the regression side, concrete inputs are the previous evaluation set versions and the current candidate model configuration. The work output is a regression readiness report that compares per-sample scores, aggregate metrics, and error clusters across versions. The review state is "ready" only when the candidate passes the full diff on the frozen baseline set, with no unexplained metric shifts or new failure categories. If the report fails—unexpected precision drop, silent truncation, or token budget overruns—we halt promotion, isolate the offending queries, and run a targeted root-cause analysis before the set is cleared for another review cycle.

Failure handling and escalation

When a new sample is added to a Dify RAG evaluation set, the concrete inputs include the question, expected answer, source document reference, and any relevant metadata tags. The work output is a validated sample record that is versioned and merged into the active evaluation set. The review state is initially "pending validation," during which an automated check confirms the sample’s structure, completeness, and answer relevance. If the sample fails validation—for example, due to missing metadata, an empty expected answer, or an ambiguous reference to the source document—the failure is logged in the evaluation set’s audit trail and the sample is quarantined. The escalation path automatically notifies the evaluation set owner, who either corrects the sample or rejects it. The new version remains unreleased until every sample passes validation, preventing defective data from entering the regression baseline.

For regression testing, the concrete inputs are a versioned evaluation set and a candidate Dify RAG pipeline configuration. The work output is a regression report that compares retrieval relevance and generation quality metrics against the previous baseline version. The review state is "under review" by the quality engineer, who checks whether the performance has degraded or remained stable. If the report indicates a failure—such as a drop in retrieval hit rate or a significant increase in answer irrelevance—the failure is escalated with the exact version diff, sample IDs, and the specific metric that triggered the alert. The engineering team then either rolls back to the last known good pipeline version or, if the test labels are outdated, updates the evaluation set and re-runs the regression. No version is approved until the failure is resolved and a clean report is produced, ensuring that every deployed RAG configuration remains accountable to a verified evaluation set.

Maintenance and stop criteria

Continue funding an evaluation set while it keeps distinguishing real signal from noise: keep the set when newly added business questions still match their reference answers, when drift counts between knowledge, model, and prompt versions stay stable, and when failures are isolated phrasing issues rather than a recurring intent gap. Rework the set when a version change creates a new failure cluster in one intent area, when reference answers no longer match the current knowledge base, or when refusal rules contradict questions that production users actually ask. Pause maintenance when no production traffic uses the affected flow yet, when the underlying business questions have not changed, or when a deployment is scheduled but not yet live.

Stop investment when two consecutive version comparisons produce no new failure types, when no new business question templates arrive during a review cycle, and when regression differences are pure noise rather than actionable drift. Merge evaluation pages when two intents share overlapping reference answers and the same retrieval sources; keep one page per distinct failure pattern. Use these handoff fields for every review: owner, business question source, reference answer version, knowledge base version, model version, prompt version, pass/fail counts, drift count, last review date, and decision (continue, rework, pause, merge, or stop). Anchor reference answers in original information and reader satisfaction, following Google’s people-first content guidance, and treat generative AI as a support layer that still needs this human-maintained evaluation loop.

Next step

If you are evaluating Dify RAG Evaluation Sets: Samples, Versions, and Regression, start with the current pages, assets, tools, and handoff process so the workflow can be diagnosed in a limited scope.