RAG Chunking Evaluation with a Fixed Question Set

Learn how to evaluate RAG chunking methods using a fixed question set. This guide covers building a question set, running a comparison experiment, and recording context completeness to make an informed choice.

RAG Chunking Evaluation with a Fixed Question Set is not a generic keyword-volume exercise. It turns the topic into an operational method that a B2B team can inspect, repeat, and revise.

The scope is deliberately limited: Compare section, semantic, and hybrid chunking with a fixed question set, recording retrieved passages, context completeness, answer support, and failure types in a reproducible experiment sheet.

Treat every section as one part of the same capability matrix and trial acceptance checklist. Confirm the decision object and inputs first, complete the topic-specific actions next, and retain evidence, exceptions, and acceptance results at the end.

Any worked example explains the method only; it does not replace the company’s own data, platform records, source review, or sales validation.

When you evaluate retrieval-augmented generation (RAG) systems, the chunking method you choose directly affects answer quality.

Without a controlled comparison, you cannot tell whether a difference in responses comes from chunk size, embedding model, or retrieval settings. RAG Chunking Evaluation with a Fixed Question Set gives you a repeatable way to isolate that variable.

By asking the same questions against the same corpus, you can compare section-based, semantic, and hybrid chunking under identical conditions.

This article walks you through building that question set, running the experiment, and recording the results in a way that supports a purchase decision.

Why a Fixed Question Set Is the Backbone of RAG Chunking Evaluation

A fixed question set is the control in your experiment. Without it, you cannot compare chunking methods fairly because different questions may favor different chunk sizes.

A question that asks for a specific number might be answered by a small chunk, while a question about a process might need a larger context. If you change questions between runs, you introduce a second variable that confounds your results.

The core evaluation criteria for chunking are retrieval precision, context completeness, and answer support. Retrieval precision measures whether the retrieved passages contain the information needed.

Context completeness checks whether the retrieved passages together cover all parts of the ground truth answer. Answer support verifies that the generated answer can be traced back to the retrieved passages.

A fixed question set lets you score each chunking method on these criteria consistently.

A decision to adopt a chunking strategy should be based on evidence, not on vendor claims. By running the same questions through different chunking methods, you generate data that shows which method retrieves the most relevant passages for your domain.

This evidence becomes part of your capability matrix and trial acceptance checklist.

Preparing Your Fixed Question Set: Coverage and Ground Truth

Start by collecting questions that reflect how your users actually ask about your content. Include questions of varying difficulty: simple fact lookup, multi-step reasoning, and questions that require synthesizing information from multiple sections.

Also include questions that span different document types in your corpus, such as product manuals, API documentation, and troubleshooting guides.

For each question, define ground truth: the exact passages or sentences that contain the answer. This is your reference for scoring context completeness. You can create ground truth by manually marking the relevant sections in your source documents.

If you have existing support tickets or FAQs, use those as a starting point, but verify that the answers are present in your corpus.

Aim for a question set that is large enough to be statistically meaningful but small enough to score manually. As an adjustable illustrative assumption, you might start with 50 questions.

That number is not a requirement; it is a starting point that you can adjust based on your time and corpus size. The key is that the same set is used for every chunking method you test.

Step-by-Step: Running the Chunking Comparison Experiment

Prepare your corpus by cleaning the text and removing any boilerplate that could interfere with retrieval. Then apply each chunking method to the same corpus. Section-based chunking splits by headings and paragraphs.

Semantic chunking groups sentences by embedding similarity. Hybrid chunking combines both, using structure as a primary split and semantic similarity to refine boundaries.

For each chunking method, index the chunks and run your fixed question set through the RAG pipeline. Use the same embedding model, retrieval top-k, and generation settings for all methods. This ensures that any difference in output is due to chunking alone.

Record the retrieved passages for each question and each method.

Run the experiment in a controlled environment, ideally on a staging server, to avoid interference from live traffic. Document the exact parameters you used, such as chunk size limits and overlap, so that the experiment is reproducible.

If you are evaluating a commercial platform, ask the vendor for a trial environment where you can run this test yourself.

Recording Retrieved Passages and Context Completeness

For each question and each chunking method, log the retrieved passages. Note the chunk IDs and the text of each passage. Then compare the retrieved passages against your ground truth. Mark each ground truth segment as covered or not covered.

Context completeness is the percentage of ground truth segments that appear in the retrieved passages.

Also record answer support: whether the generated answer can be traced to the retrieved passages. If the answer includes information not in the retrieved passages, that is a failure type.

Other failure types include missing context, where a ground truth segment is not retrieved, and irrelevant retrieval, where retrieved passages do not contain the answer.

Create a table that summarizes the results for each chunking method. Include columns for retrieval precision, context completeness, answer support, and failure types. This table becomes part of your capability matrix.

Use it to compare the methods side by side and to decide which one meets your requirements.

Your trial acceptance checklist should include criteria such as: context completeness above a threshold you define, answer support for all critical questions, and no failure types that block user tasks.

As an adjustable illustrative assumption, you might set a minimum context completeness of 80%, but that number is your own threshold, not a universal standard.

By recording retrieved passages and context completeness, you create evidence that supports your decision. You can show stakeholders exactly how each chunking method performed on your questions.

This evidence is more persuasive than a vendor demo because it is based on your own content and your own questions.

RAG Chunking Evaluation with a Fixed Question Set is a method for comparing how different chunking strategies affect retrieval-augmented generation quality.

Instead of relying on generic metrics, you define a set of questions that represent real user tasks, run them against each chunking method, and score the answers based on how well the retrieved context supports them.

This approach turns a vague decision into a reproducible experiment.

Scoring Answer Support and Categorizing Failure Types

To compare chunking methods, you need a scoring rubric that measures how well the retrieved context supports the final answer. A simple three-point scale works well: 0 for no support, 1 for partial support, and 2 for full support.

Full support means the retrieved passages contain all the facts needed to answer the question without external knowledge. Partial support means some facts are present but others are missing or contradictory.

No support means the answer is based on nothing in the retrieved context.

Beyond scoring, you should categorize failure types to understand why a chunking method fails. Common categories include missing context, irrelevant retrieval, and hallucination.

Missing context occurs when the answer requires information that is not present in any retrieved passage. Irrelevant retrieval happens when the top-ranked passages are off-topic or contain only tangentially related content.

Hallucination is when the model generates facts that are not supported by the retrieved passages at all.

By tagging each failure, you can see patterns: for example, if a method consistently produces irrelevant retrieval, the chunk boundaries may be splitting sentences or concepts.

A good scoring sheet also records the retrieved passages for each question, so you can trace the failure to a specific chunk. This traceability is essential for diagnosing whether the problem is in the chunking, the embedding, or the retrieval parameters.

Without it, you are guessing.

Example: A Sample Experiment Sheet for a Legal Document Corpus

Consider a legal document corpus containing contracts, case law, and regulatory text.

You want to evaluate three chunking methods: fixed-size chunking with 500 tokens, semantic chunking that groups by topic, and hybrid chunking that uses fixed-size with overlap.

You create a fixed question set of ten questions, such as "What are the termination clauses in the service agreement?" or "What is the liability cap in the master contract?"

Here is a sample experiment sheet for three questions. The scores are illustrative assumptions for demonstration purposes only; your actual results will vary.

In this example, the hybrid method shows the highest support scores, but that is not a universal conclusion. The sheet helps you see that semantic chunking sometimes misses context because it groups by topic, which may split a clause across chunks.

Fixed-size chunking may miss context when a clause spans more than 500 tokens. The failure types give you a clue about what to adjust.

Interpreting Results and Choosing the Right Chunking Strategy

After running your fixed question set, you will have a table of support scores and failure types. The first step is to compute the average support score for each method.

A higher average indicates better overall answer support, but you should also look at the distribution. If one method has a high average but a few catastrophic failures, it may be risky for production.

Next, examine the failure types. If a method consistently produces missing context, you may need to increase chunk size or add overlap. If it produces irrelevant retrieval, the chunk boundaries may be breaking semantic units, so semantic chunking might help.

If hallucination appears, the retrieval may be returning passages that are not relevant, leading the model to invent facts.

Your choice should also depend on your use case. For a legal document corpus where precision is critical, you might prioritize minimizing missing context over retrieval speed.

For a general knowledge base, you might tolerate some irrelevant retrieval if the overall answer quality is high. The fixed question set lets you weigh these trade-offs with data.

Remember that the results are specific to your corpus and question set. A chunking method that works well for legal documents may not work for technical manuals. The evaluation is a decision tool, not a universal ranking.

Pitfalls to Avoid and How to Validate Your Evaluation

One common pitfall is using a biased question set. If your questions are all similar or only cover one part of the corpus, the results will not generalize.

To avoid this, design a question set that covers different topics, difficulty levels, and question types, such as factoid, list, and reasoning questions. You can also ask domain experts to review the questions for representativeness.

Another pitfall is ignoring retrieval order. The support score should reflect the quality of the retrieved passages, not just the final answer. If the correct passage is ranked third, the answer may still be correct, but the retrieval is less effective.

Record the rank of the first relevant passage and consider it in your scoring.

To validate your evaluation, you can run the experiment multiple times with different random seeds or question orders to check for consistency. You can also have a second person independently score a subset of answers to measure inter-rater reliability.

If your scoring is subjective, define clear criteria for each score level.

Finally, avoid overfitting to a small question set. A fixed question set of ten questions is a starting point, but you should expand it as you refine your chunking strategy.

The goal is to create a repeatable process that you can trust for future changes to your RAG pipeline.

By following this method, you can make an evidence-based decision about chunking, rather than relying on intuition or vendor claims.

Next step

Ready to evaluate your RAG chunking strategy? Contact SHMLANG for a consultation on AI automation and retrieval quality.