RAG Corpus Chunking Strategy

Splitting documents at H2 headings with stable IDs and content hashes produces predictable, debuggable chunks that support incremental re-indexing.

Tags

ragchunkingembeddingsvector-searchcontent-hashing

RAG Corpus Chunking Strategy

The Lesson

When chunking documents for RAG retrieval, split at semantic boundaries the author already created (headings) rather than at arbitrary token counts. Assign deterministic chunk IDs and compute content hashes so that re-indexing can detect what changed without re-embedding everything.

Context

A lessons library of 116 markdown documents needed to be chunked for vector search. Each lesson is a standalone markdown file with an H1 title, optional H2 sections, and YAML frontmatter (title, summary, tags, repo source). The chunks feed into a RAG chatbot that retrieves relevant excerpts, cites sources by lesson title, and builds grounded answers. The corpus is rebuilt on every CI run (daily + on push), so re-indexing performance matters.

What Happened

  1. Chose H2 headings as the split boundary. Each lesson's content before the first H2 becomes chunk 0 (labeled "Introduction"). Each H2 section becomes a subsequent chunk. H3 and deeper headings stay within their parent H2 chunk — splitting on every heading would create fragments too small for meaningful retrieval.
  2. Assigned chunk IDs as {lesson_id}-{chunk_index} where chunk_index is the position within the lesson (0 for intro, 1+ for sections). This is deterministic: the same lesson content always produces the same IDs, regardless of when or where the build runs.
  3. Each chunk carries a heading path breadcrumb (e.g., "Git Basics > Branching") so the RAG prompt builder can tell the LLM where in a lesson the excerpt came from. This improves citation quality — the model can say "according to the Branching section of Git Basics" rather than just "according to Git Basics."
  4. Computed a SHA-256 content hash (truncated to 16 hex chars) of each chunk's text. The hash is stored in the chunk metadata. On re-indexing, the embedder can compare hashes against the vector store's existing metadata to skip unchanged chunks. This infrastructure exists but isn't wired yet — the current embedder does a full re-index every time.
  5. Token count is estimated as word_count * 1.33 (approximately 0.75 tokens per word), with a minimum of 1. The corpus validator warns if any chunk exceeds 5,000 estimated tokens but doesn't fail the build. In practice, 793 chunks from 116 lessons averaged well under 1,000 tokens.
  6. The corpus builder outputs two files: rag-chunks.json (the chunks) and rag-manifest.json (statistics: total lessons, chunks, skipped empty lessons, average tokens). The manifest is a build artifact for monitoring corpus drift over time.

Key Insights

Implementation Guide

Step 1: Split on H2 headings

Parse each markdown document into chunks at ## boundaries. Content before the first H2 becomes chunk 0 (the introduction). Each H2 section becomes a subsequent chunk:

import re

def chunk_document(lesson_id: str, title: str, content: str) -> list[dict]:
    chunks = []
    # Split on H2 headings, keeping the heading with its content
    sections = re.split(r"(?=^## )", content, flags=re.MULTILINE)

    for i, section in enumerate(sections):
        text = section.strip()
        if not text:
            continue

        # Extract heading name (if this section starts with ##)
        heading_match = re.match(r"^## (.+)", text)
        if heading_match:
            heading = heading_match.group(1).strip()
            heading_path = f"{title} > {heading}"
        else:
            heading_path = title  # Intro chunk

        chunks.append({
            "chunk_id": f"{lesson_id}-{i}",
            "chunk_index": i,
            "lesson_id": lesson_id,
            "heading_path": heading_path,
            "content": text,
        })

    return chunks

Keep H3 and deeper headings within their parent H2 chunk. Splitting on every heading level creates fragments too small for meaningful retrieval.

Step 2: Add content hashes for incremental re-indexing

Compute a SHA-256 hash of each chunk's text content. This allows the vector store to detect which chunks changed between builds without re-embedding everything:

import hashlib

def add_content_hash(chunk: dict) -> dict:
    normalized = " ".join(chunk["content"].lower().split())
    chunk["content_hash"] = hashlib.sha256(
        normalized.encode()
    ).hexdigest()[:16]
    return chunk

Normalize whitespace before hashing so that formatting-only changes don't trigger re-embedding.

Step 3: Estimate token counts and validate

Approximate token count as word_count * 1.33 (roughly 0.75 words per token for English text). Warn on chunks that exceed your embedding model's context window:

def estimate_tokens(text: str) -> int:
    return max(1, int(len(text.split()) * 1.33))

MAX_CHUNK_TOKENS = 5000

for chunk in chunks:
    tokens = estimate_tokens(chunk["content"])
    chunk["estimated_tokens"] = tokens
    if tokens > MAX_CHUNK_TOKENS:
        print(f"WARNING: {chunk['chunk_id']} has {tokens} estimated tokens")

Don't fail the build on large chunks — some sections are legitimately long. Warn so maintainers can investigate.

Step 4: Write the corpus and manifest

Output the chunks as JSON along with a manifest file containing build statistics for monitoring corpus drift:

import json
from datetime import datetime, timezone

def write_corpus(chunks: list[dict], output_dir: str):
    # Write chunks
    with open(f"{output_dir}/rag-chunks.json", "w") as f:
        json.dump(chunks, f, indent=2)

    # Write manifest with statistics
    total_tokens = sum(c["estimated_tokens"] for c in chunks)
    manifest = {
        "generated_at": datetime.now(timezone.utc).isoformat(),
        "total_chunks": len(chunks),
        "total_lessons": len(set(c["lesson_id"] for c in chunks)),
        "total_estimated_tokens": total_tokens,
        "avg_tokens_per_chunk": round(total_tokens / len(chunks), 1),
    }
    with open(f"{output_dir}/rag-manifest.json", "w") as f:
        json.dump(manifest, f, indent=2)

The manifest is your build-over-build comparison tool. Track total chunks and average tokens — sudden changes indicate content structure issues.

Examples

Input markdown:

This lesson covers Git basics.

## Branching

Create branches with `git checkout -b feature`.

## Merging

Merge with `git merge feature`.

Output chunks:

chunk_id chunk_index heading_path content
git-basics-0 0 "Git Basics" "This lesson covers Git basics."
git-basics-1 1 "Git Basics > Branching" "## Branching\n\nCreate branches..."
git-basics-2 2 "Git Basics > Merging" "## Merging\n\nMerge with..."

Applicability

Heading-based chunking works when:

Consider fixed-size windows instead when:

Related Lessons

Related Lessons