1 min read
Sample post — placeholder content
What my first RAG pipeline taught me about chunking
I assumed retrieval quality was about embeddings. Three rebuilds later: it's mostly about what you feed them.
My first RAG pipeline had a beautiful vector database and terrible answers. The culprit wasn't the model or the embeddings — it was 512-token chunks that sliced tables in half and separated headings from the paragraphs that explained them.
Chunk along meaning, not tokens
Documents have structure: sections, lists, tables, captions. Splitting on token counts throws that structure away. Splitting on structure — then merging small neighbors up to a budget — kept every chunk self-explanatory.
- Parse structure first (headings, tables, lists) — then chunk.
- Prepend the heading path to every chunk: context survives retrieval.
- Measure retrieval hit-rate on a labeled set before touching the prompt.
The eval numbers told the story: structural chunking moved retrieval hit-rate more than swapping embedding models did. Measure retrieval on its own — if the right chunk never reaches the context window, no prompt can save you.