Consider two engineers indexing the same HR handbook for a manufacturer with plants in Hosur, Pune and Sanand. One build answers leave questions correctly. The other mixes up the casual leave allowance for Pune with the one for Hosur, because the two sections sit a page apart and the split separated each allowance from its plant heading. Same model, same embeddings, same PDF; the only difference is where the text was cut.
The model only ever sees the chunks
A chunk is the unit the index stores and retrieval returns. In a RAG pipeline, the model never sees the handbook. It sees the five or six chunks that came back for this question and nothing else.
If the sentence that answers the question sits across a chunk boundary, the model receives half of it. If the chunk has lost the heading that said which plant it applies to, the model receives a rule with no scope. In both cases it still produces a fluent answer, because producing fluent text from whatever it is given is what it does. The model does not know a heading is missing. It only sees the chunk.
So chunking decides which questions the system can answer well. That makes it a product decision rather than a library setting, and it depends on the documents you hold and the questions people actually ask. We would rather spend a day reading chunks than a week comparing embedding models.
What fixed-size splitting breaks
The simplest splitter cuts every so many characters. It is fast, works on any text, and is the default in most frameworks. It also cuts through sentences, lists and tables, and it drops the heading above the cut.
- Cuts a table across two chunks
- Separates the allowance from the plant heading
- Overlap repairs some cuts, never the lost heading
- Splits at headings, then paragraphs
- Keeps a small table whole with its caption
- Prepends the heading path to every chunk
A structure-aware splitter is not a product you buy. It is a short set of rules written for your documents: where the headings are, what a table looks like once the text is extracted, which files need different treatment. For a handbook like this one it is a morning's work, and it changes the answers more than a model swap would.
Headings, tables and PDFs need their own rules
Headings
Headings carry the context a paragraph assumes. Here is a chunk from the fixed-size build:
"Employees are entitled to twelve days of casual leave per calendar year, credited quarterly."
On its own, that sentence is true for one plant and false for the other two. The structure-aware build stores the same text with its heading path in front:
"HR handbook > Leave > Pune plant > Casual leave. Employees are entitled to twelve days of casual leave per calendar year, credited quarterly."
Prepend the heading path before embedding, so both the vector and the model see it. It costs a few tokens per chunk, and for policies and manuals it is the most effective single change we know of.
Tables
A table row means nothing without its header, and a table split across two chunks cannot be reassembled by the model. Keep small tables in one chunk with their caption and header row. For large tables, such as a shift allowance schedule across grades and plants, write one chunk per row, with each row expressed as column and value pairs and the table title in front.
PDFs
PDF extraction returns text in the order the file stores it, which is not always reading order. Two-column layouts interleave, footers land in the middle of a paragraph, and a scanned circular contains no text at all until OCR runs. Read the extracted text for a sample of files before chunking anything. Nothing downstream can repair a scrambled page.
Overlap and metadata
Overlapping chunks repeat the last few sentences of the previous chunk, so a fact near a boundary appears whole in at least one of them. The cost is a larger index and the chance of retrieving two near-identical chunks for one question, so deduplicate by document and position before sending them to the model. Overlap patches boundary cuts. It does not replace splitting in the right place.
Metadata is what makes a chunk usable after retrieval: document id, version, section path, owner, date, language and audience. Retrieval can filter on these fields, so a question from a Pune employee gets only the Pune policy and only the current version, and the answer can cite a section instead of an unlabelled block of text. Decide the fields before indexing. Adding one later means re-indexing every document, including every circular issued since the handbook was first written.
A chunking rule by document type
Split at the smallest unit a person would quote to answer a question, and attach whatever that person would need to read the quote on its own.
| Document type | Split at | Keep with each chunk |
|---|---|---|
| Policies, manuals | Headings, then paragraphs | Heading path, version |
| Tickets, chat threads | One ticket or thread | Product, status, date |
| Contracts | One clause | Clause number, parties, dates |
| Spreadsheets, large tables | One row | Table title, header row |
| Long reports | Section, then paragraphs with overlap | Section path, figure captions |
When a document type is not on the list, ask its owner how they look things up in it. If the HR manager says "I search by circular number", the circular is the chunk and its number is metadata.
How chunking goes wrong
The most common mistake is accepting the framework default and never reading a single chunk. The second is judging chunking by reading answers instead of the retrieved chunks. The failure looks like a model problem, a fluent answer with the wrong number in it, so someone goes looking for a larger model when the fix was a heading.
Teams also re-chunk without a saved set of questions, so nobody can say whether the change helped. They apply one splitter to a corpus that mixes contracts, support tickets written in Hinglish and slide decks, each of which needs a different rule. And they forget that any chunking change invalidates the index. Every change means a full re-index and a re-run of the evaluation set, which is a reason to keep that evaluation set ready (the method is in measuring a RAG system).
Read ten chunks before tuning anything
- Take ten real questions from the intranet search log or the helpdesk queue.
- Run retrieval alone for each and print the chunks that come back.
- Read each chunk as the model would, with no document around it, and note what is missing.
- Turn each gap into a rule: a heading to prepend, a table to keep whole, a file type to OCR.
- Re-index, re-run the same ten, and compare.
The gaps you find in step three are the chunking rules you need to write. In the handbook above, the plant mix-up shows up on the second question.
