Chunk With Overlap: Stop Losing Meaning at the Boundary

Your RAG pipeline retrieves the right document, the embedding model is fine, the vector database is fine — and the answer still comes back cut off mid-thought. Nine times out of ten, the culprit isn't the model. It's how you sliced the document before it ever reached the embedding model.

The problem: embeddings only see one chunk at a time

An embedding model doesn't read your whole document. It reads whatever chunk you hand it, in isolation, and turns that chunk into a vector. Split the document at a naive fixed offset — every 1000 characters, no matter what's on the page — and you will occasionally slice straight through the sentence that answers the user's question. Half of it ends up in chunk 4, the other half in chunk 5, and neither chunk's embedding captures the full idea. At query time the retriever can only return whole chunks, so the model gets half an answer and either hedges or hallucinates the rest.

The fix: overlap between chunks

Make consecutive chunks share a slice of text. Instead of chunk boundaries being hard cuts, each chunk repeats the tail end of the previous chunk. If a sentence gets split by the primary cut, there's a good chance the full sentence still shows up intact inside the overlap region of a neighboring chunk — so at least one embedding fully represents that idea.

A commonly used starting point:

chunk_size = 1000      # characters per chunk
chunk_overlap = 200    # characters shared with the previous chunk

That's roughly a 20% overlap — a reasonable default to tune from, not a rule to follow blindly.

How to actually do it: use a recursive splitter, not a fixed cut

Overlapping a naive fixed-size splitter still cuts wherever the character count lands — mid-word, mid-sentence, wherever. A recursive character splitter tries the cleanest possible break first and only falls back to a rougher one if the chunk is still too big:

1. Try splitting on paragraph breaks first.
2. If a "paragraph" is still too large, fall back to single newlines.
3. If that's still too large, fall back to splitting on spaces.
4. Only as a last resort does it cut mid-word.

Most cuts land on natural document boundaries, and overlap absorbs the ones that don't. (The library and class name will vary by stack — recursive splitting with overlap is the concept that matters, not the specific package.)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
)
chunks = splitter.split_text(document_text)

Tuning per corpus

• Long, dense documents (PDFs, legal text, manuals) often need a larger chunk size so a chunk still contains a complete idea.
• Short, structured content (FAQs, product cards, changelog entries) usually wants smaller chunks — don't pad a 100-word answer to fill a 1000-character window and dilute its embedding.
• If answers still feel cut off mid-thought after switching to a recursive splitter, raise chunk_overlap before touching chunk_size.

Power tricks

• Overlap isn't free. More overlap means more total chunks for the same document — more embeddings to compute and store. Start at ~20% and adjust based on observed failures, not "just in case."
• Overlap doesn't fix a wrong chunk size. If chunks are consistently too small to hold a complete idea, raise chunk size, not overlap.
• Character count isn't semantic completeness. Structured content like tables or code blocks can still get cut awkwardly — consider a structure-aware splitter if that's a big part of your corpus.

Resources

• LangChain — Recursive character text splitting
• Pinecone — Chunking strategies for LLM applications