LangChain Data Handling — Loaders and Splitters (Python)

LangChain Data Handling — Loaders and Splitters (Python)

Overview

You have a RAG system over a Python docs site. A user asks "what does trim_messages do?" and the retriever returns this chunk:

### `trim_messages(strategy="last", include_system=True)`

Trim a message history to fit a token budget. The newest messages are kept;
older messages are dropped. Pass `include_system=True` to preserve the system

...and that's it. The chunk ends there. The code example showing the function body — the actual thing the user wanted — is in a different chunk, retrieved with a lower similarity score and dropped before the LLM sees it. The model then hallucinates the function's behavior from the signature alone.

This is pain-catalog entry P13. RecursiveCharacterTextSplitter's default separators are ["\n\n", "\n", " ", ""]. It splits on any blank line — including inside triple-backtick code fences in Markdown. The fix is a one-line swap to RecursiveCharacterTextSplitter.from_language(Language.MARKDOWN), which treats the fence as an atomic unit, but you have to know the bug exists.

The sibling failures this skill prevents:

• P49 — PyPDFLoader splits by page. A 5-row financial table that spans a page break gets torn in half; rows 1-3 go in one chunk, rows 4-5 in another with no header. A RAG answer sourced from the second chunk misquotes the numbers because the column meanings are in the first chunk. Fix: use PyMuPDFLoader or UnstructuredPDFLoader, which detect tables and emit them as distinct structured elements.
• P50 — WebBaseLoader's default User-Agent is python-requests/2.x. Cloudflare-protected sites flag this as a bot and return a 403 interstitial HTML page ("Checking your browser...") instead of real content. The crawler indexes the challenge page. You notice weeks later when every retrieval from that source returns the same Cloudflare text. Fix: set a realistic header_template={"User-Agent": "Mozilla/5.0 ..."}, respect robots.txt, and rate-limit per-host to 1 req/sec.