LangChain Rate Limits (Python)

LangChain Rate Limits (Python)

Overview

A team deploys 10 Cloud Run workers. Each worker initializes its ChatAnthropic with InMemoryRateLimiter(requests_per_second=10) — they read the docs, they picked a safe-looking number, they shipped. Thirty seconds later the dashboard lights up with 429s: the cluster is pushing 100 RPS to Anthropic's 50 RPM tier-1 ceiling, not the 10 RPS they configured. The name is the fix — InMemoryRateLimiter is in-process. Each worker has its own counter. Ten workers × 10 RPS = 100 RPS to the provider. This is pain-catalog entry P29 and it lands on every team that scales past one pod.

Three more traps wait on the same code path:

• P07 — .with_fallbacks([backup]) defaults exceptions_to_handle=(Exception,), which on Python <3.12 swallows KeyboardInterrupt. Ctrl+C during a 429 retry storm silently falls through to the backup chain and keeps billing.
• P30 — ChatOpenAI and ChatAnthropic default max_retries=6. That is retries, not attempts: 7 total requests per logical call on flaky networks. One .invoke() can bill 7x.
• P31 — Anthropic's RPM counts cache reads, cache writes, and uncached calls uniformly. Cache-heavy workloads at 50 RPM can 429 on cache writes while the ITPM dashboard shows headroom.

This skill covers measuring demand before picking a limit; the InMemoryRateLimiter vs Redis-backed limiter vs asyncio.Semaphore decision tree; the narrow exceptions_to_handle whitelist; max_retries=2 math; and the provider-specific limit taxonomy (RPM, ITPM, OTPM, concurrent, cached-vs-uncached). Pin: langchain-core 1.0.x, langchain-anthropic 1.0.x, langchain-openai 1.0.x. Pain-catalog anchors: P07, P08, P29, P30, P31. For .batch(max_concurrency=...) tuning, see the sibling skill langchain-performance-tuning — this skill is about provider-facing rate caps.

Prerequisites

• Python 3.10+ (3.12+ fixes the KeyboardInterrupt half of P07)
• langchain-core >= 1.0, < 2.0
• At least one provider: pip install langchain-anthropic langchain-openai
• For multi-worker prod: redis >= 4.5 client and a Redis server reachable from every worker