What Is an LLM Context Window? Why Long Threads Get Cut

What is an LLM context window? It is the fixed amount of text — measured in tokens — that a language model can read and write during a single interaction. Everything the model uses to generate a response has to fit inside that window: the system instructions, the email thread, any retrieved context, and the output it produces. When the thread is long enough to overflow that budget, the model cannot see the whole conversation at once, and the oldest messages get dropped.

That limitation is why an AI assistant can draft a perfectly grounded reply to a short exchange but miss a commitment made ten messages back in a sixty-message negotiation thread. The problem is not intelligence — it is arithmetic. The model never saw those earlier messages because they did not fit.

This guide defines the context window precisely, explains why email threads are a particularly hard case, covers what breaks when the window fills, and clears up the misconceptions that lead people to expect AI email tools to behave as if they have unlimited memory.

How does an LLM context window work?#

A language model does not read your email the way you do. It converts text into tokens — roughly equivalent to word fragments — and processes a sequence of those tokens in a single pass. The context window is the maximum number of tokens that sequence can contain. Every token in the prompt, the thread history, the system instructions, and the generated reply counts against that limit.

Tokens are not the same as words. Common short words are often a single token each, but longer or unusual words can span two or three tokens. A rough rule of thumb: 100 words is approximately 130 tokens in English prose. A 60-message email thread with quoted text in each reply can run to 15,000 or 20,000 tokens — well above the range at which a model delivers reliable output, even when the model's nominal window is larger.

Context windows have grown substantially since 2023. Models that once capped at 4,000 tokens now commonly support 128,000 or more. But a larger window is not a free pass. Output quality on tasks that require attending to specific information degrades as the window fills with irrelevant material, retrieval gets slower and more expensive at scale, and a 60-message thread with full quoted-text chains still pushes against practical limits. Larger windows reduce truncation; they do not eliminate the tradeoffs.

Why do long email threads fill the context window so fast?#

Email threads are a structurally difficult case for context windows, for three reasons that compound each other.

The first is quoted text. Standard email clients include the full previous message every time someone replies. A ten-message chain does not contain ten messages worth of text — it contains ten messages worth of text plus nine copies of the prior message, plus eight copies of the one before that. Token count grows faster than message count. A 20-message thread with standard quoting can easily run to 8,000 tokens before attachments or signatures.

The second is inline content. An email that pastes a contract excerpt, references a long document, or includes a newsletter digest does not add a little weight to the context — it can add thousands of tokens in a single message.

The third is that the thread is not the only thing that needs to fit. The model also needs room for its system instructions, the user's preferences, any retrieved contact history, and the output it is going to write. A model with a 128,000-token window that has already used 80,000 tokens on thread history and instructions has only 48,000 left for reasoning and output — and the first messages in the thread are the ones that get dropped when the budget runs out.

Tokens vs. words, and what counts against the context budget#

What breaks when the context window is full?#

When the token budget is exhausted, something has to give. Most systems respond by truncating: they drop the oldest messages first and present only the most recent portion of the thread to the model. That approach preserves the immediate back-and-forth but loses anything said earlier. A price agreed in message three of a forty-message thread is invisible to a model that can only see messages thirty through forty.

The failure is often invisible. The model generates a reply that sounds plausible and confident, grounded in the recent messages it can see. Only someone who remembers what was said earlier notices the omission. This is one of the conditions that produces AI hallucinations in email drafts: the model fills the gap left by truncated context with plausible-sounding but invented detail rather than admitting it does not know.

Systems that handle truncation well do not simply drop early messages. They strip quoted text before indexing so each message contributes only its unique content, summarize older parts of the thread into a compact representation, and use selective retrieval to pull back only the earlier messages relevant to the current reply. A user can usually tell whether a summary was built from a full thread by asking the AI about something said in the first few messages — if it cannot answer accurately, the context was truncated.

Common misconceptions about context windows and email AI#

The most common misconception is that a bigger context window means better answers. A larger window reduces the chance of truncation, but it does not improve reasoning quality on the content that fits. Models are not uniformly attentive across a large context — material in the middle of a very long window receives less effective attention than material at the start or end, a pattern the research community calls the lost-in-the-middle effect. A well-designed system that uses selective retrieval to keep only the relevant material in the window often outperforms one that floods a large window indiscriminately.

A second misconception is that the context window is the same as the model's memory. It is not. The model's training gave it general knowledge about language, conventions, and the world. The context window is what it can see right now, in this specific interaction. Training knowledge persists across every conversation; context window content is fresh each time. An AI that appears to remember your preferences across separate sessions is not using a persistent context window — it is loading stored information into a new window on each call.

A third misconception is that context window limits are a temporary problem that will disappear as models scale. Windows have grown dramatically and will likely keep growing, but the fundamental tradeoffs — cost, latency, and the attention quality problem — do not vanish with size. Retrieval-based approaches that keep only relevant content in the window remain useful even as nominal limits increase, because relevance filtering is cheaper and more precise than context-stuffing.

How AI Emaily handles long email threads#

The context window problem is one AI Emaily is built to handle. When you reply to a long thread, the system strips quoted text so each message contributes only its unique content, summarizes older exchanges into a compact representation, and retrieves only the earlier messages that are semantically relevant to the current draft — keeping the context budget focused on what matters for this reply rather than loading sixty messages verbatim. The result is a draft grounded in your full thread history without overwhelming the window. We build AI Emaily.