Chapter 6: Context Windows

In Chapter 4, we saw that LLMs possess zero persistent memory between API calls. In Chapter 5, we learned how developers use system prompts to establish the rules of engagement.

Every message we send (system prompts, user questions, tool results, past conversation history) must be packaged into a single payload and submitted to the model on every single turn.

This brings us to the physical boundary of an LLM call: the context window.

What is a Context Window?

The context window is the finite working memory buffer of an LLM inference call. It is measured in tokens (sub-word fragments where 1,000 tokens equal roughly 750 English words).

Think of the context window as the active desk space available to the model for a single call:

1. Shared Capacity: The context window is shared between input tokens (the prompt you send) and output tokens (the response the model generates). If a model has an 8,192 token context window and your prompt consumes 7,000 tokens, the model can generate at most 1,192 tokens before terminating.
2. Fixed Upper Ceiling: Once a context window is full, you cannot append a single additional token without receiving an error or truncating earlier messages.
3. Stateless Re-evaluation: Because the model is stateless, every single token in the context window must be processed from scratch on every turn.

+---------------------------------------------------------------+
|                      TOTAL CONTEXT WINDOW                     |
|                                                               |
|  [ System Prompt ] [ Chat History ] [ User Query ]  [ Output ] |
|  |<---------------- Input Budget ----------------->|<-Buffer->|
+---------------------------------------------------------------+

The Illusion of Perfect Attention: "Lost in the Middle"

Modern frontier models boast context windows ranging from 128,000 to over 1,000,000 tokens. It is tempting to believe that you can dump hundreds of pages of documentation, source code, and transcripts into the prompt and expect the model to reason across all of it flawlessly.

In production, this assumption breaks down due to attention degradation.

Transformers rely on self-attention mechanisms to calculate relationships between tokens. Across very long sequences, attention is not uniformly distributed. Instead, models exhibit a U-shaped recall curve known as the Lost in the Middle phenomenon:

• Primacy Effect (High Recall): Information placed at the very beginning of the prompt (the system prompt and opening instructions) receives sharp, focused attention.
• Recency Effect (High Recall): Information placed at the very end of the prompt (the latest user turn and immediate instructions) receives high attention.
• The Middle Dip (Low Recall): Critical facts, constraints, or retrieved documents buried in the middle of a massive prompt are significantly more likely to be overlooked or hallucinated away.

Engineering Rule: Position for Precision

If a constraint or piece of data is critical to the success of the call, never bury it in the middle of a long document dump. Place your overarching rules in the system prompt at the start, and place your immediate task directive at the very end.

The Economic and Latency Costs of Long Context

Context windows are not free. Expanding your prompt size introduces two linear penalties:

1. Cumulative Token Cost

API providers bill separately for input tokens and output tokens. In a multi-turn chatbot or autonomous agent loop, your input payload grows with every turn.

If your conversation accumulates 10,000 tokens per turn:

• Turn 1: You pay for 10,000 input tokens.
• Turn 2: You pay for 20,000 input tokens.
• Turn 3: You pay for 30,000 input tokens.
• Turn 10: You pay for 100,000 input tokens.

By turn 10, you have paid for 550,000 input tokens across the session, even if the user only typed a single sentence on each turn.

2. Time To First Token (TTFT) Latency

Before an LLM can emit its first word, it must compute attention across all input tokens (the "pre-fill" phase). A request with 1,000 input tokens might begin responding in 200 milliseconds; a request with 100,000 input tokens might take several seconds before generating a single character.

Strategies for Managing Context Pressure

Because context is finite, costly, and attention-degraded, production hosts implement active memory management strategies:

Strategy 1: Sliding Window (FIFO)
[ System ] [ Drop Oldest Turn ] [ Turn N-1 ] [ Turn N ] [ New Query ]

Strategy 2: Rolling Summarization
[ System ] [ Compact Summary of Turns 1..10 ] [ Turn 11 ] [ New Query ]

Strategy 3: Selective Retrieval (RAG)
[ System ] [ Top 3 Relevant Excerpts ] [ New Query ]

1. Sliding Window (FIFO Truncation)

The host code maintains only the last $N$ turns of dialogue in the request array, discarding older messages once the token count exceeds a threshold. This keeps costs predictable, though the agent will forget events that occurred earlier in the conversation.

2. Rolling Summarization

When the history array reaches a limit, the host triggers a background LLM call to summarize the older conversation into a concise paragraph, appends the summary beneath the system prompt, and trims the raw messages.

3. Selective Retrieval (RAG)

Instead of stuffing an entire document into the context window, the host queries a local search index to find the 2 or 3 most relevant paragraphs and inserts only those excerpts into the prompt.

Now that we understand how single requests are constructed, shaped, and bounded by context, we turn to capabilities. In Chapter 7, we explore What Can a Single Call Do?