Managing Small Context Windows in Language Models

In real-world LLM engineering, massive context windows come with severe operational challenges: soaring API costs, unacceptable response times, and the "lost in the middle" problem where the model ignores critical details buried inside a massive prompt.

Working with small, intelligently managed context windows yields superior outcomes: it reduces latency, keeps costs flat, and forces the LLM to focus strictly on relevant tokens.

In this article, you will learn three widely adopted strategies for managing small context windows, along with runnable Python implementations.

Key Strategies Covered

1. Context Truncation via Sliding Windows: How treating context as a FIFO queue keeps token usage flat and predictable.
2. Token Budgeting & RAG: How splitting the context window into strict zones prevents token bloat when retrieving external documents.
3. Advanced Techniques: A concise breakdown of rolling summaries, prompt compression, and observation masking for complex agents.

Strategy 1: Context Truncation (Sliding Window)

The sliding window approach treats conversation history as a First-In, First-Out (FIFO) queue. As new user and AI interactions arrive, the oldest turns are dropped once a predefined limit (max_turns) is reached.

Why Use Sliding Windows?

• Predictable Latency & Costs: Token count stays flat regardless of how long the conversation runs.
• Simplicity: Easy to implement without extra API overhead or external model calls.

Python Implementation: Sliding Window Memory

class SlidingWindowMemory:
    def __init__(self, max_turns=3):
        """Keep only the last `max_turns` of a conversation."""
        self.max_turns = max_turns
        self.history = []

    def add_interaction(self, user_text, ai_text):
        self.history.append({"user": user_text, "ai": ai_text})
        # Drop the oldest turns if limits are surpassed
        if len(self.history) > self.max_turns:
            self.history = self.history[-self.max_turns:]

    def build_prompt(self, new_query):
        prompt = "System: Answer concisely based on recent context.\n\n"
        for turn in self.history:
            prompt += f"User: {turn['user']}\nAI: {turn['ai']}\n"
        prompt += f"User: {new_query}\nAI:"
        return prompt


# --- Testing the Sliding Window mechanism ---
memory = SlidingWindowMemory(max_turns=2)

# Simulating a conversation sequence
memory.add_interaction("Hi, I'm learning Python.", "Great choice!")
memory.add_interaction("What are lists?", "Lists are mutable arrays.")
memory.add_interaction("Can they hold mixed types?", "Yes, they can.")

# The prompt will only contain the last 2 turns
print(memory.build_prompt("How do I append to one?"))

Strategy 2: Token Budgeting & RAG

When building Retrieval-Augmented Generation (RAG) systems, retrieving too many chunks can easily overwhelm small context windows.

Token Budgeting solves this by dividing the available prompt allocation into explicit "budget zones":

• System Prompt Zone: e.g., 20% budget
• Chat History / Query Zone: e.g., 20% budget
• Retrieved Chunks Zone: e.g., 60% budget

As document chunks are added to the prompt, the system dynamically checks the remaining word or token count and stops packing context the moment the limit is reached.

Python Implementation: Budgeted Prompt Construction

def build_budgeted_prompt(system_prompt, retrieved_chunks, user_query, max_words=50):
    """Packs context chunks into a prompt until a strict word budget is hit."""
    # Fixed baseline cost
    base_words = len(system_prompt.split()) + len(user_query.split())
    current_words = base_words
    included_chunks = []

    for chunk in retrieved_chunks:
        chunk_words = len(chunk.split())
        # Only add chunk if it fits strictly inside the remaining budget
        if current_words + chunk_words <= max_words:
            included_chunks.append(chunk)
            current_words += chunk_words
        else:
            print(f"Budget hit! Left out {len(retrieved_chunks) - len(included_chunks)} chunk(s).")
            break

    context_str = "\n---\n".join(included_chunks)
    return f"{system_prompt}\n\nContext:\n{context_str}\n\nUser: {user_query}"


# --- Testing the Budgeted Prompt Mechanism ---
system_msg = "Use the context to answer."
query = "What is the capital of Spain?"
docs = [
    "Seville is a city in Andalusia, Spain.",
    "Madrid is the capital of Spain.",
    "Spain is located in Southwestern Europe.",
    "The population of Spain is roughly 47 million."
]

# Run with a tight budget to observe chunk exclusion
print(build_budgeted_prompt(system_msg, docs, query, max_words=30))

Strategy 3: Specialized Techniques

For complex AI workflows and autonomous agents, consider these additional context reduction methods:

1. Rolling Summaries : An auxiliary model periodically condenses older chat turns into a high-level summary block. Preserves long-term memory, but introduces extra API latency and cost.
2. Prompt Compression : Algorithmic removal of filler words, redundant tokens, and stop words prior to model input. Lowers latency directly, but aggressive pruning risks losing subtle semantic context.
3. Observation Masking : Hides raw structural noise (e.g., SQL queries, verbose logs) from agent trajectory histories. Keeps LLM agents focused on reasoning, but requires domain-specific masking logic.

Summary & Next Steps

Small context windows are not an obstacle—they are an architectural feature that encourages faster inference, lower compute bills, and cleaner attention mechanisms. By implementing Sliding Windows for ongoing chats and Token Budgeting for RAG pipelines, you can build production-ready LLM systems that stay fast and economical at scale.