AI agents are transforming enterprise workflows, delivering automation and intelligence at scale. However, as these agents routinely interact with large language models, the cost of managing token consumption and context relevancy escalates quickly. Without smart optimization, organizations risk ballooning AI expenses that can undermine project ROI and stall AI adoption.

Enter Microsoft Foundry’s approach to context engineering—a strategic method to precisely control what an AI agent knows, can access, and remembers. Context engineering focuses on balancing token usage with the quality of interactions to lower costs and improve agent performance. This approach is critical for enterprises building complex AI workflows that must handle rich, variable enterprise data while also preserving responsiveness and accuracy.

In this post, we’ll unpack the economics behind agent context windows, describe what context engineering entails in practice, and explore how Microsoft Foundry’s platform enables scalable AI agent optimization. Whether you’re developing intelligent automation, AI-powered assistants, or domain-specialized agents, these principles will help you create efficient, cost-effective AI solutions.

Architecture Overview

┌─────────────────────────────────────────────┐
│           Enterprise Data Sources           │
├─────────────────────────────────────────────┤
│ • Databases (Azure SQL, Cosmos DB, PostgreSQL) │
│ • Documents & Knowledge Bases (Foundry IQ)  │
│ • Operational Systems & APIs                 │
└─────────────────────────────────────────────┘
                     ↓
┌─────────────────────────────────────────────┐
│          Microsoft Foundry Platform          │
├─────────────────────────────────────────────┤
│ • Model Access & Optimization (Foundry Models) │
│ • Agent Factory & Runtime (Foundry Agent Service) │
│ • Memory & Context Management (Procedural, Session, User Memory) │
│ • Semantic Caching & Token Rate Limiting      │
└─────────────────────────────────────────────┘
                     ↓
┌─────────────────────────────────────────────┐
│           AI Applications & Agents           │
├─────────────────────────────────────────────┤
│ • Internal AI Assistants                     │
│ • Automation Workflows & Tooling             │
│ • Customer-Facing Conversational Agents      │
└─────────────────────────────────────────────┘

This architecture shows how enterprise data sources are integrated into a cohesive AI platform via Microsoft Foundry. Foundry acts as the bridge between raw data and AI agents, managing models, memories, and optimization layers that intelligently curate context for each interaction.

Microsoft Foundry enables cost-efficient AI agent development and optimization — source: Microsoft Azure Blog

Key Technical Observations

• Context Window Directly Impacts Token Costs — The length and content of the context window dictate token consumption in language model interactions. Optimizing what data fits within this limited window is essential for cost control.

• Multi-Layered Memory Architecture — Foundry distinguishes between session memory (current conversation state), user memory (persistent facts and preferences), and procedural memory (learned workflows). This separation enables context persistence without inflating prompt size unnecessarily.

• Semantic Caching Reduces Repeated Query Costs — By caching semantically similar queries and their results, Foundry avoids redundant token usage for identical or overlapping information requests, cutting query costs and latency.

• Toolbox Abstraction for Agent Workflows — Agents access tools via a single endpoint with tool search enabled, allowing flexible, modular capabilities with cost control over tool activation and data retrieved.

• Token Rate Limiting Enforces Budget Discipline — Foundry’s token rate limiting mechanism dynamically restricts usage based on defined cost thresholds, preventing runaway charges while maintaining agent responsiveness.

• Data Grounding via Foundry IQ — Connecting knowledge bases to agents dramatically improves grounding and reduces token bloat by selectively surfacing relevant enterprise data, preventing agents from consuming costly irrelevant information.

How It Works

Understanding the Context Window Economics

Token usage in AI interactions largely depends on how much information you feed the model at once—the context window. Larger windows mean more context but exponentially higher token costs. The core challenge is balancing what the agent needs to know with what fits efficiently in a request.

Microsoft Foundry addresses this by engineering context at multiple granularities:

• Including only relevant enterprise data to ground responses
• Using procedural memory for common workflows, minimizing repeated context
• Employing session and user memory to persist essential conversational states without re-prompting irrelevant details each time

The result is a lean, optimized context that lowers token consumption and improves response quality.

Practical Context Engineering: What Does the Agent Know?

Context engineering starts by defining the essential knowledge an agent must have to perform its task. This involves:

• Selecting trusted enterprise datasets to feed into Foundry IQ knowledge bases
• Indexing critical documents and systems selectively rather than wholesale ingestion
• Setting clear boundaries on knowledge recency and relevance to avoid signal noise

This targeted knowledge curation ensures that agents are informed but not overwhelmed, saving tokens and reducing hallucinations.

Configuring toolboxes in Foundry allows modular access for agents — source: Microsoft Azure Blog

What Should the Agent Be Able to Reach?

Beyond static knowledge, agents need tool access to execute tasks. Foundry provides a toolbox abstraction that centralizes tools at a single endpoint. Enabling tool search dynamically selects the right capability, avoiding token-heavy redundant descriptions.

This approach:

• Simplifies integration of new tools
• Minimizes token overhead by invoking only the relevant tool documentation
• Enables better cost control through limiting tool access

How Should the Agent Do the Work?

Procedural memory stores workflow patterns and task execution knowledge that agents use internally. This moves common instructions out of the prompt and into a reusable memory store, making agents faster and cheaper.

Procedural memory allows agents to:

• Automate predictable steps without re-encoding instructions every session
• Adapt workflows dynamically based on task success or user input
• Be more modular, with reusable subroutines encoded in memory

What Should the Agent Remember?

Session memory holds the current conversation state, while user memory stores preferences and persistent facts across sessions:

• Session Memory preserves context locally to keep conversations coherent.
• User Memory allows personalization and continuity across interactions without reinjecting prompt tokens repeatedly.

Together, they reduce token overhead while improving agent engagement quality.

Quick Tips & Tricks

1. Prioritize Context Relevance Over Volume — Avoid dumping large documents; instead, index and surface only the data relevant to the immediate query to conserve tokens.

2. Leverage Procedural Memory for Repeated Tasks — Encode workflows into procedural memory so agents don’t reprocess instructions every session, cutting prompt size drastically.

3. Use Semantic Caching to Cut Redundant Queries — Enable caching layers to return previous answers for semantically similar prompts, enhancing speed and reducing token spend.

4. Centralize Tool Access via Toolboxes — Group agent capabilities behind a toolbox endpoint, enabling dynamic tool selection and reducing token cost for tool descriptions.

5. Implement Token Rate Limiting Rules — Set up dynamic controls to avoid runaway token usage and control costs without sacrificing user experience.

6. Ground Agents with Foundry IQ Knowledge Bases — Connect targeted enterprise data using Foundry IQ to improve accuracy and reduce prompt tokens compared to ad hoc retrieval.

Conclusion

Context engineering is the cornerstone for building cost-effective, high-quality enterprise AI agents. By carefully calibrating what an agent knows, can do, and remembers, Microsoft Foundry communities can substantially lower token costs while improving agent relevance and memory. Context windows are expensive resource constraints—but with tools like semantic caching, procedural memory, and streamlined toolboxes, those costs become manageable.

As enterprises continue scaling AI agent usage, serving increasingly complex workflows with multiple data sources, context engineering will evolve into a comprehensive system—one that integrates data grounding, dynamic memory management, and intelligent tooling. Microsoft Foundry positions itself uniquely to power this transformation, enabling both innovation and cost discipline in enterprise AI.

References

1. AI agent optimization: How context engineering lowers AI costs | Microsoft Azure Blog — Original blog post detailing Microsoft Foundry’s context engineering approach.
2. Microsoft Foundry — Explore the Foundry platform powering enterprise AI agent optimization.
3. Foundry Agent Service — Details on the agent runtime and memory management service.
4. Foundry IQ — Knowledge base integration service grounding AI agents in enterprise data.
5. Azure OpenAI in Foundry Models — Model management and access within Foundry leveraging Azure OpenAI.
6. Azure Kubernetes Service (AKS) — Infrastructure supporting scalable deployments of enterprise AI workloads.