Modern multi-agent systems, especially those leveraging large language models (LLMs), often span multiple services and toolsets. Observing these distributed workflows end-to-end is crucial yet challenging because each agent may invoke different tools and models independently. Without a unified tracing solution, understanding which agent did what, when, and with which model becomes a puzzle.

This post unpacks how to leverage OpenTelemetry's standardized tracing protocol combined with Microsoft Foundry’s agent framework integrations to achieve distributed observability across multi-agent systems. We examine how trace context propagates seamlessly across agents, how GenAI-specific semantic conventions standardize telemetry, and practical ways to implement tracing in a simulated incident drill scenario.

We'll walk through the architectural design, key technical patterns, and concrete examples of observability in action—culminating in a powerful unified trace that reveals the entire agentic workflow from root cause to resolution.

---

Architecture Overview

┌─────────────────────────────────────────────┐
│           Simulated Outage Event             │
├─────────────────────────────────────────────┤
│         Incident Commander Agent             │
│  • Uses model-router deployment              │
│  • Orchestrates specialist agents            │
└─────────────────────────────────────────────┘
                      ↓
┌─────────────────────────────────────────────┐
│               Specialist Agents              │
├─────────────────────────────────────────────┤
│  • Logs Agent (query_logs)                   │
│  • Metrics Agent (query_metrics)             │
│  • Runbook Agent (lookup_runbook)            │
│  • Each runs on gpt-5-mini deployment        │
└─────────────────────────────────────────────┘
                      ↓
┌─────────────────────────────────────────────┐
│                Azure Monitor                 │
├─────────────────────────────────────────────┤
│  • Receives OTLP telemetry                  │
│  • Visualizes traces, metrics, and logs     │
│  • Monitors token consumption, tool calls   │
└─────────────────────────────────────────────┘

Diagram adapted from the original multi-agent game-day crew illustration by Will Velida on DEV Community.

---

Key Technical Observations

• Trace Context Propagation via W3C Standard
  Agents propagate trace context using the W3C Trace Context standard, ensuring child spans share the same trace ID but use unique span IDs. This avoids fragmentation of traces across agent boundaries, keeping the entire multi-agent workflow connected.

• Use of Baggage for Contextual Metadata Propagation
  OpenTelemetry’s Baggage allows passing custom key-value pairs alongside trace context immutable states. Attaching drill-specific metadata like drill.id or incident_id ensures consistent observability across asynchronous calls and agent interactions.

• GenAI Semantic Conventions for Standardized Attributes
  Adhering to OpenTelemetry’s GenAI semantic conventions (gen_ai.*), such as gen_ai.agent.name and gen_ai.operation.name, standardizes how AI agent telemetry is emitted. This facilitates unified dashboards irrespective of the underlying agent framework.

• Hierarchical Spans to Capture Agent Interactions and Tool Usage
  The system models agent-to-agent interactions as client spans (SpanKind.CLIENT) with child spans capturing individual agent operations, chat interactions, and tool calls. This hierarchical model offers granular visibility into multi-agent workflows.

• Microsoft Foundry’s Agent Framework with Built-in Instrumentation
  The Microsoft Agent Framework automatically instruments calls like invoke_agent, chat, and execute_tool, providing an immediate baseline for observability with minimal developer effort.

• Azure Monitor Integration via OpenTelemetry Protocol (OTLP)
  Azure Monitor can ingest telemetry either through the open source OTLP pipeline or Microsoft's OpenTelemetry Distro client, supporting comprehensive telemetry ingestion of traces, metrics, and logs with optional sensitive data scrubbing.

---

How It Works

Propagating Context Across Agents

In multi-agent workflows, a single request triggers a chain of agent calls. The span context, including trace and span IDs, must flow between agents to stitch together telemetry into one coherent trace.

The code uses a Python context manager drill_context that attaches Baggage—a set of custom key-value metadata—into the OpenTelemetry context:

@contextmanager
def drill_context(**entries: str) -> Iterator[None]:
    ctx = context.get_current()
    for key, value in entries.items():
        ctx = baggage.set_baggage(f"drill.{key}", value, context=ctx)
    token = context.attach(ctx)
    try:
        yield
    finally:
        context.detach(token)

Span creation employs get_tracer().start_as_current_span() with SpanKind.SERVER for the root span, ensuring Application Insights recognizes the transaction origin.

Child spans created within this context become part of the same trace hierarchy, connected via propagated context—even across asynchronous agent boundaries. Without this propagation, traces would fragment, making debugging nearly impossible.

The GenAI Semantic Conventions

The OpenTelemetry GenAI semantic conventions provide well-defined attribute namespaces for agent operations, including:

• gen_ai.agent.name — Agent identifier
• gen_ai.operation.name — Operation type such as invoke_agent, execute_tool, or agent_to_agent_interaction
• gen_ai.usage.* — Token usage and costs

For example, during an agent hand-off, a span models the interaction:

with get_tracer().start_as_current_span(
    "agent_to_agent_interaction",
    kind=SpanKind.CLIENT,
    attributes={
        "gen_ai.operation.name": "agent_to_agent_interaction",
        "gen_ai.agent.name": agent.name,
        "game_day.from_agent": COMMANDER_NAME,
        "game_day.to_agent": agent.name,
        "game_day.role": role.value,
    },
) as span:
    response = await agent.run(f"{brief}\n\n{SPECIALIST_ANGLE[role]}")

This structure captures who passed the task to whom, creating clarity in tracing multi-agent workflows.

OpenTelemetry and Azure Monitor Integration

To export telemetry, the sample app configures OpenTelemetry exporters conditionally, falling back to either console output or Application Insights within Azure Monitor:

async def configure(client: FoundryChatClient, *, sensitive_data: bool) -> str:
    otlp_endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "").strip()

    if otlp_endpoint or flag("ENABLE_CONSOLE_EXPORTERS"):
        configure_otel_providers(enable_sensitive_data=sensitive_data, views=_metric_views())
        destination = otlp_endpoint or "the console"
    else:
        connection_string = await client.project_client.telemetry.get_application_insights_connection_string()
        configure_azure_monitor(
            connection_string=connection_string,
            resource=create_resource(),
            enable_live_metrics=True,
            views=_metric_views(),
        )
        destination = "Application Insights, via the Foundry project connection"

    enable_instrumentation(enable_sensitive_data=sensitive_data)
    return destination

Notably, instrumentation is disabled by default in the agent framework and must be explicitly enabled to emit telemetry.

---

Building Our Observable Multi-Agent System

Beyond automatic instrumentation, tracing tool usage within agents adds vital context. For example, the Logs specialist agent records details such as the service queried and the severity filter:

@tool(approval_mode="never_require")
async def query_logs(
    service: Annotated[str, Field(description="Service name from the service map, for example 'checkout-api'.")],
    min_level: Annotated[str, Field(description="Lowest severity to return: info, warn or error.")] = "warn",
) -> str:
    key = _clean(service).lower()
    level = _clean(min_level).lower()
    _count("query_logs", key)

    span = trace.get_current_span()
    span.set_attribute("game_day.tool.service", key)
    span.set_attribute("game_day.tool.min_level", level)

    # ... query logic ...
    span.set_attribute("game_day.tool.result_count", len(lines))
    return _ok(service=key, min_level=level, lines=lines)

This enriches spans with actionable metadata, such as:

• Which service logs were read (game_day.tool.service)
• Minimum log level filter used (game_day.tool.min_level)
• Number of lines returned (game_day.tool.result_count)

This detailed telemetry enables fine-grained troubleshooting when tracing complex agent interactions.

---

Seeing It Work

Running the simulated incident drill produces an intuitive command-line summary:

$ python demo.py

[1] INC-A1C212  checkout latency spike     normal  sev1      48.5 s  drill 1b1b480ad6e2
    logs       24.9 s  Checkout-api p99 latency and error spike caused by payments-gateway connection-pool exhaustion leading to timeouts and a circuit-breaker opening.
    metrics    24.4 s  ~09:25 spike in checkout-api p99 and errors, traced to payments-gateway timeouts after a 09:20 deploy that reduced its connection pool.
    runbook    12.5 s  RB-014 (Downstream connection pool exhaustion) selected; confirm pool.max vs prior release and restore it if it was lowered.
    commander  routed to grok-4-1-fast-reasoning
    tool calls  lookup_runbook 1  query_logs 4  query_metrics 6    total 11

Within Azure Monitor’s classic Foundry portal, the entire drill trace shows as a root span with child spans representing agent-to-agent hand-offs and specialist work (see below).

Detailed span metadata confirms consistent trace IDs and reveals each agent’s timing and operation.

Meanwhile, Application Insights' Transactions view renders the drill as a single end-to-end transaction waterfall.

Clicking into model calls reveals GenAI-specific properties including token usage and latency:

Custom properties show comprehensive agent telemetry, including drill IDs and tool usage:

Operational metrics such as agent run counts, token consumption broken down by model, and tool calls can be explored in Application Insights' preview Agents view:

---

Quick Tips & Tricks

1. Always Propagate Context with W3C Trace Context Standard
   Use standardized trace context propagation to maintain trace continuity across asynchronous, remote or agent-to-agent calls.

2. Apply Baggage for Cross-Cutting Metadata
   Attach important workflow-level metadata (e.g., drill ID, scenario name) to baggage keys with a common prefix to enable easy filtering and correlation in telemetry tools.

3. Instrument Tool Calls Within Spans
   Rather than creating new spans inside tools, augment the existing execute_tool span with detailed attributes about inputs, outputs, and outcomes—this keeps telemetry unified and easier to investigate.

4. Use GenAI Semantic Conventions to Standardize AI Telemetry
   Follow established semantic conventions such as those in gen_ai.* attributes to ensure consistency and compatibility across different agent frameworks and AI providers.

5. Explicitly Enable Instrumentation in Agent Frameworks
   Remember to call enable_instrumentation() to activate telemetry emission. Framework defaults may disable this for performance or privacy reasons.

6. Disable Sensitive Data Emission in Production
   Set the enable_sensitive_data flag to false in production environments to avoid logging prompts or user data, maintaining privacy and compliance.

---

Conclusion

Observing complex multi-agent AI systems in a connected, holistic way requires more than per-agent telemetry. OpenTelemetry’s context propagation, coupled with semantic conventions for GenAI and the Microsoft Agent Framework’s integrations, provide a robust solution for distributed tracing of agent workflows.

The ability to correlate agent calls, tool usage, and token consumption in a unified trace—visible seamlessly within Azure Monitor and Application Insights—empowers engineers to diagnose issues faster, understand agent decision-making, and optimize their multi-agent architectures effectively.

As AI-enabled workflows and agentic systems grow in scale and complexity, standardized observability practices like these will be essential for maintaining reliability and transparency. The future holds promise for even richer telemetry frameworks built on these foundational standards.

---

References

1. Distributed tracing for multi-agent systems with OpenTelemetry - DEV Community — Original article analyzed
2. Game-day multi-agent incident simulator (GitHub sample) — Sample code repository
3. OpenTelemetry GenAI Semantic Conventions — Specification for GenAI telemetry attributes
4. Microsoft Foundry Observability Concepts — Official documentation on trace and agent observability
5. W3C Trace Context Standard — Protocol for distributed context propagation
6. Azure Monitor OpenTelemetry Options — Integration details for OpenTelemetry and Azure Monitor
7. Microsoft Agent Framework OpenTelemetry Integration — Agent Framework observability guide