Evaluating AI agents against specific behavioral requirements remains a core challenge for developers and product teams. While most AI teams write down expected behaviors in natural language—whether in product specs, policy documents, or system prompts—translating those intentions into robust, operationally actionable tests is far from straightforward. Evaluations often lack nuance, miss key edge cases, or rapidly become outdated as policies and models evolve.

Enter ASSERT (Adaptive Spec-driven Scoring for Evaluation and Regression Testing), an open-source framework from Microsoft that bridges this crucial gap. ASSERT takes natural language behavior specifications and transforms them into executable evaluation pipelines tailored for AI models and agents. It doesn’t stop at pass/fail verdicts but produces rich evidence like test scenarios, detailed failure rationales, instrumented traces, and interpretable scorecards.

In this post, we’ll explore ASSERT’s architectural design, walk through its operational workflow, dive into a practical example—a multi-tool travel-planning agent—and share insights on how teams can harness this framework to elevate their AI evaluation practice.

Architecture Overview

┌─────────────────────────────────────────────┐
│               Behavior Specification        │
│      • Natural language policies & specs   │
│      • Product requirements, checklists    │
└─────────────────────────────────────────────┘
                ↓
┌─────────────────────────────────────────────┐
│           ASSERT Evaluation Framework        │
│  ┌────────────────────────────────────────┐ │
│  │ Concept Systematization & Taxonomy     │ │
│  │ • Explicit behavior definitions        │ │
│  │ • Editable permissible/impermissible   │ │
│  ├────────────────────────────────────────┤ │
│  │ Stratified Test Generation              │ │
│  │ • Scenario synthesis                    │ │
│  │ • Multi-turn, adversarial & benign     │ │
│  ├────────────────────────────────────────┤ │
│  │ Instrumented Inference                  │ │
│  │ • Trace recording (tool calls, steps)  │ │
│  ├────────────────────────────────────────┤ │
│  │ Scoring & Reporting                     │ │
│  │ • Policy-cited verdicts & rationales   │ │
│  └────────────────────────────────────────┘ │
└─────────────────────────────────────────────┘
                ↓
┌─────────────────────────────────────────────┐
│               AI Model / Agent               │
│    • Target system under evaluation         │
│    • Multi-tool agents, workflows            │
└─────────────────────────────────────────────┘
                ↓
┌─────────────────────────────────────────────┐
│                Evaluation Outputs            │
│   • Test cases & datasets                     │
│   • Verdicts with rationale                   │
│   • Scores & metrics                          │
│   • Trace evidence for auditability           │
└─────────────────────────────────────────────┘

ASSERT’s architecture systematically pipelines written behavior specs into an actionable evaluation suite covering the whole lifecycle—from definition, through scenario synthesis and execution, to detailed scoring, complementing AI agent development with auditable insights.

Figure: Overview of ASSERT evaluation framework pipeline. Source: Microsoft Responsible AI Team

Key Technical Observations

• Spec-First Evaluation Design: ASSERT elevates natural language behavior specifications from informal context to a structured, first-class artifact. This systematization prevents superficial or underspecified evals by breaking down broad behaviors into explicit, nuanced taxonomies.

• Stratified Test Generation: Rather than naïve, narrow prompt-based eval generation, ASSERT produces a stratified, dimensional test set based on declared relevant factors like task type, persona, or tool availability. This ensures broad behavioral coverage and avoids “shortcut” evaluations.

• Trace-Instrumented Inference: For agentic systems, ASSERT captures detailed runtime traces including tool calls, intermediate decisions, and routing paths. This instrumentation is critical to diagnosing not only if failure occurred but exactly where and why, transforming evaluation from black box to transparent.

• Policy-Aligned Scoring with Rationale: Scoring outputs include not only pass/fail, but policy citations and verdict rationales tied explicitly to taxonomy entries, promoting interpretability and auditability rarely seen in automated evaluation.

• Validation Against Human Judgements: LLM-based judges exhibit high fidelity (80–90% agreement) with human annotators across diverse behaviors, supporting reliable automation while highlighting the need for expert review in subtle or domain-specific cases.

• Broad Applicability Yet Context-Sensitivity: While ASSERT excels on clear, narrow-behavioral definitions, it cautions against overreliance on aggregate metrics or vague specs. Evaluation remains a human-assisted process enriched, not replaced, by automation.

How It Works

Concept Systematization and Taxonomy

The first stage takes an abstract behavior goal like “avoid harmful financial advice” and systematizes it into a nuanced concept specification. Rather than a single label, it encodes patterns, edge cases, operational definitions, and distinctions grounded in prior work (e.g., Agarwal et al., 2026). This specification is then translated into an editable behavior taxonomy describing permissible vs. impermissible actions. Policy experts can review and refine this taxonomy, ensuring evaluation aligns with organizational rules and values.

Stratified Test-Set Generation

With a structured taxonomy in hand, ASSERT generates a diverse, stratified test set. Developers specify dimensions important to the application—such as tool combinations, request types, or environmental parameters—and ASSERT synthesizes scenarios to cover these systematically. This includes both benign interactions and adversarial probes, often with multi-turn exchanges. The stratification avoids narrow “test factory” outputs, offering richer and more representative evaluation vectors.

Instrumented Inference Execution

When running evaluations, ASSERT interfaces with the target AI model or agent, capturing not only final outputs but full execution traces: tool invocations, intermediate states, routing decisions, and evidentiary context. For agents employing multiple tools or multi-step workflows, this detailed logging is essential for post-hoc reasoning about failures, rather than treating outputs as opaque responses.

Scoring with Contextual Rationale

Finally, ASSERT applies automated judges to compare executions against the behavior taxonomy. Judges output a verdict, annotated with a specific policy citation and a rationale explaining precisely which action or decision triggered the judgment. This level of transparency greatly facilitates debugging, continuous improvement, and compliance audits.

```yaml

Example snippet from taxonomy.yaml (simplified)

harmful_financial_advice:
permissible:
- "Provides generic financial advice without personal risk assessment"
impermissible:
- "Recommends high-risk investments without disclaimers"
- "Encourages behaviors leading to financial harm"
```

Example: a simple YAML-based taxonomy defining behavior boundaries.

A Worked Example: Travel-Planning Agent

Consider a travel-planning AI agent designed to assist users with itineraries. At first glance, its function seems straightforward: find flights, recommend hotels, check weather, and build plans.

However, real-world requirements add nuance: the agent must use tools in correct order, respect user constraints like budgets, and avoid subtle failures such as fabricating flight prices or making discriminatory assumptions based on traveller attributes.

ASSERT tests such an agent using a multi-agent LangGraph planner equipped with five tools:

• search_flights
• search_hotels
• check_weather
• check_travel_advisories
• validate_budget

The evaluation spans six turns, capturing the entire run trace. ASSERT classifies failures into quality issues (e.g., skipped tool use, fabricated details) and safety issues (e.g., stereotyping, prompt injection). Running

assert-ai run --config eval_config.yaml

executes this evaluation, generating artifacts:

• taxonomy.json: the systematized behavioral concept specification
• test_set.jsonl: stratified, multi-turn prompts and scenarios
• inference_set.jsonl: full agent execution traces with tool results
• scores.jsonl: detailed per-trace verdicts and rationales
• metrics.json: aggregate performance roll-ups

These outputs enable teams to pinpoint why a travel agent failed—was it a budget violation or a failure to call a necessary tool? Instead of an opaque pass/fail, developers get actionable insights to refine their agents.

Figure: Example evaluation interface showing verdicts and failure types for a travel-planning agent. Source: ASSERT project

Quick Tips & Tricks

1. Write Clear, Narrow Behavior Specs
   The quality and coverage of generated evaluations depend heavily on well-defined, focused specifications. Clarity upfront reduces ambiguity downstream.

2. Use Taxonomies to Capture Nuance
   Convert broad intentions into detailed taxonomies. This pays dividends by surfacing subtle edge cases and enabling targeted testing.

3. Leverage Stratification Dimensions Thoughtfully
   Select evaluation dimensions (tool use, personas, request types) relevant to your domain to maximize test coverage diversity.

4. Inspect Trace Evidence for Debugging
   Don’t rely solely on aggregate scores; inspect the instrumented execution traces to understand failures contextually.

5. Combine Automated Judges With Human Review
   Automated verdicts are a force multiplier but need domain expert validation, especially for subtle or high-risk behaviors.

6. Iterate Behavior Specs and Tests Together
   Treat evaluation as an evolving artifact. Refine behavior specs based on test results and failure patterns to continuously enhance coverage.

Conclusion

ASSERT represents a significant step toward closing the gap between natural language behavior specifications and actionable AI agent evaluation suites. By systematizing behavioral intent, generating stratified test scenarios, instrumenting agent inference, and producing transparent, rationale-backed verdicts, ASSERT makes AI evaluation more explicit, broader in coverage, and easier to audit.

Its practical focus on application-specific behavioral requirements addresses a critical blind spot in generic AI metrics. While it doesn’t replace the need for human judgment or domain expertise, ASSERT accelerates iteration and strengthens the reliability of AI deployments.

Looking ahead, frameworks like ASSERT will play a central role in trustworthy AI lifecycle management, especially as heterogeneous multi-agent systems become pervasive. Embracing specification-driven evaluation pushes us toward more responsible, interpretable, and robust AI.

References

1. Turn specs into evals for any agent with ASSERT - Microsoft Command Line — Official announcement and detailed framework overview from the Microsoft Responsible AI team
2. ASSERT GitHub Repository — Open-source code and examples for ASSERT evaluation framework
3. ASSERT Project Site — Documentation, tutorials, and resources for using ASSERT
4. Agarwal et al. (2026) — Grounding behavioral specifications to structured taxonomies (referenced research in framework design)
5. Microsoft Responsible AI — Broader initiatives around responsible and ethical AI development at Microsoft

Figure: Microsoft logo — source of ASSERT framework

Figure: Command Line branding associated with the Microsoft developer blog