Microsoft developer blogs detailed the emerging field of Agent Experience (AX), a concept coined by Netlify co-founder Mathias Biilmann in January 2025 to describe how AI agents discover, select, and use software tools and developer platforms. As software engineers increasingly rely on AI coding agents to integrate APIs, build SDKs, and run command-line tools, evaluating how effectively agents navigate these platforms has become essential for tool creators.
Measuring Propensity, Efficacy, and Real Task Costs
AX evaluation relies on two primary pillars: propensity and efficacy. Propensity measures whether an AI agent independently finds and selects a product when given a task that does not explicitly name it, while efficacy assesses whether the agent uses the tool accurately once selected. Evaluating AX requires scoring both output quality and overall task costs rather than relying solely on per-token pricing models.
Microsoft noted that token pricing drops do not always yield lower execution costs. In tests upgrading SharePoint Framework (SPFx) projects within GitHub Copilot Chat, Claude Sonnet 5 had 33% lower per-token pricing than Sonnet 4.6, yet resulted in a 3.7-times higher cost per run across three scenarios and 15 runs per model.
Prioritizing Documentation Over Custom Extensions
AX evaluations demonstrate that refining existing documentation often yields better results than shipping specialized agent extensions. In Microsoft's SPFx upgrade tests using Claude Sonnet 4.6 on Windows, the baseline agent passed only 30 of 80 configuration checks. Explicitly directing the agent to use the CLI for Microsoft 365 raised the passing rate to 75 out of 80, but required developers to know about or install that specific skill.
By analyzing the agent's step-by-step action logs, Microsoft observed that the model regularly retrieved SPFx release notes during execution. Updating those release notes directly enabled the agent to utilize the CLI across all runs without requiring developers to install additional extensions or custom skills.
Practical Workflows for AX Optimization
To practice AX, technical teams establish a bare baseline by testing real-world prompts under a fixed agent profile, which specifies the operating system, harness app, AI model, and loaded extensions. Teams then measure whether documentation or interface adjustments create performance improvements, known as lift, or unintended regressions, referred to as drag.
Frameworks and formats such as llms.txt or Model Context Protocol (MCP) servers serve as potential agent interfaces, but Microsoft advises treating them as hypotheses to be tested. Improving core APIs and documentation remains the primary starting point, with custom agent extensions reserved as a secondary option.
AI-generated editorial illustration; not a documentary photograph.
Source: What is Agent Experience (AX)?
SIGNAL TITANS
Leave a Reply