Expo, working alongside researchers from the Georgian AI Lab, has introduced an evaluation harness to measure how AI coding agents locate and apply specialized developer skills during mobile application development. The system addresses the limitation of testing direct skill invocation in isolation, evaluating instead whether agents autonomously discover appropriate guidance during standard development workflows.
Measuring Trajectories Across Three Layers
The evaluation harness executes coding agents, such as Claude Code and Codex, against realistic product requirement documents representing mobile app development scenarios. Rather than judging only the final application output, the harness records full agent trajectories and executes parallel assessment pipelines. Static checks measure whether expected skills were loaded and whether their specific recommendations appear in the generated codebase. Simultaneously, an agentic evaluator builds and runs the mobile app to verify functional behavior, dividing evaluation into skill triggering, skill uptake, and runtime execution.
Improving Discovery with Entrypoint Skills
Initial trace data revealed that coding agents frequently completed app builds using existing internal knowledge without loading available Expo skills. To streamline discovery across a catalog of more than twenty specialized skills, the team introduced an entrypoint router skill named expo-overview for Expo and EAS tasks. In test evaluations using Claude Code with sonnet-5-high across ten mobile app scenarios, adding the entrypoint skill increased the proportion of agent sessions loading at least one downstream skill from 9% to 55%. However, relevant skill recall reached 18%, as agents consulted skills primarily for initial architecture and setup while relying on internal knowledge for later implementation details.
Catalog Constraints and Context Budgets
The project also highlighted discovery challenges caused by context window allocations in CLI interfaces like Codex. In Codex, available skill descriptions share a catalog budget limited to 2% of the context window, or 5,440 tokens. When a catalog exceeds this budget, descriptions are truncated, potentially stripping key context needed for an agent to determine relevance. Consequently, skill authors are advised to place critical triggering cues within titles and opening lines to ensure discovery in crowded environments.
AI-generated editorial illustration; not a documentary photograph.
Source: Your skills need an evaluation mechanism
SIGNAL TITANS
Leave a Reply