AI-generated editorial illustration; not a documentary photograph.
An experiment published on the Microsoft Developer Blog found that an AI model's stated knowledge cutoff did not reliably predict which versions of developer products it could work with. Tests of GPT-5.6 Luna produced failures on features released well before its cutoff and occasional successes on releases that came afterward.
No Clear Version Boundary
The experiment used the GitHub Copilot SDK and evaluated GPT-5.6 Luna on tasks derived from release notes for Dev Proxy and SharePoint Framework (SPFx). External documentation and web search were removed to limit access to answers. Other models extracted and classified product changes, generated prompts and rubrics, or judged the results.
Across 336 Dev Proxy tasks covering 53 versions, Luna passed 61, or 18%. It also passed 61 of 413 SPFx tasks spanning 40 versions, or 15%. Successes and failures were distributed across each product's history instead of forming a point where performance consistently declined.
Cutoff Sets a Ceiling
OpenAI lists February 16, 2026 as GPT-5.6 Luna's knowledge cutoff. Dev Proxy 2.1.0, released January 28, was the latest stable version available by then, but the model estimated that its Dev Proxy knowledge ended around version 0.29.0, released June 25, 2025. Its SPFx estimate was version 1.21.1.
The results indicate that a cutoff describes the latest material a model could have encountered, not whether a particular product or historical detail was represented and retrievable. In one SPFx task, the model gave plausible architectural guidance but missed that SystemJS support had been removed from the local workbench. In a Dev Proxy task, it supplied the current executable name instead of the name used by version 0.5.0.
Later Tasks Can Still Pass
Luna also passed one of two tested tasks from each of Dev Proxy 2.3.4, 3.0.0 and 3.1.0, released after its cutoff on April 23, May 28 and July 1, 2026. The blog cautions that these results do not show direct knowledge of those releases: the model may have transferred familiar programming patterns, inferred behavior or guessed correctly.
Evaluate the Actual Workload
For developers building agent extensions, the experiment recommends testing representative tasks without added documentation, then measuring how targeted documentation changes performance. This workload-based approach can expose gaps from older releases while avoiding the assumption that every post-cutoff feature must be supplied in context. The findings come from this specific evaluation setup and do not establish performance across other models, products or workloads.
Source: Knowledge cutoff is a poor proxy for model capability
SIGNAL TITANS
Leave a Reply