In a Microsoft Research podcast, researcher Jennifer Neville says her team’s own studies find that language models perform worse when tasks unfold over multiple turns or long editing workflows than on single-turn benchmarks. The findings come from papers by the vendor’s own team; the interview gives no figures, and the review status of the papers is not stated.
How this is computed
- Key claims supported: 3 of 6
- Type of source: Company on its own work
- Independent confirmation: None
Rates the source, not the topic and not our analysis.
Why it matters
Our reading: If the cited papers hold up, they illustrate how single-turn benchmark scores might overstate how well models perform in everyday, back-and-forth use. The source is Microsoft discussing its own research, with no independent check, so this is a lead to verify rather than an established result.
What the source says
- Conditionally supported Neville says that when public single-turn benchmark tasks are re-specified over multiple turns with simulated users, model performance “actually degrades significantly over multiple turns”; the underlying paper is LLMs Get Lost In Multi-Turn Conversation, run by her team.
- Conditionally supported Neville says errors in long agentic workflows, such as repeated document edits, can accumulate and produce surprising failures, citing the April 2026 paper LLMs Corrupt Your Documents When You Delegate.
- Conditionally supported Neville advises users whose chat has gone off track to start a fresh chat with one fully specified prompt; she gives no measured effect.
- Unverifiable Neville says standard ML benchmarks are fairly simple compared with real use, which motivates her team’s focus on multiturn, collaborative and long-horizon tasks.
- Vendor claim only Neville says Microsoft Research analyzes consumer product logs at scale for failure patterns in a privacy-preserving way, without researchers viewing the data.
- Unverifiable Neville expects AI progress to come much faster than she would have predicted as a student in 2000; she gives no timeline.
The key caveat
All findings were produced and presented by Microsoft’s own team, the multiturn study relies on simulated users rather than real ones, and the interview names no models, tasks or size of the performance drop.
Open questions
- How large is the measured performance drop, and on which models and benchmarks?
- Have the two cited papers been peer-reviewed, and do their results hold with real rather than simulated users?
- How much does restarting with a single fully specified prompt actually recover?
Source: Microsoft Research Blog, 6 October 2026: What AI gets wrong and what failure teaches us
Produced by Zeitgeist’s AI agents (research, editing, standards check) with Claude. No human review.