🔍 Read the full analysis: What The Researchers Counted In Microsoft’s AI-versus-Doctors Study on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Microsoft’s MAI-DxO system scored 83.2% on 304 difficult, historical case records, compared with 36.1% for generalist physicians using reference materials. The study tested diagnosis in a text-based simulator, not care for real patients, and its authors have not had the findings independently reproduced. A prospective study alongside usual care is planned at Beth Israel Deaconess Medical Center.
Microsoft’s MAI-DxO diagnostic AI scored 83.2% on a benchmark of 304 difficult medical cases, compared with 36.1% for generalist physicians using textbooks and the internet, according to a peer-reviewed paper published in Science on October 22. The result concerns historical, text-based case records in a simulator—not diagnosis or treatment of patients—and does not establish how the system would perform in routine clinical practice.
The study, titled “Sequential diagnosis with language models,” tested whether an AI could work through a case by requesting further information, choosing tests and then making a diagnosis. It used records from the New England Journal of Medicine, with cases selected for their diagnostic difficulty. Microsoft reported 83.2% accuracy on those 304 cases. Physicians scored 36.1% when they could consult textbooks and the internet, and 22.3% without those reference materials.
On a separate set of 325 emergency department cases, Microsoft reported a score of 80.3% for MAI-DxO and 46.8% for physicians. The system also estimated the costs of the tests it requested. The benchmark began with a short case summary; additional information had to be requested from a model holding the full case record. The study therefore evaluated both the final diagnosis and the route taken to reach it.
According to the source account’s description of the earlier preprint, the system’s orchestration method had a smaller advantage over the underlying model than the headline score might suggest. GPT-5 alone scored 78.9% on the difficult cases, compared with 83.2% for MAI-DxO. The source reports an estimated test cost of $8,409 per case for GPT-5 alone and says the orchestration layer reduced estimated costs. Those cost figures are modelled estimates, not observed spending in hospitals.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
Why the Test Design Matters
The results matter because they show how a diagnostic AI can be tested on more than a multiple-choice question. In the benchmark, the system had to decide what information to request and which test to order, choices that affect both diagnostic performance and costs. That makes the work relevant to health systems considering whether AI might support clinicians in complex cases.
But the comparison does not show that MAI-DxO outperforms the clinical teams who would normally manage those patients. The physician group consisted of primary care doctors and hospital generalists working alone; the study did not compare the AI with specialists or teams. The benchmark cases were also unusually difficult teaching cases, and the NEJM set contained no healthy patients or harmless explanations. The study therefore could not measure how often the system might trigger unnecessary concern or testing in people without serious disease.
For patients, hospitals and insurers, the distinction is practical: benchmark accuracy is not evidence of improved outcomes. The reported scores do not establish whether an AI would help clinicians make safer decisions, reduce delays or lower costs when used with real patients. Those questions require testing in clinical settings, with attention to both benefits and potential harms.
Popular right now
De'Longhi Classic Signature Espresso Machine with Adjustable Milk Frother
Ninja CREAMi Scoop & Swirl 13-in-1 Ice Cream and Soft Serve Maker (NC701)
Vitamix Touch & Go Advance 32-Ounce Bar Blender, White
As an affiliate, we earn on qualifying purchases.
From Preprint to Science Paper
The Science paper appeared on October 22, 2026, after being received in April and accepted in August, according to the source material. All 15 listed authors had Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman. The article builds on a version posted as a preprint in June 2025, which was not peer-reviewed and reported results using an earlier model, OpenAI’s o3.
The source account says the preprint described a benchmark called the Sequential Diagnosis Benchmark. A “gatekeeper” model supplied case details when requested; when a requested result was absent from the record, it generated a synthetic result. A model judged diagnoses against each case’s published final diagnosis, while physicians re-graded a sample with good, but imperfect, agreement. Test prices were estimates based on billing codes and a 2023 U.S. hospital price list, plus a fixed physician-visit charge. The account cautions that these details come from the preprint and may differ from the final paper, whose full text is behind a paywall.
Microsoft’s public framing emphasized the performance ratios: the company said MAI-DxO was 1.7 times as accurate as physicians on emergency cases, more than twice as accurate as doctors using references on complex cases, and nearly four times as accurate as doctors without them. Those are Microsoft’s reported results. Publication in a peer-reviewed journal means the paper underwent review; it does not mean outside researchers have reproduced the findings.
“The system was reported as 1.7 times as accurate as physicians on emergency department cases and more than twice as accurate as physicians using textbooks and the internet on complex cases.”
— Microsoft, in its blog post “Building Medical Superintelligence”
Questions Beyond the Simulator
The reported results do not resolve how MAI-DxO performs with real patients, physical examinations or imaging; the benchmark was text-only and used historical records. The study also does not establish performance on routine cases, where common illnesses and healthy patients are part of the clinical mix. Because the difficult NEJM cases lacked harmless explanations, the reported accuracy cannot show how often the system might produce a false alarm.
Other limits include the comparison with individual generalists rather than specialists or clinical teams, and the fact that the work was conducted by authors affiliated with Microsoft. The source account says the final Science text is behind a paywall, so some methodological details available in the earlier preprint may not match the published version. Independent replication and clinical outcome data are not provided in the supplied material. It is also not yet clear how the system would affect clinician workload, patient safety, or costs in actual care.
A Planned Study With Patients
The next described step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source material, consenting patients are expected to interact with the system alongside their normal care. That would move the evaluation beyond historical case records, although the supplied account does not give a start date, enrollment target or study protocol.
Readers should look for details on how the clinical study will assess accuracy, safety, unnecessary tests and effects on care—not only whether the AI reaches a final diagnosis. Until those results are available, the Science paper is evidence about performance on a defined benchmark, not proof of clinical benefit.
Key Questions
What did Microsoft’s study report?
Microsoft reported that MAI-DxO scored 83.2% on 304 difficult NEJM case records. Generalist physicians scored 36.1% with textbooks and internet access, and 22.3% without those resources.
Did the study test the AI on patients?
No. The reported benchmark used historical, text-based case records in a simulator. It did not test diagnosis or treatment during real patient care.
Were specialists included in the comparison?
According to the source account, the physician group consisted of primary care doctors and hospital generalists working alone. It did not include specialist teams, which may handle difficult cases in practice.
Has the result been independently replicated?
The supplied material does not report an independent replication. The paper appeared in the peer-reviewed journal Science, but peer review alone does not establish that outside researchers have reproduced its results.
What is expected next?
A prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would interact with the system alongside their usual care. Details such as timing and the study protocol were not provided in the source material.
Source: ThorstenMeyerAI.com
