What Microsoft’s AI Results Do—and Don’t—Tell Us About Doctors

🔍 Read the full analysis: What Microsoft’s AI Results Do—and Don’t—Tell Us About Doctors on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI achieved higher diagnostic accuracy than generalist physicians in a study of historical medical cases published in the peer-reviewed journal Science on Oct. 22. The comparison found an 83.2% score for the system on 304 complex cases, versus 36.1% for physicians using textbooks and the internet, but it did not test the AI on real patients or against specialist teams.

The paper, titled Sequential diagnosis with language models, evaluates Microsoft’s approach using its Sequential Diagnosis Benchmark. In this simulated process, a system starts with a short case summary and must request additional information—such as questions or tests—from a model holding the full case record. The benchmark scores the final diagnosis and estimates the cost of obtaining information.

Microsoft reported 83.2% accuracy for MAI-DxO on 304 complex case records drawn from the New England Journal of Medicine. Physicians scored 36.1% when allowed textbooks and internet access, and 22.3% without those references. On a separate set of 325 emergency-department cases, Microsoft reported an 80.3% score for the AI system, compared with 46.8% for physicians. These figures are results reported by the paper’s Microsoft-affiliated authors.

The system uses a language model with an orchestration layer that has virtual agents propose diagnoses, request information and challenge one another over multiple rounds. Microsoft’s chart also reports that GPT-5 alone scored 78.9% on the complex cases. That result suggests most of the measured diagnostic performance came from the underlying model, while the MAI-DxO system added several percentage points and reduced estimated testing costs, according to the source account.

What the Accuracy Gap Can Show

The findings matter because they test more than whether an AI can select a diagnosis from a completed case history. MAI-DxO had to choose what information to request, a decision closer to one part of clinical reasoning. The benchmark also attempted to account for the cost of tests, giving researchers a way to compare diagnostic performance with information-gathering choices.

But the reported gap is not evidence that the system is ready to replace doctors or improve patient outcomes. The study tested historical, text-only cases in a simulator, rather than care involving examinations, imaging, incomplete records and real-time patient responses. It also compared a system made up of multiple AI agents with individual generalists working alone. Hospitals, insurers and patients would need evidence about safety, costs and performance in everyday clinical settings before drawing practical conclusions.

The scores also do not tell readers how often the system might wrongly flag a healthy person or recommend unnecessary tests. The source account says the complex NEJM set contained no healthy patients or harmless explanations, limiting the ability to measure false alarms. Accuracy on a collection of difficult diagnostic cases is only one measure of clinical usefulness.

How the Benchmark Was Built

Medical diagnosis benchmarks often provide all case details and ask for a final answer. Microsoft’s sequential benchmark instead requires participants to request each additional detail from a “gatekeeper” model. Each test carries an estimated price, and the system must decide what to ask before settling on a diagnosis. This design focuses attention on the sequence of questions and tests, as well as the final answer.

The source account describes methodological details primarily from the paper’s earlier 2025 preprint, and says some details may differ in the final Science version. In that version, a model judged answers against published diagnoses; physicians re-graded a sample and their agreement was good but not perfect. When a requested test result was absent from the case record, the gatekeeper generated a synthetic result. Test prices were estimates based on billing codes and a 2023 U.S. hospital price list, with a fixed $300 physician-visit cost. The authors characterized those costs as first-order approximations.

The physician comparison also needs qualification. The preprint described 21 U.S. and U.K. generalists—17 primary-care doctors and four hospital generalists—with a median of 12 years of experience. They worked individually, not as teams, and were not specialists. The doctors were tested on a subset of cases while the AI was tested across the full 304-case set, according to the source account. The Science paper reports the open-book result alongside the no-reference score.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft CEO Mustafa Suleyman, speaking to TechCrunch

Limits of the Doctor Comparison

The paper is peer-reviewed, but the results have not been independently reproduced, according to the source account. Peer review evaluates a study’s methods and reporting; it does not establish that another research group will obtain the same results or that the system is safe and effective in practice. The paper’s 15 authors are listed as Microsoft-affiliated, including Suleyman.

It is also unclear how MAI-DxO would perform against specialists or multidisciplinary clinical teams. The comparison involved individual generalists on difficult teaching cases that may be referred to specialists in routine care. The benchmark’s synthetic missing test results, model-based scoring and estimated costs introduce further questions about how closely its results represent real decisions.

The scores cannot establish whether using the system would improve patient health, reduce overall costs, or lead to more unnecessary tests. Nor do the reported accuracy figures provide a complete account of false positives and false negatives across the broader mix of patients seen in clinics and emergency departments.

A Study With Patients Planned

The next announced step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients are expected to interact with the system alongside their usual care. That design could provide evidence on how the tool functions in a clinical workflow, although the available information does not specify the study’s start date, enrollment target, endpoints or publication schedule.

Until those results are available, the published numbers are best read as performance on defined simulated case sets—not as a measure of how accurately MAI-DxO would diagnose patients in hospitals. Further evaluation would need to clarify performance across a wider range of cases, how clinicians use or override its suggestions, and whether its recommendations change patient outcomes or costs.

Key Questions

What did Microsoft report about MAI-DxO?

Microsoft’s paper reported an 83.2% diagnostic accuracy score on 304 complex historical cases. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without those resources, according to the study.

Was the AI tested on real patients?

No. The reported benchmark used historical, text-based cases in a simulator. The source account says a prospective study involving consenting patients alongside normal care is planned at Beth Israel Deaconess Medical Center.

Did MAI-DxO outperform medical specialists?

The reported comparison was with generalist physicians working alone, not specialist teams. The results do not establish how MAI-DxO would compare with specialists or groups of clinicians.

Does peer review mean the results are independently confirmed?

No. Publication in Science means the paper underwent peer review, but it does not mean independent researchers have reproduced its results. The study authors are listed as Microsoft-affiliated, according to the source account.

Source: ThorstenMeyerAI.com