Cancer Care AI Failure In 405-Patient Clinical Test

Researchers say that cancer care AI stumbled after diagnosis in NPC-RubricBench, a new test where 30 chatbots made 48.2% staging mistakes and missed many later care steps.

Key Takeaways

NPC-RubricBench used 405 patients and 17,688 patient-specific criteria to evaluate phase-appropriate decisions
The authors evaluated 30 general-purpose and medical models across 941 patient-phase units
Explicit decision errors hit 48.2% in anatomical staging, and omissions dominated later phases
Broad cancer care AI open-ended prompts hurt decision quality, while detailed checklist prompts improved decision quality but reduced evidence faithfulness and safety adherence

The authors wrote on Monday that they evaluated 30 general-purpose and medical models across 941 patient-phase units and found explicit decision errors hit 48.2% in anatomical staging while omissions dominated later phases. The study argues that static medical benchmarks miss the way model failure changes over time.

Cancer Care AI Breaks After The First Step

Many medical AI tests ask one question at one moment, which is useful for diagnosis or short clinical tasks, but cancer care AI in real care unfolds in stages, with a patient assessed, staged, worked up, treated and followed over time.

The paper built a benchmark around nasopharyngeal carcinoma and tracked those transitions, limiting each assessment to the evidence that would have been available at that phase.

That matters because hindsight can make a model look more capable than a doctor or patient would have seen in real practice.

The model ranking also shifted across phases, with high diagnostic performance not translating into better downstream care, and some medically adapted models even showed higher omission rates than general-purpose systems.

Also Read: Agentic LLM Inference Warning Calls Labs Halve Latency Cut Costs 83%

From One Good Answer To Six Care Phases

This is the core warning in the paper, a model can look excellent on the first encounter and still fail once the job changes. In anatomy staging, the main problem was wrong decisions.

In initial work-up and follow-up, for cancer care AI, the main problem was missing needed decisions.

That distinction matters for hospitals evaluating AI copilots, because a wrong answer and a skipped step create different operational risks, one misclassifies the case, the other may leave a patient without a test, referral or follow-up action.

The study also found a prompt tradeoff, broad open-ended requests hurt decision quality across all 30 models, while more detailed checklist prompts improved decision quality but reduced evidence faithfulness and safety adherence. In practice, the safest prompt may not be the most clinically useful one.

Why NPC-RubricBench Changes Medical AI Testing

NPC-RubricBench evaluates performance across six care phases and 941 patient-phase units drawn from a dataset of 405 patients and 17,688 patient-specific criteria, forcing models to make phase-appropriate decisions with only decision-time evidence.

In this dataset the same models that ranked highly at diagnosis often declined in later phases, with a shift from explicit decision errors in anatomical staging to omission errors in work-up and follow-up.

That phase-specific degradation is the main finding hospitals need to consider.

For health systems like the two cancer centers in this paper, that changes what validation should look like. Procurement teams often ask whether a model answers correctly, but this study asks whether the same model keeps answering correctly at the right moment with only decision-time evidence, and those are different tests.

The next thing to watch is whether model vendors publish phase-based results on benchmarks like NPC-RubricBench, because health care buyers will need that evidence before they trust AI beyond note drafting.