Clinical Reasoning Graphs: Why Diagnostic Accuracy Alone Fails Medical LLM Evaluation

When a medical LLM reaches 65% diagnostic accuracy on a clinical case benchmark, hospital AI committees face a question that accuracy scores cannot answer: is the model reasoning like a clinician, or pattern-matching its way to correct answers? A preprint published this week introduces clinical reasoning graphs—structured representations that expose the difference [1]. For Singapore hospitals evaluating medical LLMs for clinical decision support, RAG systems, or triage workflows, this matters: pattern matching breaks under distribution shift, reasoning generalizes.

This post is for hospital CIOs, clinical informatics teams, and AI engineers in Singapore and Asia evaluating medical LLMs for deployment. We explain why accuracy-only benchmarks fail, what clinical reasoning graphs reveal, and how to build evaluation frameworks that distinguish competence from consistency.

Key takeaways

  • Modern medical LLMs reach 60-70% diagnostic accuracy on complex cases, but accuracy alone cannot distinguish stable clinical reasoning from pattern matching [1]
  • Clinical reasoning graphs—structured representations with 5 node types and 7 edge types—extract and evaluate the logical structure of LLM diagnostic traces, revealing inconsistencies invisible to accuracy metrics [1]
  • New domain-specific benchmarks like mamabench (25,949 maternal/neonatal health QA items) and mamaretrieval (chunk-level guideline retrieval) expose gaps in general medical benchmarks [4]
  • Adaptive rank budgeting methods show that medical questions differ substantially in confidence and clinical coverage, requiring question-specific model capacity allocation [5]
  • Singapore hospitals deploying medical LLMs should evaluate reasoning structure, retrieval precision, and cross-domain consistency—not just final answer accuracy

Why diagnostic accuracy fails as an evaluation metric

Most medical LLM benchmarks report a single number: percentage of correct diagnoses or treatment recommendations. A model scores 68% on MedQA, 72% on PubMedQA, and the vendor claims clinical-grade performance. But accuracy conflates two fundamentally different capabilities:

  1. Pattern matching: The model has seen similar cases in training data and retrieves the statistically likely answer
  2. Clinical reasoning: The model constructs a causal chain from symptoms to pathophysiology to differential diagnosis

Pattern matching works until it doesn't. When a Singapore hospital deploys a medical LLM trained predominantly on Western datasets, it encounters local disease prevalence (dengue, tuberculosis, hepatitis B), different comorbidity patterns (diabetes, hypertension), and population-specific risk factors. Pattern matching trained on one distribution fails on another. Clinical reasoning—grounded in pathophysiology and causal mechanisms—generalizes.

The clinical reasoning graphs paper demonstrates this empirically [1]. Models achieve similar accuracy scores but produce structurally different reasoning traces: some construct coherent causal chains, others jump between disconnected observations. Accuracy metrics cannot see this difference. Graph-based evaluation can.

What clinical reasoning graphs reveal about medical LLMs

Clinical reasoning graphs represent diagnostic traces as structured graphs with five node types (symptoms, findings, diagnoses, tests, treatments) and seven edge types (supports, contradicts, requires, rules-out, temporal, causal, conditional) [1]. The method extracts these graphs from free-text LLM outputs using a domain-grounded ontology, then evaluates:

  • Structural coherence: Do reasoning steps connect logically, or does the model jump between disconnected observations?
  • Causal consistency: Does the model maintain consistent causal relationships across similar cases?
  • Evidence grounding: Are diagnostic conclusions supported by explicit symptom-to-pathophysiology chains?

The key finding: models reach similar accuracy (60-70% on complex cases) but differ dramatically in reasoning structure [1]. Some models produce coherent graphs with clear causal chains; others produce fragmented graphs that happen to reach correct conclusions through statistical association rather than clinical logic.

For Singapore hospitals, this distinction matters during deployment. A model with fragmented reasoning will fail unpredictably when it encounters:

  • Rare presentations of common diseases (atypical dengue, silent myocardial infarction)
  • Common presentations of rare diseases (tropical infections, genetic conditions more prevalent in Asian populations)
  • Multi-morbidity cases requiring integration across organ systems

Graph-based evaluation exposes these failure modes before deployment, not after.

Domain-specific benchmarks expose general-purpose gaps

General medical benchmarks (MedQA, PubMedQA, MMLU-Medical) sample broadly across specialties but miss domain-specific reasoning patterns. The mamabench paper illustrates this gap [4]. The authors assembled 25,949 maternal, neonatal, and reproductive health QA items from seven expert-authored sources, plus mamaretrieval, a chunk-level relevance benchmark for guideline retrieval.

Why does this matter? Because nurse-midwives, obstetricians, and neonatologists ask questions that general benchmarks rarely cover:

  • Postpartum hemorrhage risk stratification in resource-limited settings
  • Neonatal jaundice management across different gestational ages
  • Contraceptive counseling for patients with complex medical histories

A medical LLM that scores 70% on MedQA might score 45% on mamabench because it lacks the domain-specific reasoning patterns, guideline knowledge, and risk assessment frameworks that maternal-fetal medicine requires. Singapore hospitals deploying LLMs for obstetrics, pediatrics, or other specialized domains should evaluate on domain-specific benchmarks, not general medical QA.

The mamaretrieval benchmark adds another layer: chunk-level retrieval evaluation [4]. RAG systems for clinical guidelines must retrieve not just relevant documents, but relevant passages within documents. A guideline document might be 50 pages; the clinically relevant passage is three sentences on page 37. Chunk-level benchmarks test whether retrieval systems surface the right passage, not just the right document.

For Singapore hospitals building RAG systems on Ministry of Health guidelines, hospital protocols, or specialty society recommendations, chunk-level retrieval evaluation is essential. We covered RAG evaluation methods in a previous tutorial; the mamabench work reinforces that document-level retrieval metrics (recall@k, MRR) miss the precision required for clinical use.

Adaptive rank budgeting: matching model capacity to question difficulty

Most medical LLM deployments use fixed model capacity: every question gets the same parameter budget, whether it's a straightforward drug interaction query or a complex multi-system diagnostic case. The TriageRA-CCF paper challenges this assumption [5].

The authors introduce adaptive rank budgeting for parameter-efficient medical QA: for each question, the adapter decides whether to activate a small, medium, or large subset of LoRA rank channels based on:

  • Clinical confidence: How certain is the model about the answer?
  • Coverage: How well does the question align with the model's training distribution?
  • Cross-domain difficulty: Does the question require integration across specialties?

The insight: medical questions differ substantially in difficulty, and allocating the same capacity to all questions wastes compute on easy questions and under-allocates to hard ones [5].

For Singapore hospitals deploying medical LLMs under compute constraints (edge devices, on-premise servers, cost-controlled cloud deployments), adaptive rank budgeting offers a practical efficiency gain. But it also exposes an evaluation challenge: if model capacity varies by question, how do you benchmark performance?

The answer: evaluate across difficulty strata. Report accuracy separately for:

  • High-confidence, in-distribution questions (routine drug interactions, common diagnoses)
  • Medium-confidence, partial-coverage questions (uncommon presentations, multi-step reasoning)
  • Low-confidence, out-of-distribution questions (rare diseases, novel drug combinations, complex multi-morbidity)

This stratified evaluation reveals where the model is reliable and where it needs human oversight—critical information for designing clinical workflows that integrate LLM outputs safely.

Building a reasoning-aware evaluation framework for Singapore hospitals

Singapore hospitals evaluating medical LLMs for clinical deployment should move beyond accuracy-only benchmarks to multi-dimensional evaluation frameworks that assess:

1. Reasoning structure (clinical reasoning graphs)

  • Extract structured graphs from diagnostic traces using domain ontologies [1]
  • Evaluate causal coherence, evidence grounding, and consistency across similar cases
  • Flag models that reach correct answers through pattern matching rather than clinical logic

2. Domain-specific performance (specialty benchmarks)

  • Test on domain-specific benchmarks (mamabench for maternal-fetal medicine, similar benchmarks for oncology, cardiology, infectious disease) [4]
  • Evaluate on local disease prevalence and population-specific risk factors
  • Include cases that require integration of Singapore Ministry of Health guidelines

3. Retrieval precision (chunk-level RAG evaluation)

  • For RAG systems, evaluate chunk-level retrieval, not just document-level [4]
  • Test whether the system surfaces the clinically relevant passage within guidelines
  • Measure precision at clinically relevant recall thresholds (e.g., precision@3 for top-3 retrieved chunks)

4. Confidence calibration (stratified by difficulty)

  • Evaluate performance separately for high-, medium-, and low-confidence questions [5]
  • Assess whether model confidence scores correlate with actual accuracy
  • Design clinical workflows that route low-confidence cases to human review

5. Cross-domain consistency

  • Test the same clinical scenario presented in different formats (case vignette, structured EHR data, conversational query)
  • Evaluate whether the model maintains consistent reasoning across formats
  • Flag models that produce contradictory outputs for equivalent inputs

This framework aligns with the NIST AI Risk Management Framework's emphasis on validity, reliability, and safety evaluation [3]. For Singapore hospitals, it also aligns with HSA's expectations for AI-enabled medical devices: evidence of performance on the intended use population, not just general benchmarks.

Why this matters in Singapore and Asia

Singapore's healthcare AI ecosystem is maturing rapidly. The HSA AI-SaMD exemption pathway (which we covered in a previous post) allows public healthcare institutions to deploy certain AI systems without full regulatory approval, but institutions remain accountable for safety and effectiveness. Accuracy-only evaluation is insufficient for this accountability.

Asian populations differ from the predominantly Western datasets used to train most medical LLMs:

  • Disease prevalence: Higher rates of hepatitis B, nasopharyngeal carcinoma, gastric cancer; different HIV and tuberculosis epidemiology
  • Pharmacogenomics: CYP2C19 polymorphisms affecting clopidogrel metabolism, ALDH2 variants affecting alcohol metabolism, HLA-B*15:02 and carbamazepine hypersensitivity
  • Comorbidity patterns: Earlier onset of diabetes and cardiovascular disease, different obesity thresholds for metabolic risk

A medical LLM that pattern-matches on Western training data will fail on these population-specific factors. A model that reasons from pathophysiology can adapt—if its reasoning is coherent and consistent. Graph-based evaluation distinguishes the two.

For Singapore hospitals building clinical AI services, reasoning-aware evaluation is not a research curiosity; it's a deployment requirement.

What to do next

  • Audit your current evaluation framework: If you're evaluating medical LLMs solely on accuracy metrics, add reasoning structure evaluation using graph-based methods or chain-of-thought consistency checks [1]
  • Build domain-specific test sets: Assemble 100-200 cases from your institution's clinical domains (oncology, cardiology, infectious disease) that reflect local disease prevalence and population characteristics
  • Implement chunk-level RAG evaluation: If you're deploying RAG systems on clinical guidelines, test chunk-level retrieval precision, not just document-level recall [4]
  • Stratify performance by confidence: Report model performance separately for high-, medium-, and low-confidence questions; design clinical workflows that route low-confidence cases to human review [5]
  • Engage clinical domain experts: Graph-based reasoning evaluation requires clinical input to define valid causal relationships and evidence chains; involve clinicians in benchmark design and evaluation

If you're evaluating medical LLMs for deployment in Singapore hospitals and need guidance on reasoning-aware evaluation frameworks, start a conversation with our team. We've built evaluation pipelines for institutional partners that combine accuracy, reasoning structure, retrieval precision, and confidence calibration.

FAQ

What are clinical reasoning graphs and why do they matter for medical LLM evaluation?

Clinical reasoning graphs are structured representations of diagnostic traces with five node types (symptoms, findings, diagnoses, tests, treatments) and seven edge types (supports, contradicts, requires, rules-out, temporal, causal, conditional) [1]. They matter because they distinguish models that reason coherently from models that pattern-match to correct answers—a distinction invisible to accuracy metrics but critical for deployment safety.

How do domain-specific benchmarks differ from general medical QA benchmarks?

General benchmarks (MedQA, PubMedQA) sample broadly across specialties but miss domain-specific reasoning patterns. Domain-specific benchmarks like mamabench (25,949 maternal/neonatal health items) test whether models handle the specialized knowledge, risk assessment frameworks, and guideline adherence required in specific clinical domains [4]. A model that scores 70% on MedQA might score 45% on a specialty benchmark.

Should Singapore hospitals build custom evaluation benchmarks or use public benchmarks?

Both. Public benchmarks (MedQA, mamabench, domain-specific sets) provide standardized comparison and cover common clinical scenarios [1][4]. Custom benchmarks (100-200 cases from your institution) test performance on local disease prevalence, population-specific risk factors, and institutional guidelines. Use public benchmarks for vendor comparison, custom benchmarks for deployment readiness.

How does adaptive rank budgeting affect medical LLM evaluation?

Adaptive rank budgeting allocates different model capacity to different questions based on difficulty and confidence [5]. This improves efficiency but complicates evaluation: you must report performance stratified by difficulty (high-, medium-, low-confidence questions) rather than a single accuracy number. Stratified evaluation reveals where the model is reliable and where it needs human oversight—critical for safe clinical workflow design.

Sources

[1] Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency. arXiv preprint, June 29, 2026. https://arxiv.org/abs/2606.29876v1

[2] mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health. arXiv preprint, June 28, 2026. https://arxiv.org/abs/2606.29467v1

[3] TriageRA-CCF: Source-Side Clinical Confidence and Coverage Signals for Adaptive Rank Budgeting in Medical LLMs. arXiv preprint, June 28, 2026. https://arxiv.org/abs/2606.29375v1

[4] NIST AI Risk Management Framework. National Institute of Standards and Technology. https://www.nist.gov/itl/ai-risk-management-framework