Medical Imaging Foundation Models: Interpretability Audits vs Benchmark Scores

Medical Imaging Foundation Models: Interpretability Audits vs Benchmark Scores

Foundation models for medical imaging—trained on millions of scans to learn general-purpose representations—are arriving in hospital procurement pipelines. Vendors pitch impressive benchmark scores: 95% sensitivity on chest X-ray pathology detection, state-of-the-art segmentation on CT scans, multi-modal reasoning that combines images with clinical notes. But recent research from July 2026 reveals a troubling pattern: models that excel on predictive benchmarks often fail the interpretability tests that matter for clinical deployment, regulatory approval, and post-market surveillance in Singapore hospitals.

This post is for hospital AI procurement teams, clinical informatics leads, and medical imaging department heads evaluating foundation model vendors—especially those navigating HSA's AI-SaMD pathways or building governed deployment pipelines.

Key takeaways

• Benchmark performance does not predict interpretability: ECG foundation models with similar predictive accuracy show wildly different levels of internal feature consistency and clinical alignment when tested with sparse autoencoders [4]
• Vision-language models exhibit position bias, not reasoning: Medical VLMs change diagnoses based on where reasoning appears in the prompt, not the clinical content—a failure mode invisible to standard accuracy metrics [5]
• Covariate shift threatens medical world models: Imaging models trained to forecast disease progression fail silently when scanner protocols, patient demographics, or referral patterns shift between training and deployment sites [1]
• Language-guided segmentation reviews highlight governance gaps: Recent systematic reviews of foundation models for medical image segmentation reveal limited reproducibility documentation and sparse post-deployment monitoring frameworks [3]
• Singapore deployment requires interpretability infrastructure: Hospitals need sparse autoencoder pipelines, counterfactual reasoning tests, and shift-detection monitoring—not just validation AUC scores

Why predictive benchmarks miss deployment-critical failures

When we evaluate imaging foundation models with hospital partners, procurement teams typically receive vendor slide decks showing AUC curves, sensitivity-specificity tables, and comparisons to radiologist performance on held-out test sets. These metrics matter—but they measure what the model predicts, not how it arrives at predictions or whether those mechanisms generalize to your hospital's patient population, scanner fleet, and referral patterns.

Two July 2026 papers expose this gap systematically:

ECG-InterpBench introduces a benchmark specifically designed to evaluate whether ECG foundation models learn clinically interpretable features [4]. The researchers trained matched-scale sparse autoencoders (SAEs)—a technique that decomposes model representations into human-interpretable components—on multiple foundation models with similar predictive performance. The results: models with nearly identical accuracy on downstream tasks showed dramatically different levels of feature consistency, clinical alignment, and reproducibility across independent SAE training runs. One model's internal representations could be reliably decomposed into features corresponding to known ECG patterns (ST elevation, QRS morphology, heart rate variability); another model with the same benchmark AUC produced features that were unstable, clinically uninterpretable, and changed substantially when the SAE was retrained.

For Singapore hospitals deploying these models under HSA oversight, this matters: if you cannot reliably interpret why a model flagged an ECG as high-risk, you cannot build effective clinical decision support workflows, you cannot investigate adverse events, and you cannot demonstrate the "explainability" requirements increasingly expected in AI-SaMD submissions. Benchmark AUC does not predict this capability.

Position, Not Provenance demonstrates a similar failure mode in medical vision-language models [5]. These models generate chain-of-thought reasoning before answering clinical questions—a feature vendors market as "transparent" and "explainable." The researchers developed CoT-Mediate, a framework that perturbs a single clinically meaningful attribute in the model's own generated reasoning (e.g., changing "no pleural effusion" to "moderate pleural effusion") and measures whether the final diagnosis changes accordingly.

The finding: medical VLMs exhibit strong position bias but weak reasoning mediation. Models change their answers based on where in the prompt reasoning appears, not the clinical content. When reasoning is placed early in the prompt, models follow it; when placed later, they ignore it—even when the clinical facts are identical. This is not reasoning; it is sycophancy to prompt structure. For hospital workflows where clinicians rely on model-generated explanations to decide whether to order additional imaging or escalate care, this failure mode is dangerous and invisible to standard benchmarks.

What covariate shift does to medical world models

Foundation models for medical imaging increasingly aim to learn not just static pattern recognition ("does this scan show pneumonia?") but world models—latent representations of patient physiology and disease progression that can forecast how a patient's condition will evolve under different interventions [1]. These models promise to support treatment planning, digital twin simulations, and longitudinal risk stratification.

But they fail predictably under covariate shift: when the distribution of imaging protocols, patient demographics, referral patterns, or disease prevalence at the deployment site differs from the training data. The CalTwin preprint introduces Fisher-information regularization as a technique to improve robustness, but the core problem remains: medical imaging data are fragmented across institutions, scanner manufacturers, and clinical workflows. A chest CT foundation model trained on tertiary oncology referrals in the US may learn features that do not generalize to community hospital screening populations in Singapore.

We have seen this in radiology AI deployments: a lung nodule detection model trained on thick-slice CT protocols fails when deployed on a hospital's thin-slice screening program; a fracture detection model trained on emergency department scans underperforms in outpatient orthopedic clinics where referral patterns select for subtle injuries. Foundation models amplify this risk because their scale and complexity make it harder to audit what they learned and where those learned features will break.

For Singapore hospitals, this means:

• Site-specific validation is non-negotiable: Vendor benchmark scores from external datasets do not predict performance on your scanner fleet, your patient population, or your referral workflows
• Shift detection must be continuous: You need monitoring infrastructure that detects when input distributions drift—scanner protocol changes, demographic shifts, new referral sources—and triggers re-validation
• Interpretability infrastructure enables shift diagnosis: When performance degrades, sparse autoencoders and feature attribution tools help you diagnose why—which learned features no longer align with your data

This aligns with the deployment architecture we discussed in our radiology AI PACS integration guide and the longitudinal monitoring frameworks in our clinical deterioration AI fragility post.

Why this matters in Singapore

Singapore's healthcare AI ecosystem is maturing rapidly. HSA's AI-SaMD sandbox and exemption pathways (covered in our HSA sandbox explainer) create regulatory clarity, but they also raise the bar for explainability, reproducibility, and post-market surveillance. Hospital clusters are building governed AI platforms (see our platform engineering guide) that require standardized model cards, performance monitoring, and audit trails.

Foundation models for medical imaging arrive in this context with a mismatch: vendors emphasize benchmark performance and general-purpose capabilities, but Singapore hospitals need interpretability infrastructure and shift-robust deployment pipelines. The research from July 2026 provides a roadmap:

1. Demand interpretability benchmarks alongside predictive benchmarks: Ask vendors for sparse autoencoder analyses, counterfactual reasoning tests, and feature stability metrics—not just AUC curves
2. Build site-specific validation into procurement: Require vendors to validate on a sample of your hospital's imaging data, scanner protocols, and patient demographics before contracting
3. Invest in continuous monitoring infrastructure: Deploy shift-detection pipelines that monitor input distributions, feature attributions, and model explanations—not just prediction accuracy

This is not theoretical. We have worked with institutional partners in Singapore health systems to build these pipelines for radiology AI, and the interpretability gaps are real: models that pass initial validation fail months later when scanner protocols change, referral sources shift, or new radiologists join the department and interpret edge cases differently.

What to do next

• Audit your current imaging AI procurement criteria: Do your RFPs require interpretability benchmarks, or only predictive performance? Add sparse autoencoder analyses, counterfactual reasoning tests, and shift-robustness evaluations to vendor requirements
• Pilot sparse autoencoder pipelines on existing models: If you have deployed imaging AI, run interpretability audits using open-source SAE libraries (e.g., from the ECG-InterpBench framework [4]) to understand what your models learned and where they might fail
• Build shift-detection monitoring into your MLOps platform: Integrate distribution drift detection, feature attribution tracking, and explanation consistency monitoring into your clinical AI platform—treat interpretability as a continuous deployment requirement, not a one-time validation step
• Require site-specific validation in vendor contracts: Negotiate contract terms that require vendors to validate on a sample of your hospital's data before go-live and to provide interpretability audits (not just accuracy metrics) as part of post-market surveillance
• Engage with HSA on interpretability standards: As Singapore's AI-SaMD regulatory framework matures, hospital feedback on interpretability requirements will shape future guidance—share your deployment experiences with HSA and industry working groups

If you are building governed imaging AI pipelines or evaluating foundation model vendors, our clinical AI services include interpretability audits, shift-detection infrastructure, and HSA-aligned deployment frameworks. Start a conversation about your imaging AI roadmap.

FAQ

What is a sparse autoencoder and why does it matter for medical imaging AI?

A sparse autoencoder (SAE) is a neural network trained to decompose a foundation model's internal representations into a small number of interpretable features. For medical imaging, this means taking the high-dimensional "hidden state" the model uses to make predictions and breaking it down into components that correspond to clinically meaningful patterns (e.g., "presence of pulmonary edema," "calcification in coronary arteries"). SAEs matter because they let you audit what the model learned and whether those learned features are stable, reproducible, and clinically aligned—properties invisible to standard accuracy benchmarks [4].

How do I know if my hospital's imaging AI is vulnerable to covariate shift?

If your model was trained on external data (vendor-provided datasets, public benchmarks, or data from other hospitals), assume vulnerability until proven otherwise. Covariate shift occurs when your hospital's imaging protocols, patient demographics, referral patterns, or disease prevalence differ from the training data. Practical tests: (1) Compare your hospital's imaging metadata (scanner models, slice thickness, contrast protocols) to the model's training data documentation; (2) Run the model on a sample of your data and compare performance to vendor-reported benchmarks—large discrepancies suggest shift; (3) Monitor prediction distributions over time—sudden changes in the fraction of high-risk predictions or feature attributions indicate drift [1].

Should I avoid foundation models for medical imaging?

No—but procurement and deployment practices must evolve. Foundation models offer real advantages: they can be fine-tuned with less labeled data, they generalize across imaging modalities, and they enable multi-modal reasoning (combining images with clinical notes, lab results, and patient history). The risk is deploying them with consumer-AI procurement practices: trusting benchmark scores, skipping site-specific validation, and neglecting interpretability infrastructure. Treat foundation models as high-capability, high-risk systems that require interpretability audits, continuous monitoring, and shift-detection pipelines—not plug-and-play solutions [3][4][5].

What interpretability requirements should I include in imaging AI RFPs?

Require vendors to provide: (1) Sparse autoencoder analyses or equivalent feature decomposition showing that internal representations correspond to clinically meaningful patterns; (2) Counterfactual reasoning tests (for vision-language models) demonstrating that explanations causally influence predictions; (3) Shift-robustness evaluations showing performance across different scanner protocols, patient demographics, and referral patterns; (4) Documentation of training data distributions (scanner models, imaging protocols, patient demographics) so you can assess alignment with your hospital's data; (5) Post-deployment monitoring plans that include distribution drift detection and explanation consistency tracking—not just prediction accuracy [4][5].

Sources

[1] CalTwin: Towards Calibrated, Shift-Robust Medical World Models via Fisher-Information Regularisation. arXiv preprint, July 29, 2026. https://arxiv.org/abs/2607.26752v1

[2] Qamar S. Language-Guided Segmentation of Medical Images: A Review of Foundation Models. Bioengineering (Basel, Switzerland), July 1, 2026. https://pubmed.ncbi.nlm.nih.gov/42510469/

[3] ECG-InterpBench: Benchmarking the Interpretability of ECG Foundation Models with Matched-Scale Sparse Autoencoders. arXiv preprint, July 29, 2026. https://arxiv.org/abs/2607.27404v1

[4] Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models. arXiv preprint, July 29, 2026. https://arxiv.org/abs/2607.27304v1