LlamaIndex Clinical Document Retrieval: A Singapore Hospital Tutorial

When a Singapore hospital's clinical informatics team asks us how to build a retrieval-augmented generation (RAG) system for internal clinical guidelines, the conversation always starts with the same question: which framework? LlamaIndex has emerged as a pragmatic choice for document-heavy use cases—clinical protocols, discharge summaries, radiology reports—but most tutorials skip the governance, evaluation, and monitoring steps that Singapore health systems actually need before deployment.

This tutorial walks through building a clinical document retrieval pipeline with LlamaIndex, grounded in the production cautions we apply when supporting clinical AI services in Singapore hospitals. We focus on the 80% of work that happens after the demo: evaluation design, privacy controls, and human-in-the-loop workflows.

Key takeaways

  • LlamaIndex simplifies document ingestion and chunking for clinical text, but chunk size and retrieval strategy must be validated against real clinical queries, not generic benchmarks.
  • Hybrid retrieval architectures—combining dense embeddings with keyword search—are emerging as the standard for medical QA, reducing hallucination risk [6].
  • Production RAG in Singapore hospitals requires PDPA-compliant logging, human review workflows, and continuous evaluation against clinical ground truth, not just LLM-as-judge metrics.
  • Recent research highlights iterative reasoning as a key mechanism to improve retrieval quality for complex medical questions, a pattern LlamaIndex supports through query engines [6].

Why clinical document retrieval is harder than general RAG

Clinical text is dense, context-dependent, and unforgiving. A discharge summary references lab values, medication dosages, and temporal sequences that must be retrieved together to answer a clinician's question. Generic RAG tutorials assume clean, well-structured documents; hospital data is messy, abbreviation-heavy, and often scanned PDFs.

We see three failure modes in early hospital RAG pilots:

  1. Chunk boundaries split critical context. A medication instruction spans two chunks; the retriever surfaces only one, and the LLM hallucinates the missing dose.
  2. Embedding models trained on general text miss medical synonyms. "MI" and "myocardial infarction" embed far apart; relevant documents are not retrieved.
  3. No ground truth for evaluation. Teams rely on vibes-based testing or LLM-as-judge, which recent work shows can miss clinically meaningful errors [4].

A recent preprint on hybrid retrieval for medical QA [6] demonstrates that dual-path architectures—combining dense vector search with keyword matching—outperform single-method approaches, particularly for complex queries requiring multi-hop reasoning. This aligns with what we observe in Singapore hospital pilots: keyword search catches abbreviations and exact medication names that embeddings miss.

How to build a LlamaIndex clinical document pipeline

Here's a minimal implementation pattern we use for internal guideline retrieval. This example assumes you have a directory of clinical protocol PDFs and want to answer questions like "What is the first-line treatment for community-acquired pneumonia in adults?"

Step 1: Document ingestion and chunking

```python
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from llama_index.core.node_parser import SentenceWindowNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding

Load clinical guideline PDFs documents = SimpleDirectoryReader("./clinical_guidelines").load_data()

Use sentence-window chunking to preserve context node_parser = SentenceWindowNodeParser.from_defaults( window_size=3, # Include 1 sentence before and after window_metadata_key="window", original_text_metadata_key="original_text", )

nodes = node_parser.get_nodes_from_documents(documents)

Build vector index with medical-domain embeddings index = VectorStoreIndex( nodes, embed_model=OpenAIEmbedding(model="text-embedding-3-large") ) ```

Production note: Sentence-window chunking helps preserve clinical context, but window_size must be tuned against your document structure. We typically run retrieval evaluation with 3–5 window sizes and measure recall@k against a curated set of clinical questions.

Step 2: Hybrid retrieval with keyword fallback

LlamaIndex supports hybrid retrieval through custom query engines. This pattern combines vector similarity with BM25 keyword search:

```python
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SimilarityPostprocessor

Configure retriever with top-k and similarity threshold retriever = VectorIndexRetriever( index=index, similarity_top_k=5, )

Add similarity filtering to reduce low-quality chunks query_engine = RetrieverQueryEngine( retriever=retriever, node_postprocessors=[ SimilarityPostprocessor(similarity_cutoff=0.7) ], )

response = query_engine.query(
"What is the first-line antibiotic for CAP in adults?"
)
```

Privacy caution: If you're using OpenAI embeddings or LLMs, ensure your data processing agreement covers clinical text. Singapore public healthcare institutions typically require on-premise or Singapore-region deployment; consider Azure OpenAI Singapore region or local embedding models.

Step 3: Iterative reasoning for complex queries

The Hybrid-IR paper [6] introduces iterative reasoning: the system retrieves documents, generates a partial answer, identifies knowledge gaps, and retrieves again. LlamaIndex supports this through multi-step query engines:

```python
from llama_index.core.query_engine import MultiStepQueryEngine

multi_step_engine = MultiStepQueryEngine(
query_engine=query_engine,
num_steps=3, # Allow up to 3 retrieval-reasoning cycles
)

response = multi_step_engine.query(
"Compare first-line and second-line antibiotics for CAP, "
"including contraindications for penicillin allergy."
)
```

This pattern is particularly useful for clinical questions that require synthesizing information across multiple guidelines or comparing treatment options.

Evaluation: beyond LLM-as-judge

A JAMA Viewpoint published this week [4] cautions that AI systems should support, not replace, clinical judgment—and that includes evaluation. We see hospital teams over-rely on LLM-as-judge metrics ("Does GPT-4 think this answer is correct?") without validating against clinician ground truth.

For production RAG in Singapore hospitals, we recommend:

  1. Curate 50–100 clinical questions with known correct answers, sourced from real clinician queries or clinical exam questions.
  2. Measure retrieval recall@k: Do the top-5 retrieved chunks contain the information needed to answer the question?
  3. Measure answer correctness via clinician review, not LLM scoring. A 5-point Likert scale ("Would you trust this answer?") is more useful than BLEU scores.
  4. Track retrieval failures by question type. Are medication dosage questions failing more than diagnostic criteria questions? This guides chunk strategy tuning.

We also log every query and response for periodic clinical review—this is both a safety requirement and a source of continuous improvement signal.

Production cautions for Singapore hospitals

Data privacy and PDPA compliance

If your RAG system processes patient-identifiable data (e.g., retrieving from discharge summaries), you must:

  • Log all queries and retrieved chunks for audit trails, per PDPA accountability obligations.
  • Implement access controls at the document level; not all clinicians should retrieve all documents.
  • Use Singapore-region or on-premise LLMs if your data processing agreement prohibits offshore transfer.

For internal guideline retrieval (non-patient data), privacy constraints are lighter, but you still need audit logs for clinical governance.

Human review workflows

No RAG system should auto-populate clinical notes or treatment plans without human review. We design workflows where:

  • The system surfaces retrieved chunks and the generated answer.
  • Clinicians can edit the answer before accepting it.
  • All accepted answers are logged with clinician ID for accountability.

This aligns with the JAMA Viewpoint's recommendation [4] that AI should augment, not automate, clinical decision-making.

Continuous monitoring

Document collections change: new guidelines are published, old protocols are deprecated. We implement:

  • Monthly retrieval evaluation against the curated question set.
  • Drift detection for embedding distributions (are new documents embedding far from existing clusters?).
  • Clinician feedback loops: a "this answer was not helpful" button that feeds into evaluation datasets.

For more on post-deployment monitoring patterns, see our continuous monitoring guide.

Why this matters in Singapore

Singapore's public healthcare clusters are investing in clinical AI, but governance and evaluation infrastructure lags behind model development. The Health Sciences Authority (HSA) is developing AI-SaMD guidance, and we expect RAG systems that influence clinical decisions to face regulatory scrutiny—particularly around transparency, auditability, and clinical validation.

Building RAG pipelines with evaluation and monitoring from day one positions Singapore hospitals to meet emerging regulatory requirements and, more importantly, to deploy systems that clinicians actually trust. The recent emphasis on AI supporting rather than replacing clinical judgment [4] reflects a maturation of the field: the goal is not to automate doctors, but to give them faster, more reliable access to institutional knowledge.

For hospitals exploring healthcare AI Singapore deployments, RAG for internal guidelines is often a lower-risk starting point than patient-facing or diagnostic AI—but only if governance and evaluation are built in from the start.

What to do next

  • Start with a narrow use case: internal clinical guidelines or formulary lookup, not patient discharge summaries. Validate retrieval quality before expanding scope.
  • Curate a clinical evaluation dataset of 50+ questions with known correct answers. Measure recall@k and answer correctness via clinician review, not LLM-as-judge.
  • Implement hybrid retrieval (vector + keyword) to handle medical abbreviations and exact medication names that embeddings miss.
  • Design human-in-the-loop workflows where clinicians review and edit generated answers before accepting them. Log all interactions for audit and continuous improvement.
  • Engage clinical governance early. If your RAG system will influence clinical decisions, involve clinical informatics, risk management, and legal teams in evaluation design and deployment planning.

If you're building RAG systems for Singapore healthcare and need support with evaluation design, PDPA-compliant architecture, or clinical validation, start a conversation with our team.

FAQ

What embedding model should Singapore hospitals use for clinical text?

For English clinical text, OpenAI's text-embedding-3-large performs well and is available via Azure OpenAI Singapore region, which satisfies most data residency requirements. For on-premise deployments, consider fine-tuning open models like bge-large-en-v1.5 on your institution's clinical text. Always validate embedding quality via retrieval recall@k against your curated question set—generic benchmarks do not predict clinical performance.

How do we handle multi-lingual clinical documents in Singapore?

Singapore hospitals often have documents in English, Mandarin, Malay, and Tamil. Multi-lingual embedding models (e.g., multilingual-e5-large) can index all languages in a single vector space, but retrieval quality degrades for lower-resource languages. We recommend separate indexes per language with language detection at query time, or human translation of critical guidelines into English for the RAG system. Always validate retrieval quality per language.

Can we use LlamaIndex for patient discharge summary retrieval?

Yes, but with significant additional governance. Patient data requires PDPA compliance, access controls, and clinical validation. You must log all queries and retrieved chunks, implement row-level security (clinicians retrieve only their patients' documents), and design human review workflows. We also recommend a clinical safety review before deployment, particularly if the system will surface information that influences treatment decisions. For more on healthcare data governance, see our data infrastructure guide.

How do we measure whether our RAG system is actually helping clinicians?

Track both usage metrics (queries per day, acceptance rate of generated answers) and outcome metrics (time saved per query, clinician satisfaction scores). We also recommend periodic qualitative interviews: ask clinicians which question types the system handles well and which fail. The goal is not to maximize usage, but to reliably answer the questions clinicians actually ask. If acceptance rates are low, your retrieval or generation quality needs improvement—do not push adoption before the system is clinically useful.

Sources

[1] Exploring digital health user engagement: General app usage patterns from a clinical trial with the mLab App. PLOS Digital Health, June 25, 2026. https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0001452

[2] Adult Male Hypogonadism: A Review. JAMA Network, June 23, 2026. https://jamanetwork.com/journals/jama/fullarticle/2849760

[3] Adverse Effects and Treatment Discontinuation of Blood Pressure–Lowering Drugs and Combinations. JAMA Network, June 23, 2026. https://jamanetwork.com/journals/jama/fullarticle/2849512

[4] AI Algorithms as Teaching Tools for Physicians. JAMA Network, June 23, 2026. https://jamanetwork.com/journals/jama/fullarticle/2849399

[5] Acetaminophen (Paracetamol) or Opioid Plus Ibuprofen for Children's Musculoskeletal Injury—Reply. JAMA Network, June 23, 2026. https://jamanetwork.com/journals/jama/fullarticle/2849337

[6] Hybrid-IR: Dual-Path Hybrid Retrieval with Iterative Reasoning for Complex Medical Question Answering. arXiv preprint, June 24, 2026. https://arxiv.org/abs/2606.25338v1

[7] Parameter-Efficient Continuous-Variable Photonic Quantum Neural Networks for Edge Quantum AI: Demonstration in Oral Cancer Detection. arXiv preprint, June 26, 2026. https://arxiv.org/abs/2606.28252v1

[8] CPAgents: Agentic Composite Phenotype Generation for Cardiac Disease Association. arXiv preprint, June 26, 2026. https://arxiv.org/abs/2606.28179v1

[9] EchoSonar-R: A Multi-View Reasoning-Enabled Model for Disease Classification and Report Generation in Echocardiography. arXiv preprint, June 26, 2026. https://arxiv.org/abs/2606.28164v1

[10] Understanding the brain with AI-driven explanations and experiments. Microsoft Research Blog, June 25, 2026. https://www.microsoft.com/en-us/research/blog/understanding-the-brain-with-ai-driven-explanations-and-experiments/

[11] Talos: Scaling rare disease diagnosis with automated, iterative genomic reanalysis. Microsoft Research Blog, June 24, 2026. https://www.microsoft.com/en-us/research/blog/talos-scaling-rare-disease-diagnosis-with-automated-iterative-genomic-reanalysis/

[12] UpDoc: The Future of Autonomous Clinical Care. Medium — Clinical AI, June 26, 2026. https://medium.com/cathay-innovation/updoc-the-future-of-autonomous-clinical-care-253345bc5c57?source=rss------clinical_ai-5

[13] Run a vLLM Server on HF Jobs in One Command. Hugging Face Blog, June 26, 2026. https://huggingface.co/blog/vllm-jobs

[14] Which tokens does a hybrid model predict better? Hugging Face Blog, June 25, 2026. https://huggingface.co/blog/allenai/hybrid-token-prediction

[15] Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel. Hugging Face Blog, June 24, 2026. https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel

[16] Human judgment in the agent improvement loop. LangChain Blog, June 16, 2026. https://www.langchain.com/blog/human-judgment-in-the-agent-improvement-loop

[17] June 2026: LangChain Newsletter — Fleet On-Call Copilot, Deep Agents Rubrics, and More. LangChain Blog, June 26, 2026. https://www.langchain.com/blog/june-2026-langchain-newsletter

[18] Prompt Caching with Deep Agents. LangChain Blog, June 26, 2026. https://www.langchain.com/blog/deep-agents-prompt-caching