Med-PaLM
By Google Research
S. Medical Licensing Examination (USMLE)-style question sets.
Definition
Med-PaLM is a large language model developed by Google Research, fine-tuned from PaLM (and later PaLM 2, as Med-PaLM 2) specifically to answer medical questions, aiming to match clinician-level performance on benchmarks like the U.S. Medical Licensing Examination (USMLE)-style question sets.
Overview
Med-PaLM was introduced in a December 2022 Google Research paper as one of the first large language models explicitly evaluated against a rigorous, multi-dimensional framework for medical question answering (MultiMedQA), which combined datasets like MedQA (USMLE-style questions), MedMCQA, PubMedQA, and consumer medical queries. The original Med-PaLM, built on the 540B-parameter PaLM using instruction prompt tuning, became the first AI system to exceed the passing score on USMLE-style questions, though physician reviewers still found meaningful gaps versus real clinician answers on quality and safety dimensions. Med-PaLM 2, built on PaLM 2 and detailed in a 2023 paper, substantially closed that gap: it achieved "expert" test-taker performance on USMLE-style questions (scoring around 85%+ accuracy in reported evaluations) and, in blinded physician evaluations, its long-form answers were rated comparable to or preferred over physician-generated answers on several axes, including factuality and helpfulness — while researchers cautioned that these results reflect controlled evaluation settings, not deployment-ready clinical judgment. Google made Med-PaLM 2 available to select healthcare organizations via Google Cloud's MedLM offering for tasks like drafting clinical documentation and answering medical questions, rather than releasing it broadly or for autonomous diagnosis. Med-PaLM is explicitly research-oriented and framed by Google as augmenting, not replacing, clinicians, given the high stakes of medical error. It sits alongside other medical-domain LLM efforts, including OpenAI's exploratory medical benchmarking of GPT-4 and various fine-tuned open medical models, as a marker of how general-purpose LLMs are being adapted and rigorously evaluated for high-stakes professional domains.
Key Features
- Fine-tuned from Google's PaLM and PaLM 2 foundation models
- Evaluated using MultiMedQA, a benchmark suite spanning USMLE-style, research, and consumer medical questions
- First AI system reported to pass USMLE-style question benchmarks (original Med-PaLM)
- Med-PaLM 2 achieved 'expert' test-taker level accuracy on medical licensing-style exams
- Uses instruction prompt tuning to adapt a general LLM to the medical domain
- Evaluated by physician panels on axes like factuality, helpfulness, and potential harm
- Offered to healthcare partners via Google Cloud's MedLM, not released as open weights