Large language models (LLMs) now answer medical licensing-exam questions with high accuracy and generate long-form medical responses that receive favourable clinical ratings. Yet almost all of this evidence is in English. Whether a model can interpret medical concepts and communicate reliably in other languages, especially those under-represented in training data and existing evaluations, remains largely untested. Multilingual assessment is therefore needed to determine where medical competence transfers across languages and where it fails.

Several multilingual medical benchmarks already cover question answering and clinical language understanding, but most are built by machine-translating English test sets. Translation can alter medical terminology, syntax and contextual meaning, and these changes affect model scores and rankings. A lower score in a target language may therefore reflect weaker model capability, translation error, or both. Without expert review, these sources of error cannot be separated.

HealMed (Human-verified Evaluation Across Languages for Medical AI) addresses this gap. It is an expert-reviewed medical benchmark spanning nine languages, nine source datasets and three task families: multiple-choice question answering (MCQA), natural language inference (NLI) and open-ended question answering (QA). 23 physicians and medical experts across nine countries and regions contributed to its construction and review over two years, and every target-language instance passed a structured two-stage review by two medical experts fluent in English and the target language. Both the original machine-translated and the final expert-reviewed versions are retained, enabling paired evaluation of how translation quality affects cross-language results.

HealMed contains 1,000 aligned examples in each of nine languages. Six are higher-resource languages (English, German, Spanish, Portuguese, Japanese and Chinese) and three are lower-resource languages (Thai, Swahili and Zulu). The examples are drawn from nine established English medical benchmarks and cover three complementary task formats, from fixed-choice prediction and relation classification to free-form generation.

Benchmark components

• Multiple-choice question answering (MCQA, n = 300): HeadQA (75), MedQA (75), MedExpQA (75) and MMLU-Pro (75). Models must select the correct option from a fixed set.

• Natural language inference (NLI, n = 300): BioNLI (150) and MedNLI (150). Models classify the logical relation between paired biomedical or clinical statements as entailment, contradiction or, where applicable, neutral.

• Open-ended question answering (QA, n = 400): ExpertQA-Bio (40), ExpertQA-Med (160) and LiveQA (200). Models generate free-form answers that are scored against expert-verified reference answers.

Construction and expert review

We selected 1,000 English examples and machine-translated them into the eight target languages. Every translated instance then went through a two-stage review by two medical experts fluent in English and the target language. Reviewer 1 compared the translation with the English source, rated its accuracy, fluency and completeness on five-point scales and corrected it where necessary. Reviewer 2 independently rated the original translation, verified the first revision and corrected any remaining errors. Across all languages, the mean expert ratings of the machine translations were 4.72 (accuracy), 4.71 (fluency) and 4.88 (completeness), and 4.8% of words were revised on average; German (12.1%) and Thai (9.1%) required the most revision. Both the quality scores and the final expert-revised translation are retained for every instance.

Data format and examples

Each HealMed instance contains six fields: id, language, task, source_dataset, input and target. The id links the aligned versions of the same source example across the nine languages, so every model can be evaluated on exactly the same content in each language. Representative English records from the three task families are shown below (the NLI premise and the long-form reference answer are shortened for presentation).

MCQA - HeadQA (id: mcqa-headqa-0000)

Question: Motor end-plate is the junction between the motor neuron and the:

Options: A. Smooth muscle | B. Skeletal muscle | C. Cardiac muscle | D. Muscle spindle (Muscle spindle organ) | E. Tendon

Target: B

NLI - BioNLI (id: nli-bionli-0060)

Premise: We examined the ability of sucralfate to prevent secretagogue-induced duodenal ulcer in the rat. [...]

Hypothesis: We conclude that tubastatin A prevents the formation of secretagogue-induced duodenal ulcer in the rat.

Options: A. Entailment | B. Contradiction

Target: B

Open-ended QA - ExpertQA-Med (id: qa-expertqa-med-0058)

Question: How can you manage your patient expectations?

Reference answer: To manage patient expectations, follow these steps: develop rapport and build a trusting therapeutic relationship with your patients. [...]