External validation for medical AI: what a good test dataset looks like

"External validation is needed." It's one of the most common sentences in medical AI papers. In our analysis of 1,959 papers that state a data gap, 13% asked for exactly that. A 2019 review of studies of AI for medical image diagnosis found that only 6% had validated their algorithm on external data.

Here's what external validation actually requires, and why a held-out slice of your training data isn't it.

Internal, temporal and external

• Internal validation: a random split of the same data source. It tells you the model learned something, not that it travels.
• Temporal validation: later data from the same sites. It catches change over time, but the patients, scanners and habits are the same.
• External validation: data from different sites, ideally different health systems or countries, that played no part in building the model. This is the test that best predicts how the model will do in the real world.

What makes a test set independent

• It comes from different sites than the training data.
• There's no patient overlap. The same patient can be seen at two hospitals, and the same images can turn up in more than one public dataset.
• The model is locked before you look. No tuning thresholds on the test set.
• Where possible, someone other than the development team holds it and runs the test.

Representative of the intended use

The test set should look like the patients and settings where the model will be used: the same intended population, the scanners and protocols in use there, the real-world prevalence of the condition (or both enriched and real-world estimates), and the full range of severity and confounders, such as implanted devices, prior surgery and poor image quality.

Subgroups

Report performance by age, sex, ethnicity where recorded, site and scanner maker. A model can look good overall and still fail one group; chest X-ray models trained on large public datasets have been shown to underdiagnose disease more often in under-served groups. Plan subgroup sizes before you collect data, so each subgroup has enough cases to say something.

The reference standard

Decide how the truth is established: the original radiology report, the consensus of several expert readers, pathology results or follow-up outcomes. Reports are cheaper but noisier; expert consensus and outcomes cost more and are easier to defend. Record the readers' experience and how disagreements were settled.

How many cases

Enough to estimate sensitivity and specificity with useful precision, and that depends on prevalence. A rare finding needs many cases in total, or an enriched set, to get enough positives. Work it out with a statistician before you acquire data, not after.

Reporting

Reporting guidelines help reviewers trust the result: TRIPOD+AI for prediction models and CLAIM for AI in medical imaging. For a regulatory submission, check the regulator's guidance for your device type and market.

Where external data comes from

• Collaborating hospitals: strong evidence, but slow to set up.
• Public datasets: only if they weren't used in training and the licence allows your use. See where public imaging datasets fall short.
• Licensed datasets from sites and populations your model hasn't seen.

If you need an external validation set, tell us the modality, population and size you need.