Free medical imaging datasets: where they fall short for commercial AI

Public imaging datasets changed medical AI. Three chest X-ray collections alone gave thousands of research groups something to train on:

• ChestX-ray14 (NIH Clinical Center, 2017): 112,120 frontal chest X-rays from 30,805 patients.
• CheXpert (Stanford, 2019): 224,316 chest X-rays from 65,240 patients.
• MIMIC-CXR (Beth Israel Deaconess Medical Center, 2019): 377,110 images from 227,835 studies, with the free-text reports.

They're excellent for learning, prototyping and benchmarking. But teams building a product find their limits, usually at the worst moment: when the model meets a hospital it has never seen, or when a regulator asks how it was tested.

1. The licence may not cover your use

Public doesn't mean unrestricted. Some public datasets are licensed for non-commercial research only. Others need credentialed access, training and a signed data use agreement, and forbid passing the data on. Terms can also differ between versions of the same dataset.

Before you build a product on a public dataset, read the licence for the exact version you downloaded, and have counsel confirm it covers training and selling a commercial model.

2. A few hospitals stand in for the world

Each of these datasets comes from one academic hospital, or a handful. A model trained on them learns those hospitals' patients, scanners, protocols and habits along with the disease.

Zech and colleagues showed how much this matters. They trained a pneumonia detector on chest X-rays from several hospital systems and tested it on others. Performance dropped at hospitals it hadn't seen, and the model could tell which hospital an image came from: a sign it was learning the site as well as the disease.

3. The labels are often inferred from reports

Large public datasets usually get their labels by running natural-language processing over radiology reports, not from a radiologist reviewing each image. That makes huge datasets possible, but the labels are noisy. Oakden-Rayner's review of public chest X-ray datasets found many labels that didn't match the images. Noisy labels put a ceiling on what a model can learn and make test scores less trustworthy.

4. The population may not be your market

The patients in these datasets reflect who attends those hospitals. Seyyed-Kalantari and colleagues found that chest X-ray models trained on large public datasets underdiagnosed disease more often in under-served groups, including female patients, Black patients and patients on Medicaid. If you plan to sell in Asia, Europe or community hospitals, a US academic dataset won't tell you how your model performs there.

5. Everyone has the same data

If your model is trained on the same public data as every competitor's, the data gives you no edge. And because so many models have been tuned against the same benchmarks, scores on them can overstate how a model will perform in practice.

6. Regulators want independent test data

For a regulatory submission, performance has to be shown on data that's independent of the training data and representative of the patients the device is intended for. Reusing a public dataset that also fed your training, or one from a population unlike your intended users, makes that case harder.

What to use alongside public data

Use public datasets for what they're good at: learning, pretraining and early prototypes. Then add licensed data where it counts:

• Training data from your target population, scanners and care settings.
• External validation data from sites your model has never seen, held back from training.
• Subgroup coverage by age, sex, ethnicity and equipment, so you can show the model works for everyone it's meant for.
• Reports or expert labels you can trust as ground truth.

When you evaluate a licensed dataset, ask the same questions you'd ask of a public one: what exactly the licence allows, how many sites and scanners it covers, how the labels were made, who the patients are, and how it was de-identified.

BCONZ has opened early access to de-identified medical imaging data with reports, licensed for AI research and development. See medical imaging datasets or tell us the modality, population and volume you need.