Clinical AI deployment: eight safety cases

By Dr Tolu Ajidahun · Physician and clinical LLM evaluator

What works in a benchmark may fail in a clinic. These cases examine the population, medicines, infrastructure and review process behind a deployment decision.

Each is a composite teaching scenario, not a report of an actual product, patient or measured outcome. The suggested calls apply to the facts given; they are not clinical advice or deployment certification.

Try the interactive Deploy Check or read the explanations below.

01. Unvalidated population

Emergency department, Lagos teaching hospital

A sepsis triage model arrives with strong validation results from three American ICUs. Its heaviest input signal is pulse oximetry. The vendor offers a free pilot.

Suggested call: stop and resolve the gap

Pulse oximetry can overestimate oxygen saturation in people with darker skin. A model that relies on it could inherit measurement bias. Review subgroup performance and validate in the deployment population before clinical use.

02. Context omission

Primary-care clinic chain, southwest Nigeria

A prescribing assistant recommends first-line antibiotics straight from international guidelines. Several of those drugs are chronically out of stock in the region. Leadership wants it live this quarter.

Suggested call: stop and resolve the gap

Guidance that keeps naming unavailable drugs teaches clinicians to ignore the tool or improvise around it. It needs the national essential medicines list and real stock data behind it first.

03. Appropriate technology

Postpartum follow-up program, rural maternity network

A rules-based SMS system sends appointment reminders and danger-sign warnings to new mothers on feature phones. Local obstetric staff wrote and reviewed every message. It queues messages during outages, sends them when the network returns and has a nurse callback pathway.

Suggested call: proceed with safeguards

Local clinical review and a nurse escalation pathway make this a candidate for a monitored rollout. Confirm consent, delivery reliability and a fallback for urgent symptoms; queued SMS cannot deliver warnings while the network is down.

04. Dataset shift

Direct-to-consumer app, marketed across West Africa

A skin-lesion classifier reports dermatologist-level accuracy. Its training data is overwhelmingly lighter skin tones, and the pitch deck calls darker skin "an expansion opportunity."

Suggested call: stop and resolve the gap

Accuracy measured on one skin-tone distribution does not transfer to another. Its reported accuracy does not establish performance in the population the tool is being sold to.

05. Workflow mismatch

Chronic-disease clinics, public health system

A dosing tool hard-blocks every recommendation until renal function labs are entered. In the target clinics, patients pay for labs out of pocket and results take up to two weeks.

Suggested call: stop and resolve the gap

Hard-gating on labs the setting cannot deliver can delay treatment or encourage workarounds, including invented values typed in to unlock the tool. Safety features have to survive contact with the infrastructure.

06. Infrastructure assumption

Outpatient department, district hospital

A cloud transcription service promises to cut clinic documentation time in half. It has no offline mode. The pilot site has daily power cuts and clinicians on metered mobile data.

Suggested call: stop and resolve the gap

With no offline mode, every outage now interrupts documentation instead of degrading it gracefully. Offline-first is the entry requirement here, and this one fails at the door.

07. Anchoring bias

Pediatric telehealth line, high-malaria region

A fever triage bot trained on regional consultation data reads nearly every childhood fever as probable malaria and recommends antimalarials plus home observation.

Suggested call: stop and resolve the gap

Test whether the bot recognises urgent alternative diagnoses, including sepsis and meningitis. A high prevalence of malaria does not justify reassuring answers that miss danger signs. Resolve this gap before deployment.

08. Human in the loop

Discharge desk, urban general hospital

A drafting tool translates discharge instructions into Yoruba and Nigerian Pidgin. A clinician reviews and signs off every output before it reaches a patient. It saves each ward about an hour a day.

Suggested call: proceed with safeguards

Clinician review is a safeguard, not a guarantee. A monitored rollout should check translation accuracy with fluent reviewers, preserve urgent instructions, measure actual time saved and audit for errors that reviewers miss. The stated time saving belongs to this fictional scenario, not a measured deployment.

Evidence and further reading

These sources provide background for the review principles. They do not validate the fictional tools or outcomes in these cases.

Read about my clinical LLM evaluation work and method.