What works in a benchmark may fail in a clinic. These cases examine the population, medicines, infrastructure and review process behind a deployment decision.
Each is a composite teaching scenario, not a report of an actual product, patient or measured outcome. The suggested calls apply to the facts given; they are not clinical advice or deployment certification.
Try the interactive Deploy Check or read the explanations below.
01. Unvalidated population
Emergency department, Lagos teaching hospital
A sepsis triage model arrives with strong validation results from three American ICUs. Its heaviest input signal is pulse oximetry. The vendor offers a free pilot.
Suggested call: stop and resolve the gap
Pulse oximetry can overestimate oxygen saturation in people with darker skin. A model that relies on it could inherit measurement bias. Review subgroup performance and validate in the deployment population before clinical use.
02. Context omission
Primary-care clinic chain, southwest Nigeria
A prescribing assistant recommends first-line antibiotics straight from international guidelines. Several of those drugs are chronically out of stock in the region. Leadership wants it live this quarter.
Suggested call: stop and resolve the gap
Guidance that keeps naming unavailable drugs teaches clinicians to ignore the tool or improvise around it. It needs the national essential medicines list and real stock data behind it first.
03. Appropriate technology
Postpartum follow-up program, rural maternity network
A rules-based SMS system sends appointment reminders and danger-sign warnings to new mothers on feature phones. Local obstetric staff wrote and reviewed every message. It queues messages during outages, sends them when the network returns and has a nurse callback pathway.
Suggested call: proceed with safeguards
Local clinical review and a nurse escalation pathway make this a candidate for a monitored rollout. Confirm consent, delivery reliability and a fallback for urgent symptoms; queued SMS cannot deliver warnings while the network is down.
04. Dataset shift
Direct-to-consumer app, marketed across West Africa
A skin-lesion classifier reports dermatologist-level accuracy. Its training data is overwhelmingly lighter skin tones, and the pitch deck calls darker skin "an expansion opportunity."
Suggested call: stop and resolve the gap
Accuracy measured on one skin-tone distribution does not transfer to another. Its reported accuracy does not establish performance in the population the tool is being sold to.
05. Workflow mismatch
Chronic-disease clinics, public health system
A dosing tool hard-blocks every recommendation until renal function labs are entered. In the target clinics, patients pay for labs out of pocket and results take up to two weeks.
Suggested call: stop and resolve the gap
Hard-gating on labs the setting cannot deliver can delay treatment or encourage workarounds, including invented values typed in to unlock the tool. Safety features have to survive contact with the infrastructure.
06. Infrastructure assumption
Outpatient department, district hospital
A cloud transcription service promises to cut clinic documentation time in half. It has no offline mode. The pilot site has daily power cuts and clinicians on metered mobile data.
Suggested call: stop and resolve the gap
With no offline mode, every outage now interrupts documentation instead of degrading it gracefully. Offline-first is the entry requirement here, and this one fails at the door.
07. Anchoring bias
Pediatric telehealth line, high-malaria region
A fever triage bot trained on regional consultation data reads nearly every childhood fever as probable malaria and recommends antimalarials plus home observation.
Suggested call: stop and resolve the gap
Test whether the bot recognises urgent alternative diagnoses, including sepsis and meningitis. A high prevalence of malaria does not justify reassuring answers that miss danger signs. Resolve this gap before deployment.
08. Human in the loop
Discharge desk, urban general hospital
A drafting tool translates discharge instructions into Yoruba and Nigerian Pidgin. A clinician reviews and signs off every output before it reaches a patient. It saves each ward about an hour a day.
Suggested call: proceed with safeguards
Clinician review is a safeguard, not a guarantee. A monitored rollout should check translation accuracy with fluent reviewers, preserve urgent instructions, measure actual time saved and audit for errors that reviewers miss. The stated time saving belongs to this fictional scenario, not a measured deployment.
Evidence and further reading
These sources provide background for the review principles. They do not validate the fictional tools or outcomes in these cases.
- Sjoding and colleagues: Racial Bias in Pulse Oximetry Measurement — evidence relevant to population and measurement checks.
- WHO: Ethics and governance of artificial intelligence for health — accountability, human oversight and equitable deployment.
- WHO: Recommendations on digital interventions for health system strengthening — benefits, harms, feasibility and the health system around a tool.