AI Health Apps: Plan a Safe Handoff When Confidence Is Low
When an AI health feature cannot give a dependable result, the app needs more than a disclaimer. Design uncertainty states, safe next steps and a route to qualified human assessment.
Recent coverage of AI in dermatology, including BBC News reporting on Bradford skin cancer waiting times and a Medscape overview of developments in dermatology AI, makes a useful product question timely: what should an app do when its AI cannot provide a dependable result? The answer is not to hide uncertainty behind a generic disclaimer or make the model guess. A health feature needs defined uncertainty states, safe fallback content and a practical route to qualified human assessment. For product and engineering leaders, that means treating handoff as part of the feature’s core workflow, not an exception added after model integration. The example below is hypothetical; it does not describe an existing product.
Define what “not dependable” means before building
A low-confidence response should be an explicit product state, not simply a score that engineers interpret differently from clinicians, designers or support staff. Start by documenting the feature’s intended use and the situations in which it must not return a substantive assessment. For a hypothetical image-based skin feature, these could include an image that is too blurred, incomplete information, an input outside the feature’s intended scope, or a model result that fails an agreed reliability rule. The specific thresholds and exclusions require appropriate clinical and regulatory input; they should not be invented by the delivery team. Translate each approved condition into a testable system outcome and a plain-language user message.
Avoid presenting a confidence percentage as if it were a probability that a diagnosis is correct. A number without context can create false reassurance or unnecessary alarm, and users may treat it as clinical certainty. Instead, define product-facing states such as “unable to assess this image,” “more information needed,” and “assessment not available.” Each state should have a clear trigger, a user-visible explanation, permitted next actions and an owner responsible for reviewing the rule. In the technical specification, keep model confidence, input-quality checks and operational failures distinguishable. They have different causes and may require different remedies, even if the app presents each as a reason it cannot provide a dependable result.
- Create a decision table linking each uncertainty condition to its trigger, message, allowed actions, handoff destination and test case.
- Get qualified clinical and regulatory review of intended use, exclusion criteria and user-facing wording before implementation.
Make the fallback useful without acting like a diagnosis
When the system cannot assess an input, the app should say so directly and explain what the user can do next. A safe fallback might offer a retake guide for a technically inadequate image, let the user stop and seek a qualified assessment, or show approved information about how to access an appropriate service. Which option is suitable depends on the intended use, jurisdiction and service model. Do not imply that retrying will necessarily produce an answer, that a reassuring message rules out a condition, or that an app can determine urgency unless that function has been properly designed and governed. Keep general educational content visibly separate from any personalised assessment.
The fallback also needs to work when the preferred route is unavailable. If a handoff depends on a clinic directory, booking service or staffed support queue, define what the app displays when that integration fails or no appointment is available. Give the user a meaningful alternative rather than a dead link or an indefinite loading state. Do not promise response times unless the service owner can support them. For a cross-market product, review local service pathways and language rather than assuming one referral route works everywhere. Make information accessible, use direct language, and ensure the next step remains clear to someone who declines image sharing or does not have an account.
- Write fallback copy for the normal path, unavailable-service path and user-declines-handoff path.
- Confirm who owns destination content, directory accuracy, service availability and updates after launch.
Engineer and test the handoff as a complete workflow
A handoff is a product and integration boundary, not just a button. Specify what information, if any, may be passed to a qualified human, what the user must consent to, and whether the person receiving it can actually access the relevant context. Minimise transferred data and explain the transfer before it happens. Define how the app records the state reached, handles a failed submission and prevents duplicate requests after retries. If assessment takes place outside the app, show the user that the automated feature has stopped and identify the separate service they are contacting. Keep audit events useful for operations while avoiding unnecessary collection of sensitive health information.
Test the workflow with representative failure cases, not only successful model responses. Include poor-quality or unsupported inputs, missing data, unavailable handoff services, interrupted connectivity, user cancellation and repeated taps. Verify that the app never silently converts an uncertain result into a confident-sounding answer, and that the next-step content remains visible with assistive technology and larger text settings. Monitor operational measures such as the share of sessions reaching each uncertainty state, handoff completion and integration failures, subject to privacy and governance controls. These measures can reveal product friction; they do not by themselves establish clinical safety or performance. Assign an owner and review process for changes to thresholds, model versions and fallback content.
- Include uncertainty and handoff scenarios in acceptance criteria, QA scripts and release checks.
- Set an accountable review process for model changes, escalation rules, service links and incident reports.
A dependable AI health feature is defined partly by the moments when it declines to answer. Make uncertainty explicit, connect each state to a safe and useful next step, and verify that any human route is real, consent-aware and operationally supported. The product team should agree these behaviours with qualified clinical, regulatory, privacy and service stakeholders before committing to build scope. A clear handoff plan also gives engineering a concrete basis for interface states, integrations, testing and post-launch monitoring—without suggesting that AI replaces clinicians.