In clinical settings, speed matters—but trust matters more. If a conclusion cannot be explained, replayed, or audited, it rarely enters the real care pathway.
Black boxes demo well; they are hard to staff
Demo environments allow outputs that "look accurate." On-call environments require physicians to understand the basis, QA to trace decisions, and compliance to prove data boundaries. Those are different standards.
Auditability is not a speed bump
Making feature contributions, confidence intervals, and override records default output actually reduces tumor board friction and repeat communication. Audit capability eventually becomes efficiency.
One line for decision-makers
When selecting a medical AI system, do not ask only "how accurate is it?" Also ask: "if something goes wrong, can we explain it, review it, and keep data in bounds?"