Imagine the headline: What happens when healthcare AI makes the call without clinical review?
Imagine the headline.
“AI algorithm denied care. Patient’s condition worsened. Family sues for wrongful death.”
That’s an illustrative headline, not a real one. But it closely reflects what’s actually being alleged in an active federal lawsuit.
A class action, originally filed in November 2023 by the families of two deceased Medicare Advantage members and still active with broad discovery ordered as recently as March 2026, accuses UnitedHealth Group of using an AI tool to guide post-acute care decisions. The complaint, built on a STAT News investigation, alleges the algorithm had a 90% error rate, measured by how often denials were reversed on appeal, and that internal pressure pushed clinical staff to keep patient rehab stays within a narrow margin of what the algorithm predicted, regardless of clinician judgment.
The plaintiffs claim the resulting denials led to worsening health for patients, and in some cases, death. UnitedHealth disputes that the tool was used to make coverage decisions at all, stating that determinations are made by physicians following CMS guidance, not AI. The case is still being litigated.
Whatever the outcome, the allegation at the center of it, that an algorithm’s output moved from advisory input to the decision itself, without a clear point where a clinician’s judgment could override it, is exactly the failure mode healthcare AI governance is supposed to prevent.
This isn’t unique to healthcare, either. On October 2, 2023, a Cruise robotaxi in San Francisco struck a pedestrian who had already been hit by a separate, human-driven car. The autonomous vehicle braked, then attempted to pull over, as its software was designed to do. In the process, it dragged the woman roughly 20 feet down the street before coming to a stop with its rear wheel on her legs. She survived, but with severe injuries.
That alone would’ve been a difficult story for Cruise to recover from. The company’s actual downfall came from what happened next: regulators found Cruise had withheld the dragging detail from its initial report of the incident.
California revoked the company’s operating license. Cruise also suspended driverless operations nationwide, paid a $500,000 fine for the false report, and ceased operations entirely, with its self-driving software no longer running on any public road.
The technical failure was bad enough. The governance failure, an incomplete report, a missing detail, a system that couldn’t show regulators exactly what happened helped trigger the end of the business.
Cigna faced a comparable case in healthcare, Kisting-Leung v. Cigna Corp., built on a ProPublica investigation that found Cigna medical directors used an internal algorithm called PxDx to deny more than 300,000 claims over a two-month period in 2022, spending an average of 1.2 seconds reviewing each one, without opening individual patient files.
Cigna disputed the allegations, stating that PxDx reviews occur after treatment has already been provided, so the process doesn’t deny care, and that it reflects standard industry practice.
In March 2025, a federal judge allowed the case to proceed on a subset of claims, finding Cigna’s use and approval of the algorithm could constitute an abuse of discretion under the health plan’s own terms, while dismissing claims from three plaintiffs whose specific denials weren’t shown to have gone through PxDx. The case remains active.
Different companies, different tools, the same underlying pattern: a system built to flag or predict became, in practice, the system that decided. And the human review step, or the outside scrutiny, that was supposed to catch its blind spots went missing or became a formality.
The pattern behind AI failures
None of these stories are about AI being wrong. Prediction models are wrong sometimes, which is expected, and it’s why a human review step exists in the first place. The failure in these cases is the same: the moment where a human was supposed to catch the error, or override the system, either didn’t exist, wasn’t properly followed, or couldn’t be verified after the fact.
In each case, a person should’ve been able to step in before the AI’s output reached the end users, and those guardrails didn’t hold.
Proper healthcare AI governance
Here’s what proper governance in healthcare AI requires:
A human decision point built into the workflow itself. A denial, a discharge, a treatment plan, none of it reaches a member without a clinician signing off first.
A complete, honest record of what happened. The crash itself caused serious injuries. What compounded it was Cruise’s incomplete report of what happened. Any AI system in a regulated, high-stakes environment needs a record ready for scrutiny when a regulator, an auditor, or a plaintiff’s attorney asks, not one assembled after the fact.
An error rate the organization already knows. If a system is wrong 90% of the time on appeal, someone inside the company should’ve known that before a plaintiff’s attorney did.
The Pager Health℠ approach to care navigation governance
This is the standard Pager Health’s care navigation platform is built on: AI engages and assesses, but a clinician decides every time, with no path for a recommendation to reach a member without clinical review. Every interaction is logged end to end, tied to a durable, append-only audit record, the kind of account that holds up under scrutiny rather than needing to be reconstructed after something’s already gone wrong.
AI never renders clinical decisions. It makes the right conversation happen, then hands the clinician a complete picture. And when a member reaches out directly, that access is fast in practice: the median time to connect with a nurse is 6 seconds.¹
The takeaway
These stories about AI gone wrong share a common thread: a missing guardrail. Whether the model itself was accurate or not, the same gap shows up in every case. They all lacked a guaranteed point where a human could stop a bad outcome before it reached the end user, and a record proving that process would’ve happened.
¹ Based on Pager Health historical performance data