For about two years a lot of teams ran on the same quiet assumption. An automated decision was a black box, and the box was somehow protective. If nobody could reconstruct how the model reached its answer, nobody could be cross examined on it.
On 9 March 2026 that stopped holding.
In Estate of Gene B. Lokken et al. v. UnitedHealth Group (D. Minn., 0:23-cv-03514, filed 14 November 2023), a federal magistrate judge granted broad discovery. The order compelled production across most of the categories the plaintiffs had sought, including material going to whether the automated review tool was designed to override the clinical judgement of treating physicians.
The next part needs saying precisely.
Those are the plaintiffs' allegations. The insurer disputes the characterisation and maintains the tool does not make coverage determinations. Nothing has been decided on the merits, and a discovery order settles nothing about who is right.
Now the narrow point that nobody disputes, which is the entire article. How an automated decision was reached is discoverable. Courts will compel it. The design of the system, what it was configured to do, and what it produced in one individual case all sit inside the scope of what a plaintiff can ask for and a judge can order handed over.
If your position assumed the internals stay private, your position has a hole in it.
The economics that got everyone here
The Senate Permanent Subcommittee on Investigations released "Refusal of Recovery" on 17 October 2024, under Chairman Richard Blumenthal. The staff report found UnitedHealthcare's post-acute prior authorization denial rate went from 10.9% in 2020, to 16.3% in 2021, to 22.7% in 2022. Its skilled nursing facility denial rate rose ninefold between 2019 and 2022, from 1.4% to 12.6%. That happened over the period automated review under naviHealth's nH Predict was expanded.
Set beside that, the Lokken plaintiffs plead that roughly 0.2% of policyholders appeal, and that over 90% of denials are reversed on internal appeal or in ALJ proceedings. Allegations, again. Not findings.
Read the two together and the incentive is plain. A decision that costs almost nothing to issue and a lot to contest gets issued more often. No malice required, just cost sitting on one side of the transaction.
Then somebody contests it, and the whole thing turns into a documentation contest. The side with the better record of what actually happened wins that. For years the institution won by default, holding all the records while nobody outside got to read them. The Lokken order is what it looks like when that stops being true.
The same shape outside healthcare
Health insurance got here first because the harm is visible and the plaintiffs are sympathetic. The structure has nothing to do with insurance.
If you run an automated system that contributes to a decision someone can contest, you are standing in the same place. Credit and collections. Hiring and candidate screening. Tenancy applications. Fraud holds on accounts. Claims triage. Pricing. Account terminations that end somebody's income.
In every one of those, a court asks the same four questions. What did the system see. What did it output. Who looked at it. What did that person do about it.
29 states already ask in advance
By 1 July 2026 there were 109 AI laws across 29 states. A few that reach an ordinary operator rather than a frontier lab:
Illinois HB 1806 has been in force since 1 August 2025. It bars a licensed professional from letting an AI system make independent therapeutic decisions, produce treatment plans without licensed review, or detect emotions. IDFPR enforces it, with civil penalties up to $10,000 per violation. That is a bright line rather than a disclosure duty, and you cannot paperwork your way past it.
California AB 489 took effect on 1 January 2026. It bars AI systems and the people who develop or deploy them from using terms that imply a health care licence, and it applies to any deployer. That makes it the California AI health law most likely to catch an ordinary operator: AB 3030 binds facilities and practices, SB 1120 binds payors.
Texas HB 149 has applied since 1 January 2026. A provider using an AI system in relation to a health care service discloses that use to the recipient no later than the date the service is first provided, in plain language, free of dark patterns. Emergencies get a carve out, and disclosure still has to follow as soon as reasonably possible.
Utah is narrower than most of the commentary claims. Under Utah Code Title 13 Chapter 77, a supplier using generative AI in a consumer transaction discloses when the individual makes a clear and unambiguous request. Proactive disclosure only bites where two conditions meet: someone providing services in an occupation the Utah Department of Commerce licences or certifies, and a use that counts as a high risk AI interaction. The blanket reading people keep repeating came from a section that was repealed.
Colorado is the odd one. Enforcement of SB 24-205 was suspended by court order on 27 April 2026 in xAI v. Weiser. SB 26-189 was signed on 14 May 2026 with duties starting 1 January 2027, and the AG's rulemaking is still open, proposed rules filed 11 August 2026, comment running to 26 October 2026.
Waiting for Washington to tidy this up is not a plan. As at August 2026 no federal statute or executive order has preempted any of it. EO 14365 of 11 December 2025 created a DOJ AI Litigation Task Force and asked Congress for preemption legislation. Asking is not displacing. EO 14409 of 2 June 2026 is silent on state law. The Obernolte-Trahan "Great American Artificial Intelligence Act" went out as a discussion draft on 4 June 2026 and has never been formally introduced.
Every one of those laws creates a factual question about what your system did and who reviewed it. "We don't log that" is an answer to that question. It is just a bad one.
Holding no record is not the safe position
This is where the old instinct still bites. Plenty of operators quietly prefer thin logs, on the theory that what you never wrote down can never be produced against you.
That theory reads the risk backwards. Missing records do not make the question go away, they decide who gets to answer it. Discovery still runs. The plaintiff still builds a timeline out of the parts of your system you did keep, your vendor's records, and your staff under oath years after the fact. Then you respond with recollection.
I would rather hand a court a boring, complete, independently verifiable record showing a human reviewed the recommendation, than have my best engineer explain from memory what the model probably did in March.
What a real record has in it
Per decision, not per month. Five fields:
- What the system saw. The actual inputs at decision time, not a summary written afterwards.
- What it recommended, with the version of the model or rules that produced it.
- Who reviewed it. A named human, identified, not a role.
- Whether they agreed or overrode it, and what they wrote when they did.
- When each of those happened, in a sequence that can be shown not to have been altered after the fact.
Then retention. Set it against the limitation period for the claims you could actually face, not whatever your logging stack defaults to. A 90 day rolling window is fine for debugging and useless for a claim that lands in year four. Lokken was filed in November 2023. The discovery order came in March 2026. Count the months.
This is the work my company does. APIR issues cryptographically signed credentials to AI agents (Ed25519, W3C Verifiable Credentials), keeps hash-chained records of what they did, and scores their behaviour independently, so an organisation can evidence what an autonomous system actually did when a regulator, an insurer or a court asks. Anyone can check an agent free at apir.ai/check-agent, no account, and we have never marketed that page once.
We don't promise results. We prove whatever happens.
You do not get to pick the day somebody asks how your system decided. You only get to pick whether you wrote it down while you still could.