Doctronic's AI has been renewing prescriptions (under a doctor's review) in Utah since January under the state's AI regulatory "sandbox." The first of its kind in the country, this temporarily sets aside regulation — in this case, the licensing rules that require a doctor to authorize every prescription renewal — so a company can test a product on real patients under state supervision.
I've been writing about the pilot since launch. Ten months and 2,317 renewal requests later, the company hasn't reported any adverse events. But that's all the data we've got. The state reads Doctronic's reports, but no one outside the company officially checks them.
This is a little weird because what's happening with Doctronic in Utah is an experiment, and in medicine, experiments are supposed to be checked by people with no stake in the result. A large drug trial is run by the drug company, but it has safeguards we won't see with the Doctronic pilot:
- An independent data monitoring committee reviews the safety data as it comes in
- Patients are scheduled for follow-up visits, so there is active surveillance for problems
The physicians grading the AI work for Doctronic, and as far as I can find, nobody follows up with patients after a renewal to see if any problems arose.
| Large drug trial | Doctronic pilot | |
|---|---|---|
| Who reviews the safety data | An independent committee | Physicians Doctronic hires; the state reads Doctronic's reports |
| How harm is found | Scheduled follow-up visits and labs | Adverse events counted when reported; no patient follow-up that I can find |
| Comparison | A control group | None |
| Outside audit | FDA inspects trial sites and audits the data | None official yet; ten volunteer clinicians now reviewing the reports |
| What gets measured | Set in advance and publicly registered | Adjusted "by mutual agreement" with Doctronic; monthly reports not public |
What Utah set up
Utah's sandbox is run by the Office of AI Policy, created in 2024. Theoretically any industry could participate, but all nine agreements so far are health-related.
Doctronic's was the first to let an AI do some component of a physician's job. For now, a physician still signs off on every renewal before it reaches the pharmacy. Doctronic's agreement with the state specifically stipulates no independent evaluation, only self-reporting. Doctronic sends the state monthly reports on renewals approved and denied, plus a random sample of cases, and the office reviews them.
The state published a summary for January through April in May, which I wrote about then: it showed that the AI was more cautious than the physicians auditing it. The Utah Investigative Journalism Project obtained the monthly reports through September, and KUER, Salt Lake City's NPR station, published the figures last week:
- 2,317 renewal requests, January–September
- Fewer than 600 completed by the AI, per Zach Boyd, a mathematician who directs the state's Office of AI Policy. The rest stopped short — patients who quit partway or decided they wanted a human.
- 32 cases where the reviewing physician asked a second physician to check the AI's work
- 13 cases flagged for safety, where both physicians disagreed with the AI
- 0 adverse events reported
Utah's Boyd told KUER that only now is anyone outside the company looking at these numbers: ten clinicians who volunteered to review the reports for the state, and it apparently does not count as an official audit. I've reached out to him for more detail but haven't heard back yet.
Evaluation, Kinda
On October 5 the state arranged to pay six organizations — including the Coalition for Health AI and Stanford's Clinical Excellence Research Center — to verify the claims of companies involved in the sandbox and audit how accurately they report the AI's decisions. Cam Bronson, a legal and policy analyst in Utah's AI office, said the evaluator network grew out of the strong feedback the state got from critics, including the state medical licensing board, on the Doctronic pilot.
Three problems with this evaluation:
- Evaluators are paid by the company they're evaluating. Evaluators negotiate their fees directly with the companies they're checking.
- The evaluation itself isn't defined yet. Bronson told Healthcare IT News they're still working out what an evaluation should look like, clinically and technically.
- It's not clear it covers Doctronic. KUER describes the evaluators as being for new companies entering the sandbox. The state has said an extension of Doctronic's agreement will look like the new agreements, but no extension has been announced.
Plus, there's nothing to compare it to
In March I argued that if we're going to grade the AI, we should grade the doctor. Without that, even if Doctronic's numbers are accurate, we've got nothing to compare them to.
Doctronic's 13 cases flagged for safety are where the reviewing physicians disagreed with the AI. This assumes those reviewing physicians were right — but we can't assume that. In a study I cited in March, Cleveland Clinic pharmacists reviewing refill requests ordered a lab, scheduled a visit, or changed the drug twice as often as usual physician care did. That suggests usual physician care misses things, but the study doesn't measure exactly the same thing as Doctronic's reports, so it can't serve as the comparison.
Self-driving cars have something to compare to
Another AI project does have an available comparison: self-driving cars. Using state police crash reports and mileage data, Waymo calculates that human drivers in the cities where it's operating get into about 3.8 injury crashes per million miles. Its robotaxis have 0.7 injury crashes per million miles, so 82% fewer.
And we have data from outside the company. In July the Insurance Institute for Highway Safety, the insurance industry's research group, compared about 50 million driverless Waymo miles with 222 billion human-driven miles in the same cities and found 81% fewer injury crashes.
| Self-driving cars (Waymo) | AI prescription renewals (Doctronic) | |
|---|---|---|
| Human baseline | Police-reported injury crashes per mile, same cities | Nothing measured the same way for routine refills |
| The AI's own numbers | 82% fewer injury crashes than humans over 271 million driverless miles | 0 adverse events, 13 safety flags, fewer than 600 completed renewals |
| Who reports the numbers | Waymo, plus crash reports federal rules require from every driverless-car company | Doctronic, in monthly reports to the state |
| Independent check | IIHS: 81% fewer injury crashes, against 222 billion human-driven miles | Ten volunteer clinicians, not an official audit; paid evaluators signed in October, coverage of Doctronic unclear |
What's next
Doctronic's agreement ran through October 24. Earlier this month the state said it was negotiating a 12-month extension with more reporting; as of this writing I haven't seen anything definite as to whether that extension is going to happen. The Office of AI Policy owes the Utah legislature its annual report on every sandbox agreement by November 30. Meanwhile the sandbox has already moved past refills, though not with Doctronic: on October 5 the state approved Nolla Health, the first AI in the country allowed to write new prescriptions, for acne.
If Doctronic's extension comes through, I'm hoping it'll have what any clinical trial would have: an independent monitor with full access to the data, watching both the AI and a comparison group of usual refill care.
Sources: KUER / Utah Investigative Journalism Project · Utah Office of AI Policy, Authorized Pilots · Utah Office of AI Policy, Doctronic pilot page · Utah Office of AI Policy, May 2026 report · Mello, JAMA Health Forum · Healthcare IT News · Washington Post · FDA, Data Monitoring Committees guidance · Waymo safety impact data · IIHS · Utah Dept. of Commerce, Oct. 5