When a large midwestern health system gave its primary care clinicians an AI scribe, the first reading was modest. On the day each clinician got access, the time they spent writing notes dropped about seven percent, roughly half an hour a week. The time they spent writing notes in the evening, between half past five and seven the next morning, did not change at all.1

Five months later the picture was different. By day 150, note writing time was down sixteen percent, close to seventy minutes a week per clinician, and evening note writing was down nineteen percent.1 Anyone who wrote the pilot report in the first month would have described a tool that saved a little time during the day and did nothing for anyone's nights, and they would have been wrong on both counts.

The number that went the other way

One measure moved in the opposite direction. Right after access, clinicians signed five percent more of their notes within a day of the visit. After about day ninety that effect was gone.1 The researchers offer a reading of why, and they are careful to frame it as their interpretation. The scribe keeps a transcript of the conversation, so a clinician finishing a note no longer has to hold the visit in memory, and the pressure to sign it the same day eases.1 If that is right, the early speed was an old habit running ahead of a new tool, and the people using it adjusted once they understood what the tool was doing for them.

The study is stronger than most of what has been published on these tools. It covers 220 clinicians across thirty-six clinics and 314,845 encounters. Every measure comes from the record system's own audit logs at the level of the individual visit, comparing the 150 days before each clinician got access with the 150 days after, where much of the earlier work leaned on metrics supplied by vendors.1 It is a preprint and has not been through peer review, and its current version changed several figures from an earlier one, including the note-closing number. Its senior author also sits, unpaid, on a research steering committee for the company that makes the scribe.1 I would still take its design over most of the evidence executives are being handed.

The output line

Now the number a business case gets built on. Work relative value units, the measure an American health system uses to count what a clinician billed for, showed no change at day zero and a two percent increase by day 150.1 Three physicians writing in NEJM AI last December looked at two randomized trials of the same class of product and put their conclusion in the title: AI scribes are not productivity tools, yet. In the best case from one of those trials, the tool saved about twenty-two minutes of documentation a day. In the more typical case from the other, it saved under a minute a visit.2

The time has no owner

The study can count the minutes that came back. It cannot tell you where they went, and it does not claim to. They might be going into longer conversations with patients or into evenings that end earlier, and either would be a defensible place for an hour to land. Neither shows up on the output line, and nothing in a pilot readout tells you whether anybody chose it.

I have sat through enough release planning to recognize the shape of this without any software in the picture. A team finds a way to take a couple of days out of a release. No one has an account waiting for those days, so they go back into the release, the next one occupies the same calendar it always did, and the improvement is both real and impossible to point at. The question that changes the outcome has to be asked before the improvement lands: if this works, what are we spending what it gives back on?

Capacity nobody has assigned gets spent anyway, by whoever is closest to it, on whatever is pressing hardest that week.

In a class I took this spring, one case study described an industrial manufacturer that redesigned its operating model and cut a weekly operations review from three hours to forty minutes, with the freed time pointed at strategic planning. The case did not say whether the change lasted. What caught my attention was the order of operations, since the destination for the recovered time was part of the redesign as it was described, set before anyone had a chance to fill it.

What your pilot is actually measuring

Earlier this month I looked at a randomized trial that took an AI drafting tool away from patent lawyers to see what skill was left underneath. That finding was about what a tool does to a person. This one is about what an organization does with the time a tool hands back. I have also written recently about a manufacturing productivity number that improved because its denominator shrank, and the family resemblance is plain. The arithmetic in both cases was sound, and the story people told from it went further than the arithmetic did.

The practical version is small and slightly annoying. Your pilot has a number attached to it, and somebody is going to read that number somewhere between day thirty and day ninety. In this study, a reading taken near the start would have shown about half the eventual time savings and a note-closing speedup that was gone by the end.1 So two questions. What would your number say at day one hundred and fifty? And has anyone told the people using the tool what the time it gives back is for?