Blog

From Writing Notes to Editing Them: A New Framework for Reviewing AI-Scribed Notes

Published 2026-11-20 · Last updated 2026-11-20

Changelog
  • 2026-11-20: Published.

Physicians are shifting from writing clinical notes to editing AI-generated ones, and medical education has not kept up. That is the premise of a comment published October 12, 2026, in npj Digital Medicine, in which researchers at Carnegie Mellon and the University of Pittsburgh School of Medicine define the review of AI-generated clinical documentation as a teachable competency, taught through simulation [1]. The framework has four domains: detecting errors and omissions, judging whether a note reflects the clinical reasoning behind the visit, generating notes independently when AI is unavailable or inappropriate, and calibrating trust across contexts [1]. For psychiatry practices, the review it describes is already the daily job: the clinician who signs an AI-scribed note owns it.

Why review is the job now

The scale arrived first: in one integrated health care system, ambient scribes documented more than 2.5 million encounters in a single year [2]. The evidence on what those drafts contain points the same way: review is where quality is decided.

In the first randomized trial of ambient scribes, 238 UCLA outpatient physicians were assigned to Microsoft Dragon Ambient eXperience (DAX) Copilot, Nabla, or usual care. Clinicians rated clinically significant inaccuracies as occurring "occasionally" for both products, at 2.7 and 2.8 on a five-point scale [3]. The trial's time measure also warns anyone budgeting saved minutes: time-in-note does not count the minutes spent editing a draft inside the vendor platform [3].

Our own clinical pilot found the same shape of problem in psychiatry notes. Two vendors were flagged "Unsafe as written" on two of three encounters each: Mentalyc scored 79.0 with two critical flags, and Upheal scored 63.0 with three. The strongest performer, Heidi, scored 96.0 with zero critical flags and still missed risk-related documentation, including an explicit outpatient safety determination in a suicide-risk assessment. Full results are on our clinical pilot results page, with Heidi's pilot detail in our Heidi review. The signature makes the review binding: Medicare's signature fact sheet states that if you use a scribe, including artificial intelligence technology, you sign the entry to authenticate the documents and the care you provided or ordered [4]. The framework gives that obligation a curriculum.

The four domains, and what each asks of a psychiatrist

  • Detecting errors and omissions. The first domain asks whether a trainee can identify factual errors, omissions, hallucinations, and misattributions in an AI-generated note [1]. In psychiatry, the failure modes are specific: our pilot caught a wrong drug identity, a drug attached to the wrong person, a diagnosis the examination contradicted, and findings that never happened in the session. The five checks in our pre-signing routine target those exact places: drug names, drug ownership, diagnosis against examination, observed findings only, and what was left out. Error detection is the domain the routine already trains.
  • Judging whether the note reflects clinical reasoning. The second domain asks whether a trainee can determine if a note reflects the clinical reasons behind the encounter, not merely its facts [1]. This is the domain that matters most in psychiatry, where the note is mostly reported speech, attributed history, and inference. A note can list the facts correctly and still miss the reasoning: in our pilot, a Heidi note recommended same-day emergency evaluation for a patient at acute suicide risk but did not state that a routine outpatient plan had been judged unsafe. The facts were present. The judgment was not. Checking the reasoning means reading the assessment and plan against the history and examination and asking whether the note explains why, not only what.
  • Generating notes independently. The third domain asks whether a trainee can produce a clinically sound note on their own when AI is unavailable or inappropriate, or when a patient does not consent to its use [1]. Patients do decline. In a 2025 study of 103 patients at one academic health center, 81.6 percent consented to ambient documentation after a basic explanation, but only 55.3 percent did once details about AI features, data storage, and corporate involvement were disclosed [5]. More than a third, 35.0 percent, said they were more likely to censor what they said about mental health [5]. Our consent post covers what to tell clients and how to document the choice. A clinician who cannot write the note without the tool cannot serve the patient who declines it.
  • Calibrating trust across contexts. The fourth domain asks whether trainees adjust their reliance on AI output across clinical contexts [1]. Trust should vary twice, by situation and by vendor: a straightforward follow-up deserves less scrutiny than a suicide-risk assessment, and a vendor whose drafts have needed heavy correction deserves more. Documentation scores do not settle that question: Mentalyc (75/100, #5) and Upheal (71/100, #6) among vendors with a documented BAA on our research scorecard document well and still produced the pilot's critical flags, a split our two-rankings post works through in full. Nabla, the product that saved time in the randomized trial, scored 65 on our research scorecard and is not ranked, because no publicly documented BAA was found; our Nabla review carries the detail. Microsoft Dragon Copilot (59/100, #13), the trial's other product, sits lower on the same scorecard. Time savings, documentation quality, and note accuracy are three kinds of evidence, and trust calibration means keeping them separate.

How simulation teaches it, and what a practice can borrow

For each domain, the framework proposes learning objectives, simulation exercises, and ways to assess competency [1]. "Simulation lets trainees practice catching an AI's mistakes in a low-stakes way where it won't affect patient outcomes," says Reed Van Deusen, associate professor at the University of Pittsburgh School of Medicine and a co-author [2]. When errors are built into simulated encounters, instructors can see exactly what a learner misses and give targeted feedback [2].

A practice can borrow the method. Run a planted-error drill: take a de-identified draft, insert one wrong drug name, one misattributed statement, and one omitted counseling point, and ask each clinician to find all three. Audit a sample of signed notes each month against the same five checks. The routine stays short enough to run before every signature; the drill is how the routine stays sharp.

What the framework does not do

The authors state that the framework is preliminary and call for empirical testing [1][2]. It is a proposal for what to teach, not a validated measure of who has learned it, and it carries no patient-outcome data. That limit bounds what the framework supports: a structured way to build review skills, not proof that the training works. The next evidence to watch is whether programs that adopt it can show their trainees catching more errors in real caseloads.

The bottom line: The work of documentation has moved from writing notes to editing them, and a new framework names the four skills that editing requires: catch the errors, check the reasoning, write the note yourself when the tool is not an option, and adjust trust to the situation and the vendor. Train those skills deliberately, and keep the five-check routine as the daily form they take.

Sources

  1. Raka Y, Van Deusen R, Wiberg H, "From authoring to editing: a competency framework for trainee use of ambient AI clinical documentation," npj Digital Medicine 9, 785 (2026), published October 12, 2026, https://doi.org/10.1038/s41746-026-03351-5 (opened 2026-11-20)
  2. Medical Xpress, "As use of AI scribes spreads, researchers propose a framework for preparing medical trainees," November 17, 2026, https://medicalxpress.com/news/2026-11-ai-scribes-framework-medical-trainees.html (opened 2026-11-20)
  3. Lukac PJ, Turner W, Vangala S, Chin AT, Khalili J, Shih YT, Sarkisian C, Cheng EM, Mafi JN, "Ambient AI Scribes in Clinical Practice: A Randomized Trial," NEJM AI, 2025;2(12):AIoa2501000, https://pmc.ncbi.nlm.nih.gov/articles/PMC12768499/ (opened 2026-11-20)
  4. Centers for Medicare & Medicaid Services, "Complying with Medicare Signature Requirements" (MLN905364, July 2025), https://www.cms.gov/mln905364-complying-with-medicare-signature-requirements (opened 2026-11-20)
  5. Lawrence K, Kuram VS, Levine DL, Sharif S, Polet C, Malhotra K, Owens K, "Informed Consent for Ambient Documentation Using Generative AI in Ambulatory Care," JAMA Network Open, 2025, doi:10.1001/jamanetworkopen.2025.22400, https://pmc.ncbi.nlm.nih.gov/articles/PMC12284739/ (opened 2026-11-20)

Who curates this site

Curated by a practicing, board-certified psychiatrist, not named to avoid conflicts. From the site data: 19 vendors tracked, 354 distinct external source pages linked. Last site update 2026-11-20; last targeted vendor verification 2026-11-12; last complete research audit 2026-11-04. How we work · Editorial policy.