Status
Pilot. Vendor testing pending. This page describes how we will test AI scribe notes for psychiatry. No vendor has been tested under it yet, and no results are published. Until results appear here, every review, ranking and research score on this site stays based on public vendor documentation.
What it measures
The CAW Psychiatry Scribe Benchmark looks at the note an AI scribe actually writes from a psychiatric encounter: whether it is accurate, complete, correct about medications and treatment, and safe to use. It does not measure price, integrations or vendor documentation; the documentation-based research score covers those (see our research methodology).
Synthetic encounters
Every encounter is synthetic and fictional. No real patient, client or session is used. The pilot uses three scenarios:
- Acute suicide-risk assessment: an urgent evaluation where risk and safety planning are the focus.
- PTSD medication management plus psychotherapy: a combined visit with medication review and a therapy component.
- Depression and anxiety follow-up: a routine outpatient follow-up visit.
We do not publish the reference checklists, the specific facts each note must contain, or any deliberate challenges built into the audio, so vendors cannot tune to the answer key.
Scoring rubric
Each note starts from a base of 100 points in four categories:
- Clinical accuracy: 30 points.
- Completeness: 25 points.
- Medication and treatment: 25 points.
- Safety and risk: 20 points. Graded against a checklist written for that specific case.
Each reference fact belongs to exactly one category, so no fact is counted twice.
Hallucination deductions
Deductions apply only to unsupported additions: content in the note that is not supported by the encounter audio. Clinically equivalent paraphrases and reasonable interpretations supported by the encounter are not hallucinations.
- Minor unsupported addition: minus 3 points.
- Major unsupported addition: minus 8 points.
- Critical invented completed action: minus 15 points. Examples: documenting a hospitalization, a prescription sent to a pharmacy, or removal of lethal means that did not happen in the encounter.
The total cannot go below 0. A fact that is in the encounter but recorded wrongly is scored as incorrect (0 for that item) and is not also deducted as a hallucination.
Credit rules
- Full: the fact is present and correct. Full points for the item.
- Partial: the fact is present but incomplete or vague. Partially correct information earns 50% of the item's points unless a predefined item-specific rule applies.
- Absent: the fact is missing. No points for the item.
- Incorrect: the fact is present but wrong. No points for the item.
Critical errors
Critical errors are defined in advance for each scenario and flagged separately from the point score, so a note can score well and still carry a critical error. A note with a critical error is labeled Unsafe as written. That label means the evaluated note needs correction before clinical use. It is not a verdict on every note the vendor produces.
Test process
- Identical frozen audio: every vendor receives the same recordings, versioned by checksum so any change is visible.
- Locked references: reference checklists are locked before any vendor output is seen.
- Same instructions: the same note type and instructions for every vendor.
- Multiple runs: where a product supports it, each encounter is run more than once.
- Clinician review: a psychiatrist scores every note, and a second clinician reviews every critical error before any vendor is named.
Separate from the research score
Benchmark results will be reported separately from the documentation-based research score. The research score rates what vendors publicly document; it does not measure note accuracy or clinical safety. The two will not be combined.
Limitations
- Audio is synthetic, not recorded from real sessions.
- Voices are text-to-speech, which may be easier or harder to transcribe than real speakers.
- The number of scenarios and runs is small.
- The benchmark is not validated for real-world patient populations, and results may not generalize to your practice.
Version control
This is Pilot Methodology v1.0.1. v1.0 was dated 2026-10-19; v1.0.1 is dated 2026-10-19. Any change to the rules gets a new version number and a dated changelog entry below. If the rules change, every tested vendor is rescored under the new version so results stay comparable.
Changelog
- 2026-10-19: v1.0.1. Clarifications made before any vendor testing, so no rescoring is needed: partial credit is 50% of the item's points unless a predefined item-specific rule applies, and clinically equivalent paraphrases and reasonable interpretations supported by the encounter are not hallucinations.
- 2026-10-19: v1.0 published (pilot). Vendor testing pending.