Measure coding automation and accuracy with clear denominators, representative samples and a method for reviewing errors and exceptions.
A coding automation rate of 70 percent and one of 38 percent can describe the same month in the same department. Accuracy behaves the same way: one sample of charts can support a 97 percent claim or a 65 percent claim, depending on what counts as a correct answer.
Neither number is false. Both are incomplete without the method behind them.
This guide sets out a method a coding manager can run, an auditor can check and a CFO can defend. Every worked number below is illustrative, not a MEDIQCODE™ result.
Coding automation rate: define the denominator first
Coding automation rate is the share of encounters coded and released to billing with no human touch. The numerator rarely causes the argument. The denominator does, and there are three defensible choices.
| Illustrative month | Encounters | Automation rate if this is the denominator |
|---|---|---|
| All encounters the department coded | 10,000 | 37.8% |
| Encounter types the vendor put in scope | 6,000 | 63.0% |
| In-scope encounters the engine actually processed | 5,400 | 70.0% |
| Released with no human touch (numerator) | 3,780 |
The 600 in-scope charts the engine never processed, because of missing documents or interface failures, disappear from the third definition. Report all three rates, and count unprocessed in-scope charts as not automated. OIG takes the same stance on missing items in its self-disclosure protocol, which treats sample items without support as errors rather than dropping them (HHS-OIG, 2021).
Define "no human touch" in writing too. A coder who opens a chart and clicks approve has touched it. The rate also only means something inside one workflow tier: in a coder-in-the-loop model every chart is reviewed by design, so the useful measure there is how often coders change the draft.
Pair the rate with accuracy on the released charts specifically. Lowering the confidence threshold raises the automation rate and pushes weaker charts out the door. An automation rate shown next to an accuracy figure measured on every chart, including the ones coders fixed, overstates both.
In MEDIQCODE, a coder approves every chart and every interaction is recorded in the audit trail, so a direct-to-bill automation rate doesn't apply.
Coding accuracy: choose the unit before the number
Accuracy needs a unit, and each unit hides a different kind of error. The 2013 AHIMA Foundation study with Cleveland Clinic scored coders with two code-level measures: recall, the share of reference codes that were also assigned, and precision, the share of assigned codes that appear in the reference set (Journal of AHIMA, 2013). Most other units build on those two.
Take an illustrative sample of 200 inpatient discharges with 2,000 codes in the reference set. The engine matched 1,920, missed 80 and added 60 the reference didn't support.
| Unit of accuracy | Result on the same sample | What it hides |
|---|---|---|
| Code-level recall | 96.0% (1,920 of 2,000) | Whether misses cluster in a few charts |
| Code-level precision | 97.0% (1,920 of 1,980) | Whether the extra codes change payment |
| Encounter level, every code correct | 65.0% (130 of 200) | Which errors actually mattered |
| MS-DRG agreement | 91.0% (182 of 200) | Code errors that leave the group unchanged but still reach quality and risk data |
| Netted code count | 99.0% (net variance of 20 codes) | Everything: 60 over-codes offset 80 under-codes |
Reject the last row outright. Over-coding and under-coding carry different consequences, one a potential overpayment and the other incomplete data and lost payment, and netting them describes nothing that happened. OIG's self-disclosure protocol takes the same position for damages estimates and bars netting underpayments against overpayments.
Match the unit to how the setting is paid. Inpatient studies should report Medicare Severity Diagnosis Related Group (MS-DRG) agreement alongside code-level results, because one principal diagnosis change can move the group. Emergency department professional coding needs E/M level agreement across 99281 to 99285. Risk adjustment needs Hierarchical Condition Category (HCC) level results.
Build a reference standard that holds up
Blind, credentialed and independent
The reference set decides the grade, so who builds it decides the result. Use credentialed coders (CPC, CCS, RHIA or RHIT, matched to the setting) who can't see the engine's output or the original codes while they abstract, and who don't work for the vendor. In the Cleveland Clinic study, the gold standard was set and validated by the organization's own coding leadership and quality team.
Check how well the reference coders agree with each other before you trust them. Percent agreement overstates reliability because some agreement happens by chance, which is why Cohen's kappa exists, and a biostatistics review argues that conventional kappa cut-offs are too lenient for health research (McHugh, Biochemia Medica, 2012). Report the statistic and the number of charts behind it.
Give reference coders the guidance that applied on the dates of service, including the ICD-10-CM Official Guidelines for Coding and Reporting and AHA Coding Clinic advice in effect then. Otherwise the audit grades the engine against rules that didn't exist when the chart was coded.
Adjudicate a sample of agreements too
The cheapest design runs the engine against the existing coded record and sends only the disagreements to an expert. It is also biased. In 1996, a Lancet analysis of diagnostic test studies showed that resolving only discordant results inflates accuracy estimates, and that the bias remains even when the referee is perfect (Hadgu, The Lancet, 1996).
In coding terms, when the engine and the original coder make the same mistake, nobody reviews it. Re-abstract a random sample of agreements as well.
Score the human baseline the same way
An engine scoring 94 percent means nothing on its own. It looks strong if the department's coders score 88 on the same audit and weak if they score 97. Run the original human codes through the identical reference and units. AHIMA's 2004 work group found human coding itself variable, with physician and coder agreement differing by as much as 20 percent (AHIMA, 2004). A baseline measured any other way isn't a baseline.
MEDIQCODE's platform records who coded, who audited and the reasoning behind every proposed code, so a draft and the coder's final decision can both be scored against the same reference.
How many charts are enough
A sample answers two questions: what the accuracy is, and how sure you can be. Sample size drives the second. Here is the same observed 90 percent accuracy at three sample sizes, with 95 percent Wilson score intervals.
| Charts reviewed | Correct | Observed | 95% interval |
|---|---|---|---|
| 30 | 27 | 90% | 74.4% to 96.5% |
| 100 | 90 | 90% | 82.6% to 94.5% |
| 300 | 270 | 90% | 86.1% to 92.9% |
Thirty charts can't separate a 75 percent engine from a 96 percent one. A clean result doesn't rescue a small sample either. Under the rule of three, zero errors in n charts still leaves a plausible error rate of up to about 3 divided by n: 10 percent for 30 charts, 1 percent for 300 (Hanley and Lippman-Hand, JAMA, 1983).
For planning at 95 percent confidence and an expected accuracy near 90 percent, plan on about 139 charts for a margin of plus or minus 5 points and 385 for plus or minus 3. Those counts apply to every stratum you intend to report, so separate ED, radiology and inpatient results each need that volume. Count encounters, not codes: codes in the same chart aren't independent, and treating them as separate observations makes intervals look tighter than they are.
Stratify before you sample. A simple random draw from all encounters fills up with short, high-volume charts, which flatters the combined figure. Sample each setting or complexity band separately, report each band, and weight the bands back to your real volume mix if you publish one number.
For a regulatory reference point, OIG's self-disclosure protocol requires a random sample of at least 100 items, recommends its free RAT-STATS software, and expects larger samples where claim types are mixed. It defaults to simple random sampling but allows stratified designs when the strata are documented. Sample design carries similar weight in CMS's risk adjustment data validation (RADV) audits. An accuracy claim built on fewer charts than an overpayment estimate would need deserves a follow-up question.
The disclosure card every accuracy claim should carry
A result without these fields can't be compared with anything else.
| Field | What to state |
|---|---|
| Scope | Setting, specialties, encounter types and every exclusion |
| Population and sample | Sampling frame, sample size per stratum, random selection method |
| Time window and versions | Dates of service, ICD-10-CM, ICD-10-PCS and CPT versions, engine version |
| Timing | Whether charts were scored before release or after payment |
| Unit and metrics | Code-level precision and recall, plus encounter, MS-DRG, E/M or HCC agreement |
| Error direction | Over-coding and under-coding reported separately |
| Reference standard | Who abstracted, credentials, blinding, independence from the vendor, inter-rater statistic |
| Adjudication design | Whether agreements were sampled or only disagreements reviewed |
| Human baseline | Original coders scored against the same reference and units |
| Uncertainty | Confidence intervals alongside every point estimate |
| Automation rate | Numerator and all three denominators |
Much of the card mirrors what OIG already expects in a self-disclosure review report: a stated objective, a defined population, reviewer qualifications, the characteristics measured and a documented sampling plan.
Reading a claim against the card
Take a hypothetical vendor slide reading 98 percent accuracy and 85 percent automation. Against the card, it answers none of the eleven fields.
Is the accuracy code-level precision, which ignores missed codes, or encounter level? Which denominator sits under the automation rate? Which charts were excluded, over what dates, judged by whom, against which human baseline?
A claim that can't answer those questions isn't necessarily wrong. It just can't be checked, and a number nobody can check shouldn't drive a contract.
Keep measuring after go-live
A pre-contract study is a snapshot. Once charts release, track four signals.
- Released-chart QA sample: Re-abstract a random monthly sample of automated charts, blind, and report it with the same card.
- Coder change rate: For reviewed charts, track how often coders alter the draft. A rate near zero can mean the engine is good or that coders stopped reading, and only an audit sample tells you which.
- Denials by coding source: Compare coding-related denials for automated and human-coded claims, using the remittance codes.
- Code-set cutovers: Re-run the QA sample in the weeks after October 1 and January 1, when ICD-10-CM and CPT versions change.
The cleanest place for the first full study is shadow mode, where the engine codes alongside coders without releasing anything.
How MEDIQCODE applies this method
When a hospital asks MEDIQCODE to prove accuracy, the design we propose is a blind retrospective test on its own closed discharges, with a sample adjudicated by its own coders. Until an adjudicated study is large enough to support a rate, we call the output a disagreement analysis, not an accuracy figure.
Because a coder approves every chart, a direct-to-bill coding automation rate isn't the number that describes our platform; draft accuracy and how often coders change drafts are. Any accuracy figure MEDIQCODE publishes will carry the disclosure card above. Before your next vendor review, send the card to every vendor on the shortlist.
Frequently asked questions
What is a good coding automation rate?
No benchmark is useful until the denominator, setting and workflow tier are fixed. A rate on a narrow, pre-selected scope can't be compared with one across all encounters.
Is agreement with our coders the same as accuracy?
No. Agreement measures how often the engine matches your original coding, mistakes included. Accuracy needs an independent, blinded reference that covers a sample of agreements as well as disagreements.
How many charts should an accuracy study include?
Enough to give a narrow interval in every specialty reported. At an expected accuracy near 90 percent, about 139 charts per stratum gives a margin of plus or minus 5 points at 95 percent confidence, and about 385 gives plus or minus 3.
How accurate is AI medical coding?
Published results vary widely because studies use different units, reference standards and samples. Read any figure against the disclosure card for how to interpret those figures.
Sources
- HHS Office of Inspector General. Health Care Fraud Self-Disclosure Protocol (issued April 17, 2013; amended November 8, 2021). PDF
- Dougherty M, Seabold S, White SE. "Study Reveals Hard Facts on CAC." Journal of AHIMA 84, no. 7 (July 2013): 54–56. PDF
- McHugh ML. "Interrater reliability: the kappa statistic." Biochemia Medica 22, no. 3 (2012): 276–282. PMC
- Hadgu A. "The discrepancy in discrepant analysis." The Lancet 348, no. 9027 (1996): 592–593. Abstract
- Hanley JA, Lippman-Hand A. "If nothing goes wrong, is everything all right? Interpreting zero numerators." JAMA 249, no. 13 (1983): 1743–1745. JAMA Network
- AHIMA e-HIM Work Group on Computer-Assisted Coding. Practice brief, Journal of AHIMA 75, no. 10 (Nov–Dec 2004). Copy hosted by NCVHS