Computer Assisted Auditing for High Volume Medical Coding

1
Computer Assisted Auditing for High Volume
Medical Coding
Daniel T. Heinze, PhD Peter Feller, MS Jerry
McCorkle BA Mark Morsch, MS   A-Life Medical,
Inc. - San Diego, CA
2
Overview

• Introduction and Background
• Problem Statement
• Objectives
• Overview of Statistical Methods and Issues
• The User Connection
• Research Questions and Methods
• Sample Selection
• Specification and Control Limits
• Overview of the Methodology
• Examples RAT-STATS and CoAudit
• Conclusion
• Review of Objectives
• Results

3
Introduction and Background

• Problem Statement
• Quality control and assurance for manual coding
  of medical records has traditionally been done
  with semi-formal or ad hoc methods that do not
  scale to high volume production environments and
  the results of which are not comparable across
  time or between coders or auditors
• Auditing process is tedious, time-consuming and
  paper-based
• High volume medical coding due to Computer
  Assisted Coding (CAC) requires production
  oriented quality control and assurance
• But, there are issues to be addressed related to
• How an auditor can score a coded document
• How to measure and compensate for auditor
  variability
• How to make the process efficient

4
Introduction and Background

• Objective Computer Assisted Auditing
• Automate the audit process workflow
• Sampling method that computes sample sizes and
  randomly selects documents
• Scoring method provides results that can be used
  with statistical QA and production control
• Scoring method tracks to methods and results of
  purely human audits
• Audit results can be reported and interpreted by
  the user, forming the basis for effective
  decisions
• Audit results are meaningfully comparable between
  auditors and auditees and across time

5
Introduction and Background

• Overview of Statistical Methods and Issues
• Methods
• Accuracy Measures Recall and Precision Type I
  and Type II errors False Negatives and False
  Positives
• Inter-rater Agreement Kappa-Statistic, et al.
• Significance Statistics Chi-Square, et al.
• Issues
• Not directly applicable for issues like order and
  association
• No control for auditor variability
• Inter-rater and significance statistics are hard
  to apply to problems with high numbers of choices

6
Introduction and Background

• The User Connection
• Solution needs to be user (i.e. real world coders
  and managers) driven but also based on valid
  statistical methods
• To draw valid conclusions about audit results,
  sampling methods must be fundamentally sound
• We worked with a large group of medical coding
  billing organizations to develop a consensus on
  an audit scoring method that
• Is consistent with all aspects of information on
  which auditors evaluate coding
• Produces scoring that tracks to auditor
  evaluations
• Is easy to use in a computerized auditing system

7
Research Questions and Methods

• Sample Selection
• How many charts/reports need to be audited?
• Specification and Control Limits
• How are individual charts/reports scored?
• When is the coding process in/out of control?
• Overview of the Methodology
• Putting it all together.

8
Research Questions and Methods

• Sample Selection
• Selecting a fixed number or fixed percentage of
  charts/reports to audit creates the risk of
  either
• Under sampling the data thus producing results
  that are not valid.
• Over sampling the data thus producing undue audit
  costs.
• Statistically sound methods exist for sample size
  selection, but they require some understanding so
  that the method is correctly matched to the
  problem and so that the method parameters are set
  correctly.
• For medical coding applications, the statistical
  sample selection methods used by HHS/OIG Office
  of Audit Services, as found in RAT-STATS, can be
  considered canonical.

9
Research Questions and Methods

• Specification and Control Limits
• Specification Limits relate to units of
  production (e.g. coded charts/reports) and
  measure whether the unit is acceptable or not.
• Control Limits relate to an overall production
  process (e.g. a coder, a group of coders, a CAC)
  and measure whether the process, as a whole, is
  performing acceptably (in control) or not (out of
  control).
• Specification Limits and Control Limits are not
  comparable.
• Specification Limits are ultimately defined by
  the organization in terms of what constitutes a
  defective unit and the maximum percentage of a
  batch may be defective.
• Control Limits are statistically defined and
  indicate whether a tested (audited) sample defect
  level is statistically acceptable given the
  specified defect level.

10
Research Questions and Methods

• Specification and Control Limits
• To put it another way, the defect (error) level
  as determined by an audit of a statistical sample
  is only an approximation of the true defect
  (error) level.
• The true defect level may be higher or lower than
  the audit defect level.
• The control limits
• indicates whether the sample defect level is
  within the statistically acceptable limits, and
• indicates if a larger sample is needed in order
  to get an accurate estimate of the true defect
  level.

11
Research Questions and Methods

• Specification and Control Limits
• Because medical coding is particularly prone to
  subjective differences, and both mental and
  mechanical errors (as compared to say a
  laboratory thermometer or scale), it is necessary
  to provide a calibration for coder variability,
  or more properly, the coefficient of variation
  (CV)
• The effect in the calculations is to modify the
  sample specification score to account for Type I
  vs. Type II errors at a rate calculated from the
  CV
• CV can be established based on an educated
  estimate of the auditors skill level, or can be
  determined more accurately using CV testing
  methods over a period of time.

12
Research Questions and Methods

• Overview of the Methodology
• Select the system parameters for the desired
  level of statistical certainty, the skill level
  of the auditor, and what is to be audited
• System calculates sample size and selects a
  random sample based on the given parameters
• The auditor reviews the sample records using the
  computer interface to score each record
• System calculates the audit score for each record
  and for the entire sample as adjusted for the CV
• System calculates the control limits and reports
  the process under audit as either in control or
  out of control
• If out of control, parameters should be
  readjusted so that a larger sample is scored to
  confirm if the process is truly out of control

13
Example RAT-STATS
RAT-STATS is a package of statistical software
tools designed to assist the user in selecting
random samples and evaluating audit results. It
is the primary statistical audit tool used by the
Office of Audit Services. (Source HHS Office of
Inspector General Web Site http//oig.hhs.gov/org
anization/OAS/ratstat.html)

• Key Features and Capabilities
• In use since the early 1970s, Windows version
  available since 2001
• Four primary functions sample size
  determination, random number generation,
  attribute appraisals and variable appraisal
• Sample size determination computes sample sizes
  of an overall population based on user-defined
  parameters
• Random number generation produces sequences of
  random numbers to assist an auditor in selecting
  records, based on a sample size
• Attribute and variable appraisal methods help an
  auditor interpret the results of an audit

14
Sample Size Determination

• User selects desired Confidence Levels
• Anticipated Rate of Occurrence, Universe Size and
  Desired Precision Range entered by user

• Sample Sizes computed for selected Confidence
  Levels and Parameters

15
Random Number Generation

• User has the option to enter seed number
• Quantity of numbers to be generated with an
  option for spares
• Sampling Frame sets the range of random number
  values
• Multiple output file formats available

16
Random Number Generation

• Text file example
• Seed number generated by the program
• 10 values plus 4 spares

17
Attribute Appraisal

• Three parameters entered by user Universe Size,
  Sample Size, Items of Interest

• Output shows the range of percentages at
  different Confidence Levels.

18
Example CoAudit
CoAudit is a computer assisted auditing
application. Users can ensure coding is in
compliance with regulations and spot errors,
omissions, fraud and abuse. For a given set of
parameters, CoAudit determines a representative
sample size, displays records for viewing and
scoring, and produces detailed audit reports.

• Key Features and Capabilities
• Audits can be performed using numerous filtering
  parameters including Coder, CPT, ICD-9,
  Physician, Referring Physician, Payer Code and
  Class
• Provides scoring of coders to track accuracy and
  improve efficiency
• Saves parameters in a template for new batch runs
• Controls access using three levels of permissions
• Provides access to data across multiple sites
• Creates detailed Audit Reports based on Coder,
  CPT, ICD-9, Code Pair, Modality, Referring
  Physician, Payer Code, and Payer Class

19
Sampling Parameters

• User self-selects Confidence Level.
• Error Margin, Coder and Auditor Error Rates are
  set in application Options dialog.

• Sample Size and Control Limits are computed
  automatically

20
Default Document Review Window Layout

• Panes containing all relevant information
  including
• Document transcription
• Demographics
• CAC output
• Coder edits and comments
• CPT and ICD-9 codes

21
Generate Summary and Detail Reports
Export reports to PDF, CSV, Excel or XML
22
Measuring Results

• This sample X-Bar chart illustrates in control
  and out of control coders and how the effects of
  interventions can be observed with Coders 4 5

Upper Control Limit
Lower Control Limit
23
Results

• Review of Objectives Computer Assisted Auditing
• Automate the audit process workflow
• Sampling method that computes sample sizes and
  randomly selects documents
• Scoring method provides results that can be used
  with statistical QA and production control
• Scoring method tracks to methods and results of
  purely human audits
• Audit results can be reported and interpreted by
  the user, forming the basis for effective
  decisions
• Audit results are meaningfully comparable between
  auditors and auditees and across time

24
Conclusion

• Computer Assisted Auditing
• Similar to how CAC improves the productivity,
  accuracy and consistency of the coding process,
  Computer Assisted Auditing can yield similar
  benefits for the auditing process
• Audits can be done more frequently, with bigger
  samples
• Valid sample size, random selection and
  established specification limits (scoring) yield
  reliable results
• Auditors perform more consistently and results
  are comparable over time
• Paper describes the details
• Parameters and formulas used for sample size
  determination with recommended parameter values
• Scoring method with formulas

25
Next Steps

• Computer Assisted Auditing applications are
  emerging
• The combination of CAC, remote coding and
  offshore coding motivates more efficient and
  scalable approaches
• With RAT-STATS, statistical foundation is defined
• Leverage the platforms developed for CAC
  Secure, web-based, multi-specialty, multiple
  types of documents
• Significant benefit when auditing can be
  integrated into an electronic workflow and
  medical data repository
• Reduce or eliminate costs associated with
  manually pulling medical and billing records
• Complete workflow could be paperless, with remote
  auditing as effective as remote coding

26
More Information

• Daniel Heinze, PhD, Chief Technology Officer,
  dheinze_at_alifemedical.com Technical Questions
• David Byrd, VP Sales and Marketing,
  dbyrd_at_alifemedical.com CoAudit Contact
• RAT-STATS Home page - http//oig.hhs.gov/organizat
  ion/OAS/ratstat.html
• NIST e-Handbook of Statistical Methods -
  http//www.itl.nist.gov/div898/handbook/