Metrics And Measurements Justification, Definition, Application

1
Metrics And Measurements Justification,
Definition, Application

• Daniel E Ford, MD, MPH
• Vice Dean for Clinical Investigation
• Johns Hopkins University

2
Why do we need research study initiation
performance measures?

• Perception of sponsors and research faculty we
  are inefficient
• No other accreditation organization has taken on
  this role
• Association of Human Resource Protection Programs
  (AHRPP)
• Office of Human Research Protections (OHRP/HHS)
• Federal Drug Administration (FDA)
• Opportunity for the doers to be proactive and
  lead the way

3
Why do we need research study initiation
performance measures?

• Measurement is the first step to understanding
  the process
• Same metrics across multiple academic sites
  allows comparisons and identification of best
  processes (ask for more resources if not
  performing as expected)
• Measurement of performance metrics tends to
  reduce variability

4
Why do we need research study initiation
performance measures?

• Posting metrics for sponsors and public
  demonstrates responsibility and accountability
• Once we demonstrate accountability, can ask for
  same from sponsors and research teams

5
Performance Metrics Should Be

• Specific
• Measurable
• Attainable
• Realistic
• Timely

6
Challenges to measuring performance

• Selecting the right performance measures
• Are study protocols in the performance measure
  representative or informative?
• Are the performance measures easy to measure?
• Can the performance measures be measured with
  reliability?
• Does the performance measure vary too much by
  type of study to be informative? (pediatric vs.
  adult, first site versus 50th site)
• Do the performance measures provide hints about
  the process or what the path to improvement might
  be?

7
Challenges to measuring performance

• Is speed the only measure of performance? What
  about quality of IRB review? What about quality
  of a contract that is clear and covers multiple
  potential problems?
• Are there ways to game the performance measures?

8
Examples?

• HEDIS1 quality of care measures
• Pay for performance measures
• CMS2 quality of care measures

• Healthcare Effectiveness Data and Information Set
  (HEDIS) is a tool used by more than 90 percent of
  America's health plans to measure performance.
• Centers for Medicare Medicaid Services (CMS)

9
First Performance Metric

• Duration in days from receipt of first ready
  study protocol/site agreement received to time
  both the clinical investigational site and
  sponsor have executed the agreement

10
Performance Metric Definitions

• What studies are included?
• Multicenter clinical trials
• Easier to assign date when site was able to start
  study initiation/approval process
• Include all sponsors (NIH and commercial)
• Dates
• 2007 chosen as most study protocols will have had
  opportunity to be activated

11
Performance Metric Definitions

• When is start date?
• Date agreement sent from sponsor
• Date agreement is received
• Research teams may not have recorded this date

12
Performance Metric Definitions

• Time to executed site agreement
• Is it date when one party signed agreement or
  when all parties signed agreement?
• Possible to game this time by signing
  agreements quickly and then need multiple
  amendments to really have study ready to enroll
  participants

13
Performance Metric Definitions

• Days to execution of agreements
• How to count protocols that never were completed?
• What about outliers?
• Use median or 90th percentile?
• 25, 50 and 75 percentile?

14
Days to Contract Execution by Percentiles
15
IRB Performance Metric

• Duration in days from PI receipt of protocol to
  site IRB approval
• Measure protocol receipt to initial (department
  or IRB) submission
• Measure protocol receipt to IRB submission
  complete
• Measure protocol receipt to first IRB review

16
IRB Performance Metric
17
IRB Outcome Measures
18
Lessons Learned

• Separate offices for grants, contracts and the
  IRB, and separate databases, make it difficult to
  track performance measures
• Commercially and federally funded studies should
  be tracked separately because the data points are
  different (e.g., the initial data point for
  federally funded studies is not the date the site
  agreement was received)
• Research administration databases usually track
  by funding source difficult to track studies
  with multiple funding sources
• The PI on the grant may not be the same as the PI
  on the IRB submission, making it hard to locate
  the study
• The IRB and research administration do not have a
  common tracking number for each study

19
Lessons Learned

• Date that the PI received the protocol is not
  being tracked in existing databases PIs had to
  be contacted to get this information
• 58 PIs contacted - 42 provided information
• 7 sponsors contacted - 5 provided information
• Although the PIs and sponsors were helpful, this
  was a time consuming process and the information
  will need to be captured systematically in a
  database to be used as a performance measure