Automated Referral Workflow System

1
Automated Referral Workflow System
2
Current Referral Process
Automated Referral Workflow System
3
IRG Referral Methods Suggested by Prior Studies
Automated Referral Workflow System
4
Deployed February 2007

• Automated mining of PI requests
• Referral suggestions based on machine learning
• Decision support/workflow tool

Automated Referral Workflow System
5
Cover Letter Mining -- Approach

• Fuzzy match to full study section names
• Exact match to study section acronyms
• No semantic analysis

Automated Referral Workflow System
6
Automated Letter Classifications

• High Confidence
• MESH Study Section
• Reduced Confidence
• MI, Medical Imaging, MI
• No SRG Requests
• No acronyms or names found

Automated Referral Workflow System
7
Cover Letter Mining Results

• First electronic new unsolicited R01s (October
  2007 Council)
• 59 of applications included SRG request
• Only 47 two years ago

Automated Referral Workflow System
8
Automated Classification of All Letters
Automated Referral Workflow System
9
How Accurate Are High Confidence Classifications?

• 92 of applications reviewed by IRG identified in
  letter by ARWS
• Consistent with data from existing human referral
  process

Automated Referral Workflow System
10
Cover Letter Mining Conclusions

• Automated referral based on requests is feasible
• More High Confidence letters needed
• Better algorithms
• Structured cover letters

Automated Referral Workflow System
11
Proposed Structured Cover Letter
Please assign this application to the
following Institutes/Centers National Cancer
Institute - NCI National Institute for Dental and
Craniofacial Research NIDCR Scientific Review
Groups Molecular Oncogenesis Study Section
MONC Cancer Etiology Study Section CE Please
do not assign this application to the
following Scientific Review Groups Cancer
Genetics Study Section CG
Automated Referral Workflow System
12
Machine Learning Predictions

• Applications without requests
• Requests are rare for some mechanisms

Automated Referral Workflow System
13
IRG Assignment Prediction
Automated Referral Workflow System
14
Exit Ramp
Automated Referral Workflow System
15
Next Steps

• IMPAC II interface is critical
• Structured cover letters
• Improved machine learning
• More mechanisms
• Benefits
• Reduced referral staff
• Review meetings 2-3 weeks earlier (or later
  receipt dates)

Automated Referral Workflow System
16
Acknowledgements

• Support
• Office of the Director
• Extramural Affairs Working Group
• ARWS Project Team
• CSR Staff (Dipak Bhattacharyya, Eileen Bradley,
  Suzanne Fisher, Richard McKay, Richard Panniers,
  Laura Roman, Kalman Salata, Sean Tate)
• Discovery Logic (Kirk Barden, Marty Brown, Mike
  Pollard, Greg Young)
• IC Staff (Arthur Castle)