1 Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

Automated Referral Workflow System

3 Current Referral Process

4 Automated Referral Workflow System IRG Referral Methods Suggested by Prior Studies Referral Basis Percentage of Incoming Applications PI requests electronically mined from letter 47 Machine learning predictions when confidence is > 80% 39 Remainder referred by human experts14

5 Automated Referral Workflow System Deployed February 2007 Automated mining of PI requests Referral suggestions based on machine learning Decision support/workflow tool

6 Automated Referral Workflow System Cover Letter Mining -- Approach Fuzzy match to full study section names Exact match to study section acronyms No semantic analysis

7 Automated Referral Workflow System Automated Letter Classifications “High Confidence” “MESH Study Section” “Reduced Confidence” “MI”, “Medical Imaging”, “MI” “No SRG Requests” No acronyms or names found

8 Automated Referral Workflow System Cover Letter Mining Results First electronic new unsolicited R01s (October 2007 Council) 59% of applications included SRG request Only 47% two years ago

9 Automated Referral Workflow System Automated Classification of All Letters High Confidence55% Reduced Confidence26% No SRG Request18%

10 Automated Referral Workflow System How Accurate Are High Confidence Classifications? 92% of applications reviewed by IRG identified in letter by ARWS Consistent with data from existing human referral process

11 Automated Referral Workflow System Cover Letter Mining Conclusions Automated referral based on requests is feasible More “High Confidence” letters needed Better algorithms Structured cover letters

12 Proposed Structured Cover Letter Please assign this application to the following: Institutes/Centers National Cancer Institute - NCI National Institute for Dental and Craniofacial Research – NIDCR Scientific Review Groups Molecular Oncogenesis Study Section – MONC Cancer Etiology Study Section – CE Please do not assign this application to the following: Scientific Review Groups Cancer Genetics Study Section – CG Automated Referral Workflow System

13 Automated Referral Workflow System Machine Learning Predictions Applications without requests Requests are rare for some mechanisms

14 Automated Referral Workflow System IRG Assignment Prediction

15 Automated Referral Workflow System Exit Ramp

16 Automated Referral Workflow System Next Steps IMPAC II interface is critical Structured cover letters Improved machine learning More mechanisms Benefits  Reduced referral staff  Review meetings 2-3 weeks earlier (or later receipt dates)

17 Acknowledgements Support  Office of the Director  Extramural Affairs Working Group ARWS Project Team  CSR Staff (Dipak Bhattacharyya, Eileen Bradley, Suzanne Fisher, Richard McKay, Richard Panniers, Laura Roman, Kalman Salata, Sean Tate)  Discovery Logic (Kirk Barden, Marty Brown, Mike Pollard, Greg Young)  IC Staff (Arthur Castle)