1. Archived File The file below has been archived for historical reference purposes only. The content and links are no longer maintained and may be outdated. See the OER Public Archive Home Page for more details about archived files.archivedOER Public Archive Home Page

2. Peer Review Advisory Committee 2/1/2010 Pete Morton and Paul Sheehy Electronic Research Administration and the Receipt and Referral of Grant Applications

3. 3 Context and Drivers Continuous evolution of Processes and Policies Interdependencies of Business Processes IT Systems Development and Business Processes must be closely linked Do not want to simply re-build the same thing without seeing if business processes would benefit from a change IMPAC II is nearing 20 years old Original System Structure Needs Refreshment Lower Operations and Maintenance Costs Greater Flexibility Evolving needs of NIH’s extramural activities: Evolving needs of NIH’s extramural activities: Fundamental shift in the way IT is developed: New technologies are now available

4. 4 Three distinct activities: 1.Develop New Hardware Infrastructure »Servers and SANs »Status: Complete 2.Develop New Software Infrastructure »Underlying IT applications and data tables »Status: Ongoing 3.Reengineer Applications “Evergreening” »Incorporate new infrastructure capacities »Refresh IT to incorporate new policies and business processes »Status: Starting IT Refreshment Strategy Three distinct activites

5. 5 Evergreening a Business Function General Strategy –Identify component business processes –Analyze current processes Opportunity to identify and assess potential new support functions –Re-engineer IT systems Approach –Create Business Process Models “Current / As Is” state vs. “Future / To Be” state –Validate using focus groups of business subject matter experts –Model drives/directs IT re-engineering Business Process Re-engineering (BPR) –Synthesis of Business Process Modeling and IT Reengineering

6. 6 Business Process Reengineering Benefits –Modeling ensures an excellent understanding of actions and rules between business processes and supporting IT systems Establishes credibility –Modeling provides a “refreshed examination of business processes” before reengineering the IT support for those processes Better visualization of inefficiencies and pain points Holistic view reduces surprises If the existing model is accurate and adequate, can proceed directly to IT re-engineering –Synthesis allows for faster, cheaper software development Better defined target

7. Introduction: BPR of CSR Receipt & Referral As part of a larger effort to examine eRA reengineering, a pilot project to BPR CSR Receipt & Referral was initiated in June 2008 Mandate –Examine and document current referral business process re: IC and IRG assignment –Measure process performance: efficiency and quality of assignment –Understand referral process’ benefit to NIH’s scientific mission –Understand causes of sub-optimal process performance –Develop heuristic future state business process designed to improve performance and quality of assignment 7

8. Evergreening eRA 8 Business Process Reengineering the Receipt and Referral Module of eRA: a Pilot

9. Introduction: Division of Receipt and Referral (DRR) DRR fulfills four business purposes: –Application assignment based on the scientific content of the application Technical merit review group Institute or Center –Application quality control –Policy enforcement –Facilitate and manage changes to assignment All are critical to NIH’s scientific mission 9

10. Introduction: Business Challenges Increase in volume of applications Requirements for consistency, flexibility, transparency, equity, accountability Aggressive implementation of electronic submission Sometimes problematic IMPAC II migration to web Increasing complexity of assignment decisions –Clinical, translational and multi-disciplinary research –Complexity is both review (subject matter, techniques, etc) and referral (relevance to multiple IRGs and/or IC missions) e.g. A developmental disorder with cardiac & behavioral manifestations 10

11. Approach Validate current state business process models developed by DRR and Office of the Chief IT Architect Establish metrics for the process Identify areas of challenge for focus Perform root cause analysis on these areas Review potential solutions to issues including technology and process changes Develop a new business process model 11

12. Methodology: Components Two methodologies needed: –Modeling: To document current processes and heuristic future process Business Genetics XBML –Metrics: To measure efficiency and effectiveness of current process and provide input to future state processes Lean Six Sigma 12

13. Methodology: Activities Phase 1: Update of existing model –2006 OCITA current state business process model used as a foundation –Project team, DRR staff, IRG chiefs reviewed, corrected/updated as necessary –Focus on IRG assignment, not IC Phase 2: Obtaining process metrics –DRR staff and IRG chiefs provided estimates of the normal time taken for each major business process step –“Normal” defined as time required for 80% of applications 13

14. Evergreening eRA 14 Findings

15. Results: Process Efficiency 80% of Applications processed within one day For this 80% of applications, metrics indicate overall efficiency is world class –Work/wait time distribution equals benchmarks regardless of industry –Wait time generated in part by policy Conclusions –Efforts focused on reducing “wait-time” of grant applications will result in minimal improvement to referral process performance –Process changes should focus on the 20% of applications that take longer to process –However, there is still significant benefit to automating the less complex assignments 15

16. Results: Process Efficiency (2) Remaining applications much more variable in time to refer –20% of total applications 2/3 due to problems with application itself –13% of total applications 1/3 are more complex to assign –7 % of total Scientific overlap between IRGs Complexity of application 16

17. Distribution of Issues of Problematic 13% of Applications 17

18. Results: Process Efficiency of Complex Assignments Factors contributing to complex assignment: –Overlaps/gaps in IC mission –Variation in sponsorship and use of activities –Complexities of locus of review –All require scientific judgment Quality of assignment –Essential to validity of NIH Peer Review Process –BP and IT changes must retain and enhance quality of assignment going forward (i.e. not bouncing back from SRG) Assignment delays impact review groups –May delay meetings –May require a new special emphasis panel 18

19. System Changes: Issues with the eRA System CSR staff discussions yielded the following issues with eRA: –Performance and reliability of Referral and Review modules –Fragility – Changes tend to introduce new problems –Security model does not allow others to view all referral data –Notes Field Only place to track referral process e.g. Referral officer and IRG Chief and SRO comments Lack of carryover between Receipt and Referral (RR) and Review (REV) modules –DRR uses RR, IRG Chiefs and SROs use REV, notes are not visible to each other Lack of date/time/author stamping Lack of carryover from ExitRamp –Release of applications and visibility to reviewers in IAR –Web-related issues Inability to open more than one application window at a time 19

20. System Changes: DRR Priority List DRR staff-identified priorities: 1.Support for fully electronic Awaiting Receipt of Application (ARA) notices 2.IRG Chief push to final assignment 3.Changes, fixes and improvements in the ACR system 4.Implement single 2 day extension of the 3 day assignment window 5.Support for automated comparison of application content –Identification of duplicate applications and revisions –COTS and custom KM products to compare current and previous submissions 20

21. Evergreening eRA Recommendations 21

22. Future Model: Changes Suggested changes to existing processes: –Application Validation Automate and move validations currently performed by CSR after application receipt to eSubmission process Expand types of checks and validations applied to application prior to submission beyond the set limited by 424RR form set Use these checks to either generate warnings for applicant or flag applications for review Provide a mechanism for applications to validate their application prior to submission – “Pre-submission portal” –Incorporate the ability to create a special, customized workflow for specific applications that require handling outside the normal flow of operations –Modify the 3-day window for application release 22

23. System Changes: Automation Opportunities Other changes required in order to implement the “To Be” process –Provide system to support validation of an application against all Grants.gov and NIH business rules prior to submission to NIH “Pre- submission portal” –Support for a fully structured electronic “cover letter” Could be implemented in Commons without need for new OMB form –Enhance ARWS to production ready status Fully integrate with eRA, especially notes Make ARWS decisions trainable –Implementation of a flexible, user-controlled workflow system –Expand the eSubmission business rules –Represent Funding Opportunity Announcement as electronic business rules –Represent of referral guidelines as electronic business rules 23

24. Suggested Priorities Immediate –Fixes for eRA issues, especially notes –Support IRG Chief push to final assignment and 3 day window extension (currently planned) –Implement Electronic Cover Letter in Commons Consider providing an optional electronic form to be submitted as an interim measure –Fixes for ACR system 24

25. Suggested Priorities – cont’d Medium Term –Implement Pre-Submission Portal Expand and harden eSubmission business rules –Implement ARWS at production quality –Implement Electronic ARAs –Pilot application comparison tools Long Term –Implement Flexible Workflow –Implement electronic business rules for: Referral Guidelines (IC assignment) Funding Opportunity Announcement (application validation) 25

26. Evergreening eRA 26 Software Reengineering Phase

27. Next Steps (eRA) Goal: Brief review/update of desired business changes resulting from BPR then rebuild IT support for R&R incorporating desired changes Form core focus group –Includes staff from Division of Receipt and Referral as well as CSR IRG chiefs and IC representatives –First meeting scheduled for mid-February 27

28. Next Steps (eRA) -- Continued Working with core focus group, develop detailed requirements for new R&R module –Design to use new eRA software infrastructure Initial detailed requirements documented –mid- summer Develop prototype of new module – early fall –Update requirements as appropriate 28

29. Next Steps (eRA) -- Continued Begin iteration 1 development in fall, 2010 Work with focus group to test initial module and establish priorities for iteration 2 development – spring, 2011 Begin development of iteration 2 – summer, 2011 Work with focus group to test iteration 2 module and plan subsequent development if needed – late summer, 2011 Finalize and deploy new module into production – September 30, 2011 29

30. Evergreening eRA 30 Where do we go from here?

31. The next step: Review Module As BPM is complete for the Referral module and software reengineering is starting, the next opportunity is the review module. Much bigger challenge peer review policy and procedures are interpreted and executed in a variety of ways Operational Divisions (OpDivs) under DHHS authority utilize NIH resources but don’t necessarily follow NIH policy Agency Partners utilize NIH resources but don’t necessarily follow NIH policy Initial recruiting is underway Governance Subject matter experts 31

32. Review by NIH Governance Trans-NIH Senior Staff Extramural Activities Working Group (EAWG) –Working group of NIH Steering Committee with primary responsibility for extramural research and research training Information Technology Working Group (ITWG) –Working group of NIH Steering Committee with primary responsibility for NIH's enterprise-level IT programs and investments. Functional Areas Extramural Program Management Committee (EPMC) – Senior leadership of IC extramural research and research training Review Policy Committee (RPC) – Chiefs of Review Offices Review Users Group (RUG) –SROs with interest in electronic research administration (NIH and IC) 32