1. HRPP METRICS Cynthia Monahan, MBA, CIP IRB Director Boston University Charles River Campus IRB

2. Table of Contents Overview of Metrics IRB Metrics QI Program Metrics AAHRPP Metrics for Quality Improvement

3. What are Metrics? Standards of measurement by which efficiency, performance, progress, or quality of a plan, process, or product can be assessed http://www.businessdictionary.com/definition/m etrics.html#ixzz2a9m0cHou

4. Why collect metrics? Improve IRB processes Efficiency Quality Provide researchers with information that will assist them in planning research activities Determine adequacy of resources Identify gaps/problems Assess new policies/forms Target education Prevention

5. Define Metrics Define parameters and measures What data will be collected Calendar days vs. business days Analysis (e.g. Mean vs. Median) Starting and stopping points for review times Protocol received vs. protocol reviewed at meeting

6. IRB Metrics Turn-around time IRB review category (expedited, exempt, full board) Submission type (new, CR, amendment, etc.) Volume submitted Volume approved Submissions by department Submissions by IRB staff Workload Time in IRB Office vs. Time in PI office

7. Review Time Metrics

8. Volume Metrics

9. Metrics and the QI Program Audits IRB Research site For cause audit vs. not-for-cause audits Compare findings/observations Random audits Define criteria

10. QI Program Metrics IRB Minutes Categorization of reviews Waiver/Alteration of consent Research Sites Compliance with protocol Compliance with institutional policies Prevention

11. Association for the Accreditation of Human Research Protection Programs (AAHRPP) An independent, non-profit accrediting body AAHRPP uses a voluntary, peer-driven, educational model to ensure that Human Research Protection Programs (HRPPs) meet rigorous standards for quality and protection. To earn accreditation, organizations must provide tangible evidence—through policies, procedures, and practices—of their commitment to scientifically and ethically sound research and to continuous improvement

12. Goals, Principles, and Standards AAHRPP has adopted nine overarching principles for protecting human research participants, which serve as the foundation for the AAHRPP Accreditation Standards for Human Research Protection Programs. The AAHRPP Accreditation Standards themselves describe what Organizations can do to consistently act on these principles by applying them to the diverse organizational and cultural contexts in which research is conducted and reviewed

13. Accreditation Standards STANDARD I-5: The Organization measures and improves, when necessary, compliance with organizational policies and procedures and applicable laws, regulations, codes, and guidance. The Organization also measures and improves, when necessary, the quality, effectiveness, and efficiency of the Human Research Protection Program.

14. Elements Element I.5.A. The Organization conducts audits or surveys or uses other methods to assess compliance with organizational policies and procedures and applicable laws, regulations, codes, and guidance. The Organization makes improvements to increase compliance, when necessary. Element I.5.B. The Organization conducts audits or surveys or uses other methods to assess the quality, efficiency, and effectiveness of the Human Research Protection Program. The Organization identifies strengths and weaknesses of the Human Research Protection Program and makes improvements, when necessary, to increase the quality, efficiency, and effectiveness of the program.

15. Element I.5.A An Organization’s quality improvement program should include measures of compliance with organizational policies and procedures and applicable laws, regulations, codes, and guidance Audits and reviews to determine compliance Audits should be systematic and consistent The Organization’s quality improvement program should include an evaluation of the HRPP to determine whether it is effective in achieving compliance Review of minutes Review of categories (exempt, expedited, waivers, etc.)

16. Element 1.5 A (cont) The organization should collect objective data through audits, surveys, or other methods and use the data to make improvements and monitor compliance on an ongoing bases Share data with stakeholders Revise programs/processes based on objective data The number and breadth of audits should be determined by the organization and should be sufficient enough to provide data that inform the quality improvement program Identify measures and volume in advance

17. Element 1.5 A (cont) Essential Requirements for Written Materials: The Organization has a quality improvement plan that periodically assess compliance with the HRPP: The plan states the goal of the quality improvement plan The plan defines at least on objective to achieve or maintain compliance The plan defines at least one measure of compliance The plan describes the methods to assess compliance and make improvements

18. Element I.5.B An Organization’s quality improvement program should include measures of quality, efficiency, and effectiveness to evaluate the performance of the HRPP Performance metrics The Organization should use the results from the quality improvement program to design and implement improvements How are metrics used

19. Element I.5.B (cont) The Organization should collect objective data through audits, surveys, or other methods and use the data to make improvements and monitor quality, efficiency, and effectiveness on an ongoing basis Share data with stakeholders Revise programs/processes based on objective data

20. Element I.5.B (cont) Essential Requirements for Written Materials: Quality Improvement Plan States the goals of the quality improvement plan with respect to achieving targeted levels of quality, efficiency, and effectiveness of the HRPP Defines at least one objective of quality, efficiency, or effectiveness Defines at least one measure of quality, efficiency, or effectiveness Describes the methods to assess quality, efficiency, and effectiveness and makes improvements

21. How do you meet the Elements? Compliance Plan Audits, surveys, or data collection tools Quality Improvement Plan Evaluation Reports

22. Metrics for Quality Improvement Define responsibilities and roles Collect what you measure and measure what you collect Identify measures and goals in advance Continuous process that should be re-assessed and revised as appropriate (Annual basis) Periodic reviews External audits/inspections (FDA, etc.)

23. Metrics for Quality Improvement Track feedback Analyze results (who, what, when) Review outliers Share results Training and education based on results (individual vs. trends) Process for implementing changes Benchmark—identify peer institutions

24. Resources AAHRPP: http://www.aahrpp.org/http://www.aahrpp.org/ University of Washington: http://www.washington.edu/research/hsd/topi cs/HSD+metrics http://www.washington.edu/research/hsd/topi cs/HSD+metrics