1. Overview of OPASI’s Division of Evaluation and Systemic Assessments (DESA) Deborah Guadalupe Duran, Ph.D. Chief, Systemic Assessments Branch

2. DESA: Division of Evaluation and Systemic Assessments Evaluation Feedback System Assessments (Sci of Sci) Evaluability Organization Performance DRDA: Division of Resource Development and Analysis Knowledge Management Portfolio Analysis Public Health Need Centralized tools DSC: Division of Strategic Coordination Strategic Coordination-High Risk-High Impact Trans-NIH Initiatives gaps-potentials data-evidence strengths- improvements DRDA DSC DESA OPASI Overview

3. Two branches Systemic Assessments Branch (SAB) Evaluation Branch (EB) DESA Structure (DESA)

4.  Coordinate NIH wide evaluations and assessments  Provide technical assistance for required performance reporting  Facilitate the development of adaptive measures for science  Provide findings, integrated with portfolio analyses that foster scientific planning SAB EB DESA Mission

5. Evaluation Branch  Evaluate NIH-wide programs, key initiatives, and strategic objectives (formative and summative)  Track outcomes/recommendations  Maximize the effective use of the NIH Evaluation Set-Aside Program SAB EB DESA - EB Mission

6. Systemic Assessments Branch  Coordinate program performance activities across NIH  Comply and respond to federally-mandated performance reporting mechanisms GPRA PART PAR Scorecard  Foster the development of measures for Adaptive science assessments Organization assessments  Facilitate science of science activities SAB EB DESA – SAB Mission

7. Strategic Plans System Level - HHS/IC/Office/Program Science of Science Management Pre-evaluation Assessments Visual Analytic Tools e-Monitoring Systems Strategic Plans Project Level Independent Evaluations Evaluation Set-Aside Funds GPRA PART PAR Audit Biennial Report Scorecard Coordinating-Planning-Managing-Tracking-Reporting System Performance & Assessments Independent External Evaluations Related Processes Resources Strategic Planning & Evaluability SAB EB DESA Activities

8. Evaluation Branch  Manage Evaluation Set-Aside Program which provides dedicated funding for ICs and OD offices  Provide hands-on technical support and online guidance  Consult with ICs and OD offices on evaluation planning  Coordinate two trans-NIH committees to evaluate proposals  Report evaluation results SAB EB DESA – EB Role

9. Systemic Assessments Branch  Monitor NIH performance (not evaluation)  Report program performance information in lay language  Ensure Budget/Performance Integration  Participate in OMB / Secretary / HHS / Federal workgroups  Liaison to OMB, HHS, ICs performance issues  Develop System Assessments  Conduct internal assessments (evaluability)  Coordinate Science of Science activities  Strategic Planning SAB EB DESA – SAB Role

10. Systemic Assessments Branch: Evaluability Assessments (internal) Science of Science (internal) Monitoring & Tracking Performance Organization Performance Reporting Evaluation Branch: Funding for independent evaluations (external) Systemic Assessments Branch Evaluation Branch SAB/EB--related (ie IC Planning & Evaluation Community support) but unique functions DESA Functions

11. DESA (SAB and EB) offer comprehensive evaluations/assessments – Internal and External SAB–internal & organization assessments Evaluability Assessments Monitoring & Tracking Performance System Assessments (Science of Science) Adaptive evaluations for science EB–external Independent Evaluations of projects/programs/key initiatives Internal (SAB) External (EB) Evaluations/Assessments

12. Evaluation External- best conducted by source outside of the project/program Types  Formative and Summative Evaluations (Mid-term and End-of-Project Reviews)  Needs assessments  Process Standard Methodologies

13. Standard Evaluation Processes  Program/project as the unit of analysis  Upfront utilization focus / cause and effect  Linear logic models  Focus on achieving pre-planned goals  Findings assessed in relation to plans  Time specific  Methodologies Static designs Preference for quantitative data Randomized Control Trials-Gold Standard

14. Conditions that Challenge Evaluation  High innovation  Developmental  High uncertainty  Dynamic (complex)  Emergent SCIENCE and SCIENCE SYSTEMS

16. Science Programs New field Developing theories & tools Developing metrics & analytic methodologies NIH among the leaders (national & international) Program & Management Organization project technologies communicationadministration program human capital management A pictorial representation System Assessments

17. System Premises Systems neutrality : A system is functioning as observed for some reasons, fulfilling some functions. In whose interests is a system functioning? Who benefits? What does the homeostasis offer?

18. In a well-functioning system…  no sub-system operates at maximum  dynamic/systems change Relationships are interdependent and non-linear, rather than simple and linear (cause-effect) Transitions - worse before better  complex - science Highly emergent (difficult to plan and predict) Healthy System

19. Internal preparation for external evaluations Independent external evaluations Organizational and system assessments SAB/EB work together to coordinate assessments of program/project performance and NIH overall performance System Performance & Assessments Independent External Evaluations Related Processes Resources Strategic Planning & Evaluability SAB EB DESA Synergy

20. Providing Technical Assistance Teaching Strategic Planning course Developing and implementing evaluability survey Providing results feedback Preparing for GPRA and PART Creating visual analytic tools Offering resources Providing technical assistance Guiding application process Reviewing proposals Monitoring awards Promoting innovative methodologies Monitoring and Reporting - PART - GPRA - Audits - PAR/MD&A DESA--implemented through a number of steps/tasks System Performance & Assessments Independent External Evaluations Related Processes Resources Strategic Planning & Evaluability SAB EB DESA Implementation

21.  Be prepared for Evaluations Performance reporting Strategic planning  Utilize analysis tools for decision makers  Conduct Needs Assessments Feasibility Studies Process evaluations Outcome evaluations Evidence related activities  Score well in PART GPRA Audits PAR/MD&A SAB and EB supports ICs to System Performance & Assessments Independent External Evaluations Related Processes Resources Strategic Planning & Evaluability SAB EB DESA Outcomes

22. SAB EB Demonstrates appropriate federal stewardship Provides a systematic process for conducting evaluations/ assessments that produce evidence-based results, which can be used to  improve or adapt programs and/or management of science  comply with performance assessment requirements  plan Strategic Planning & Evaluability Independent External Evaluations Related Processes Resources System Performance & Assessments DESA Summation