Dr. Julia H. Bryan College of Public Affairs Doctor in Public Administration University of Baltimore (2013 Graduate) October 19, 2013

» There has been an increase in practice of performance measurement information but use of this information… » There has been minimal research on performance measurement use at the federal level » It remains unclear whether performance measurement information is used for decision-making and management practices » This represents a gap in the literature on ways to improve and enhance performance measurement use in federal programs 2

» Led to 3 overarching research questions: (1)What factors influence the use of performance measurement information and how? (2)Under what circumstances is performance measurement information being used? (3)Who is using this information? 3

» Has potential to positively influence decision and management practices » Helps to inform policy, research and practice -Better equip federal managers and staff to use performance measurement information in a way that best meets their program needs 4

» Builds upon a performance measurement utilization model » Several uses to consider in the decision making process +Instrumental +Non-instrumental » Factors influencing the use of performance measurement information +Rational/Technocratic Factors +Political/Culture Factors +Organizational Complexity 5

H1Use of performance measurement information consists of both instrumental and non-instrumental use and each type of use is influenced by different factors. H2Organizations with increased complexity are more likely to use performance measurement information. H3Rational/technocratic factors are positively related with performance measurement use. H4Political/cultural factors are positively related with performance measurement use. H4-aOrganizations that involve stakeholders in the performance measurement process are more likely to use performance measurement information. H4-bOrganizations with high innovative and reward practices are more likely to use performance measurement information. 6

» Comparative case study » Mixed methods approach » Why two program offices? +Diversity in size, culture and structure +Both have performance measurement systems 7

» 28-item cross-sectional survey » Piloted in January 2012 » Population: 326 managers and staff » Administered both web-based and paper-based between March 1, May 1, Thank you for your participation in this survey. The purpose of this survey is to examine how and why federal programs use performance measurement information at the program level (not individual grantee or personnel performance level). This study is important to your program office and the field of public administration because it will provide a better understanding of factors that influence the use of performance measurement information. Your knowledge and expertise can provide valuable insight on this topic and your participation is greatly appreciated.

» Dependent Variables ˃Instrumental use: accountability, improvement ˃Non-instrumental use: understanding, mobilization » Independent Variables ˃Organizational Complexity: levels of involvement, formulation and specialization ˃Rational/Technocratic: budget, staffing, training and information ˃Stakeholder Involvement: developing performance measurement, supporting change and hold office accountable ˃Organizational Culture: innovative practices, rewarding managers/staff » Control Variables ˃program office, position, length of time in Agency and program office 9

Descriptive N= 114 Managers= 29% Staff= 70% Length of time in Office= 85.5% Overall program office distinctions: (1) Non-instrumental use for understanding (2) Rational/technocratic factors (3) Organizational culture Factor Dependent variables 3 Constructs -Performance Measurement Use -Non-instrumental Use -Instrumental Use Independent variables 4 Constructs -Organizational Complexity -Organizational Culture -Rational/Technocratic -Stakeholder Involvement 10

Unstandardized B Coefficients Independent VariablesPerformance Measurement UseNon-instrumental UseInstrumental Use Organizational Complexity.259* ** Organizational Culture.207*.227*.055 Rational/Technocratic.587***.412**.419** Stakeholder Involvement Program Office Management Time in Program Office Time in Agency R-squared=.476 Significance=.000 N= 114 R-squared=.245 Significance=.0001 N= 114 R-squared=.341 Significance=.000 N= p<.05*, p<.01**, p<.001***

Interviews N= 20 Length: minutes Conducted 9/2012 – 1/2013 General Themes: program improvement; role of leadership; lack of resources; role of stakeholders; structure of program office Program Specific Themes: - Program Office 1: stakeholders role in technical assistance; timely/accessible data; policies/procedures - Program Office 2: need for understanding; challenges with developing performance measures; stakeholders role in performance measurement development Document Review Data collection and review: 3/2012 to 2/2013 Types of documents review included annual performance reports, Congressional budget justifications, funding announcements, and annual program reports Themes - Use and/or intended use of performance measurement information for accountability, program improvement and understanding 12

Interview Quotes: “There are standard operational procedures for most of the work performed by POs [project officers]. While many of these documents are “dense”, they provide a great foundation in order to execute tasks from start to finish” “Making performance information more readily accessible….” “It’s a relatively new system and … still identifying ways to help grantees actually utilize the performance data information versus just reporting it because it is a requirement” 13

» Instrumental use of performance measurement information was indicated by both program offices » Rational/technocratic factors play a key role in the performance measurement use » Instrumental and non-instrumental use were influenced by both similar and different factors Role of stakeholders in developing performance measures as well as providing technical assistance to grantee organizations Structure of the program office appears to effect the use of performance measurement information 14

» Communicating the value and importance of performance measurement information » Continuous assessment of performance measurement resources » Cultivating and rewarding innovative practices for performance measurement use » Broadening the interpretation of performance measurement use to include both instrumental and non- instrumental use » Expanding research to be inclusive of both managers and staff and to assess use at the federal program level 15

» Low response rate » Self-reported data » Response bias » Organizational types 16

» Contributed to a limited body of knowledge on the use of performance measurement information at the federal level » Added a unique perspective for examining the use of performance measurement information » Provided a greater understanding that may help determine ways to enhance the linkage between performance measurement use and management practices in federal agencies 17

18 Thank you! Julia Bryan, DPA, MPH, CHES