Bill Stone Research Administrator Missouri Department of Transportation Presented at 2014 AASHTO Research Advisory Committee Meeting, Madison WI, July 24, 2014

Identified very broadly defined performance measures used by state DOTs at the time Measures taken from public sources such as accountability reports, state DOT websites, and other web-based resources Out of 40 states, 10 had published measures and 30 had no public information Survey lists metric, source and URL

The DOT State Stats report is a synthesis of facts, figures, statistics and metrics pulled from accountability reports, online performance measurement dashboards and fact books. 2009: Need existed to easily find and access local statistics Editions available:

Annual figures compiled in easy-to- use synthesis report online. Excel dataset also available online. Created and launched by previous MoDOT Librarian; assisted by members of the Midwest Transportation Knowledge Network (MTKN) and hosted on their website edition included metrics from 42 states. Publication is used by the transportation community at large.

Searching on “research” retrieves only 4 results out of 1,580 measures Synthesis report  no index Dataset  searchable but not categorized No longer current  not updated since 2012

Obtain input from research administrators actual or anticipated use of research performance measures to document the progress and success of their research programs Measures with performance targets or metrics that track activity Measures that were program-specific not project-specific Measures from all research program areas (pure research projects, library and/or technology transfer program)

Have future plans (5) High interest (1) Have potential measure in mind (1) Discussed it (1) Track but not as measure (1) No comment (1)

Track activity 56% Have performance targets 28%

20% of states collect or report statistics for more than just one time period

CURRENT USE Customer satisfaction (AZ) Monitor progress toward strategic goals (DC); adjust performance (TX) Make a difference in organizational business processes (MO) Determine program effectiveness (IA) Identify strengths and weaknesses (IN); what’s working or not working (MD) Tied to individual performance evaluations (LA);

CURRENT USE (CONT’D) Assess researcher performance and the usefulness of completed research (NC) Select new projects (TX) Ensure that projects move forward according to set budget and schedule (UT, WY) Develop and monitor SPR Part 2 program (WI) FUTURE USE Will use to direct funding (FL) Will use to gauge and improve performance (IL)

Applied 31 unique categories to 103 submitted individual measures Measures by Status Current: 69 measures Planned: 21 measures Considering: 10 measures Past: 2 measures In progress: 1 measures Some measures have multiple categories applied

TOP CURRENT: Completion (14), Implementation (9), Library Utilization/Processing (8), Training (7), Satisfaction (6) TOP IN PROGRESS/PLANNED/POTENTIAL: Implementation (6), Completion (3), Cost Savings (3), Timeliness (3), Within Budget (3)

Related to number No. of completed research projects Related to on-time or on-budget % of research projects meeting original completion date No. of projects completed in the FY on schedule % of projects completed according to the initial budget % of projects completed on time/on budget % of studies completed within the approved schedule of the Work Program

Related to satisfaction % of completed studies deemed satisfactory by the project sponsor Customer satisfaction survey for completed Research Related to implementation In past 5 years, 75% of completed research projects provide recommendations for implementation of results endorsed by the Project Review Committee

Related to number No. of research results and best practices implemented No. of NCHRP and other external research program results implemented % of projects implemented Fully, partially, later, cannot/or should not be implemented, within 2 years of final research report (using 5 years of data) % of projects actively utilized in the field

Related to number cont’d % of projects that have resulted in a spec, policy, procedure, manual, requirement or material change In past 5 years, 75% of completed research projects provide recommendations for implementation of results endorsed by the Project Review Committee Multi-year tracking of implementation work Related to funding Amount of funding for implementation activities

CA and MN: Reviewed DC & LA: Used as basis for developing PMs IL: Used to develop a structure for functional PMs IN: Exploring how to move measures to RPM UT: Use to upload measures ID, LA, MI, MO, PA and UT: Use for HVR submittals

PROS “RPM Web was extremely useful in helping us grasp the different types of performance measures and developing a structure for functional performance measures.” “Need to re-visit the site now that our program is more developed to see how it might help.” CONS “Adapting to RPM Web would require a lot of work.” “Have mixed feelings about the usefulness.” “Did not find the tools very beneficial, because the benefits need to be customized for individual states …” “Information may be duplicated elsewhere … Would need to build the measures into projects prior to start … Input not required/hasn’t been a priority.”

Who bears the cost of developing measures for each individual project? Researcher? Program director? What is the benefit/cost difference of applying measures to all projects versus a sample? What is the right balance between qualitative vs. quantitative data to demonstrate program effectiveness? Issues/challenges with what administration wants or needs (prediction versus real cost or savings)

Bill Stone, PE Construction, Materials and Research 1617 Missouri Blvd. P.O. Box 270 Jefferson City Missouri Research Ahead