1. Research Performance Measures Survey 2014
9/8/2018 3:42 AM
Research Performance Measures Survey 2014
Presented at 2014 AASHTO Research Advisory Committee Meeting,
Madison WI, July 24, 2014
Bill Stone
Research Administrator
Missouri Department of Transportation
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. AASHTO RAC Survey State DOT Survey -- March 2010
9/8/2018 3:42 AM
AASHTO RAC Survey State DOT Survey -- March 2010
Identified very broadly defined performance measures used by state DOTs at the time
Measures taken from public sources such as accountability reports, state DOT websites, and other web-based resources
Out of 40 states, 10 had published measures and 30 had no public information
Survey lists metric, source and URL
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. This is from the 2010 AASHTO Survey (Analysis)

4. DOT State Stats http://stats.mtkn.org
9/8/2018 3:42 AM
DOT State Stats
The DOT State Stats report is a synthesis of facts, figures, statistics and metrics pulled from accountability reports, online performance measurement dashboards and fact books.
2009: Need existed to easily find and access local statistics
Editions available:
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. DOT State Stats
Annual figures compiled in easy-to-use synthesis report online. Excel dataset also available online.
Created and launched by previous MoDOT Librarian; assisted by members of the Midwest Transportation Knowledge Network
(MTKN) and hosted on their website.
2012 edition included metrics from 42 states.
Publication is used by the transportation community at large.

6. Searching on “research” retrieves only 4 results out of 1,580 measures
DOT State Stats
Synthesis report  no index
Dataset  searchable but not categorized
No longer current  not updated since 2012
State Department of Transportation Benchmarks, Facts, and Statistics
Current as of Friday Mar. 22nd, 2012
State
Measure
Year
Citation IA
Number of new transportation research dollars secured
2011 WA
Total value of research projects
Strategic planning & research fund distribution
Number of WSDOT research projects planned
Searching on “research” retrieves only 4 results out of 1,580 measures

7. RAC Value of Research Task Force State DOT Survey -- May 2014
9/8/2018 3:42 AM
RAC Value of Research Task Force State DOT Survey -- May 2014
Obtain input from research administrators actual or anticipated use of research performance measures to document the progress and success of their research programs
Measures with performance targets or metrics that track activity
Measures that were program-specific not project-specific
Measures from all research program areas (pure research projects, library and/or technology transfer program)
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. 2014 Survey Results
49% response rate
(25 states)

9. Does Your State Track Research Program Performance?
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. For States That Don’t Track Measures
Have future plans (5)
High interest (1)
Have potential measure in mind (1)
Discussed it (1)
Track but not as measure (1)
No comment (1)

11. Research Performance Measure Characteristics
9/8/2018 3:42 AM
Research Performance Measure Characteristics
Track activity
56%
Have performance targets
28%
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. How Frequently Are Measures Collected or Reported? All Responses
20% of states collect or report statistics for more than just one time period

13. Are the Measures Published? All Responses-13% of States Do Both

14. How Does Your State Use the Measures?
CURRENT USE
Customer satisfaction (AZ)
Monitor progress toward strategic goals (DC); adjust performance (TX)
Make a difference in organizational business processes (MO)
Determine program effectiveness (IA)
Identify strengths and weaknesses (IN); what’s working or not working (MD)
Tied to individual performance evaluations (LA);

15. How Does Your State Use the Measures?
CURRENT USE (CONT’D)
Assess researcher performance and the usefulness of completed research (NC)
Select new projects (TX)
Ensure that projects move forward according to set budget and schedule (UT, WY)
Develop and monitor SPR Part 2 program (WI)
FUTURE USE
Will use to direct funding (FL)
Will use to gauge and improve performance (IL)

16. Measure Categories
Applied 31 unique categories to 103 submitted individual measures
Measures by Status
Current: 69 measures
Planned: 21 measures
Considering: 10 measures
Past: 2 measures
In progress: 1 measures
Some measures have multiple categories applied

17. Measure Categories
TOP CURRENT: Completion (14), Implementation (9), Library Utilization/Processing (8), Training (7), Satisfaction (6)
TOP IN PROGRESS/PLANNED/POTENTIAL: Implementation (6), Completion (3), Cost Savings (3), Timeliness (3), Within Budget (3)

18. Completion Category Examples of Measures
Related to number
No. of completed research projects
Related to on-time or on-budget
% of research projects meeting original completion date
No. of projects completed in the FY on schedule
% of projects completed according to the initial budget
% of projects completed on time/on budget
% of studies completed within the approved schedule of the Work Program

19. Completion Category Examples of Measures Cont’d
Related to satisfaction
% of completed studies deemed satisfactory by the project sponsor
Customer satisfaction survey for completed Research
Related to implementation
In past 5 years, 75% of completed research projects provide recommendations for implementation of results endorsed by the Project Review Committee

20. Implementation Category Examples of Measures
Related to number
No. of research results and best practices implemented
No. of NCHRP and other external research program results implemented
% of projects implemented
Fully, partially, later, cannot/or should not be implemented, within 2 years of final research report (using 5 years of data)
% of projects actively utilized in the field

21. Implementation Category Examples of Measures Cont’d
Related to number cont’d
% of projects that have resulted in a spec, policy, procedure, manual, requirement or material change
In past 5 years, 75% of completed research projects provide recommendations for implementation of results endorsed by the Project Review Committee
Multi-year tracking of implementation work
Related to funding
Amount of funding for implementation activities

22. Level of Effort to Compile Measures

23. Tools Used to Collect Measures All Responses-75% of States Use More Than One

24. How Have You Used the RPM Website?
CA and MN: Reviewed
DC & LA: Used as basis for developing PMs
IL: Used to develop a structure for functional PMs
IN: Exploring how to move measures to RPM
UT: Use to upload measures
ID, LA, MI, MO, PA and UT: Use for HVR submittals

25. RPM Website Tools: How Useful?

26. RPM Web Comments PROS CONS
“RPM Web was extremely useful in helping us grasp the different types of performance measures and developing a structure for functional performance measures.”
“Need to re-visit the site now that our program is more developed to see how it might help.”
CONS
“Adapting to RPM Web would require a lot of work.”
“Have mixed feelings about the usefulness.”
“Did not find the tools very beneficial, because the benefits need to be customized for individual states …”
“Information may be duplicated elsewhere … Would need to build the measures into projects prior to start … Input not required/hasn’t been a priority.”

27. Observations/Questions
Who bears the cost of developing measures for each individual project? Researcher? Program director?
What is the benefit/cost difference of applying measures to all projects versus a sample?
What is the right balance between qualitative vs. quantitative data to demonstrate program effectiveness?
Issues/challenges with what administration wants or needs (prediction versus real cost or savings)

28. Contact Information
Research
Ahead
Bill Stone, PE
Construction, Materials and Research
1617 Missouri Blvd.
P.O. Box 270
Jefferson City Missouri