EVALUATION 101: HOW CAN WE DEMONSTRATE PROGRAM RESULTS? Jon E. Burkhardt, Westat Dr. David J. Bernstein, Westat Prepared for the National Center on Senior Transportation Easter Seals / n4a December 9, 2008

2 Presentation Outline  NCST Project Objectives  The transportation improvement process  Performance measurement and evaluation: different activities, different uses  Applying performance measurement and evaluation to transportation programs  Summary  Contact information  Sources and references

3 NCST Project Objectives Greater coordination between the aging community and the transportation industry Increasing the family of transportation options for older adults at the local level Ensuring caregivers are educated regarding transportation options Addressing barriers to implementing more transportation services for older adults

4 The Transportation Improvement Process  Analyze existing conditions: determine local transportation needs and resources  Define community goals, objectives, and evaluation strategies  Confirm working relationships  Design and assess alternative services and strategies  Implement service changes  Evaluate and improve the services 

5 Performance measures EvaluationsWhich?What?When?Why?

6 Why All the Fuss?

7 The “What:” Performance Measurement  Performance Measurement: periodic but regular monitoring and reporting of program accomplishments, particularly progress towards pre- established goals  Typical measures:  inputs (resources applied to a problem)  outputs (numeric measures of program products)  outcomes (what changed)

8 The “Why:” Evaluation  Evaluations: systematic studies to assess how well a program is working  Some of the possible components:  The process of implementing the program  Report on program objectives achieved [or not achieved]  Attribution of the results: to the program? to events outside of the program?  Recommendations for improving program results

9 The Differences  Performance measurement  will provide regular data for reports about the progress of a program  can explain what is happening within a program  Evaluations  are more in-depth  help explain why program performance did or did not change  help attribute performance measures to program results

10 Performance Measurement Data  Inputs: Measures of the resources that are applied in providing services  Activities: Measures of the services that are being provided  Outputs: Measures of the quantity of services provided or the quantity of a service meeting quality requirements  Outcomes: Measures that address the intermediate or long-term results of a program on those receiving a service

11 Program Outcome Model Resources dedicated to or consumed by the program Money # of staff Staff time # of volunteers Volunteer time Facilities Equipment Supplies Other What the program does with the inputs to fulfill its mission More staff & volunteers to provide trips I & R services New dispatch systems Driver training New vehicles The direct products of program activities More rides More riders Additional volunteers Greater service span Quicker I & R Shorter wait time Benefits for participants during and after program activities Improved access Greater sense of independence Increased customer satisfaction Increased flexibility Improved communication among providers

12 How to Use Performance Measures   Improve decision making: Ensure that programs are being implemented. Provide input to day-to-day program management, funding decisions, and support strategic planning for services to clients.   Monitor Service Performance: track resources, program production and results, and monitor the need for more comprehensive examination and analysis (service interventions, evaluations).   Report Results/Be Accountable: Provide information to various parties, including staff, funders, service providers, program partners, clients, and other stakeholders.

13 Good Performance Measures  Focus on results  Are relevant and useful to program managers and stakeholders  Are readily measurable and countable  Provide valid, verifiable, and reliable information  Are clear and understandable, requiring only minimal explanation to be understood  Can be compared to targets, comparable programs, or legal or quality standards

14 Performance Measurement Questions 1. How many people are being served now compared with before we started? 2. How many trips are provided now? How many were provided previously? 3. Is my program meeting its targets? 4. Is my program efficient? Effective? Cost effective? 5. How does my program measure up against other programs?

15 Evaluation Questions 1. Can the results of our program be attributed to our program? 2. How can we meet the needs of our community? 3. Did our program meet its goals? Why or why not? 4. Did the way we implemented our program influence the results we got, or did not get? 5. How can we improve our program?

16 Primary Data Collection Decisions  Who will collect what kinds of data?  Which data will be collected at which points in time?  How will the data be used?

17 Potential Evaluation Components  System characteristics:Resources (inputs)  Performance measures:Efficiency (cost / mile, etc.) Effectiveness (trips / mile) Cost effectiveness ($ / trip)  Service quality:Consumer and management data  Service evaluations:Outcomes and impacts

18 Statistics for Performance Measures  Services delivered  Vehicle miles of service  Vehicle hours of service  Services consumed  Unlinked passenger trips  Unduplicated persons served  Fully allocated costs  All costs required to provide transportation service

19 Service Quality Components  Acceptability: reliability, connections, trust, comfort, respect  Accessibility: can physically use, can get information to use, proximity  Adaptability: flexibility, responds to specific requests, meets trip needs and special needs of clients  Affordability: not excessive money, time, or effort required to travel  Availability: frequency, hours / days / places available

20 An Example of Outcome Evaluation GOAL: Increase seniors’ knowledge and use of transportation options EVALUATION STEPS:  Do seniors know more about the options?  Have seniors increased their use of options?  Which outreach and education activities have been undertaken?  How do we know that the outreach and education activities are responsible for the changes?

21 Sources of Service Quality Data  Dispatch and driver logs  Records of complaints  User surveys

22 Survey: What’s your objective?  Needs analysis: Ask potential riders [or their advocates] to find out their needs  Customer satisfaction: Ask current transportation users about their satisfaction with services  Volunteer driver assessment: Ask volunteer drivers [and passengers] about their experiences  Service provider perspectives: Ask existing service providers what improvements are needed

23 Survey Considerations  Different strategies required for different groups  How to administer the survey (by mail, phone, internet, in-person, or some combination?)  When to administer (before, after, or before and after?)  Pilot test your survey and refine it based on the pilot test results

24 Summary Monitoring and evaluation should  Be based on valid data and replicable analyses  Support day-to-day program management and operations: expand / contract; continue / change  Help measure progress towards goals and objectives  Indicate potential service and program improvements  Provide accountability to funding sources  Support program continuation

25 Contact Information WESTAT 1650 Research Blvd Rockville, Maryland Jon Burkhardt Phone: 301/ David Bernstein Phone: 301/

26 Sources and References Slide 6: United Features Syndicate, 3/13/99 Slides 7-9: U.S. General Accounting Office. Performance Measurement and Evaluation: Definitions and Relationships. April and Wholey, J., Hatry, H., and Newcomer, K. (2004). Handbook of Practical Program Evaluation html htmlhttp:// html Slide 10: Governmental Accounting Standards Board [GASB]. (1994). Concepts statement no. 2, Service efforts and accomplishments reporting. and Montgomery County, MD. (March 2006). Montgomery Measures Up; ec/howtoread.pdf ec/howtoread.pdf ec/howtoread.pdf

27 Sources and References (cont.) Slide 11: United Way of America (2006). Measuring Program Outcomes: A Practical Approach. Slide 12: Epstein, P.D. (1988). Using performance measurement in local government: A guide to improving decisions, performance, and accountability. New York, National Civic League Press. Slide 13: Bernstein, D.J. (2000). Local Government Performance Measurement Use: Assessing System Quality and Effects. Washington, DC: George Washington University. Available from ProQuest-University Microfilms Inc., Slides : Burkhardt, J.E. (2004) Critical Measures of Transit Service Quality in the Eyes of Older Travelers. Transportation Research Record No. 1835, Transportation Research Board, Washington DC.