1. Evaluating Driver Education
Daniel Mayhew
Senior Research Scientist and Advisor
Traffic Injury Research Foundation

2. Overview Background Purpose of Program Evaluation
Clinton & Lonero (2006). Evaluation of Driver Education Programs: Comprehensive Guidelines. AAA Foundation for Traffic Safety.
History of Driver Education Evaluations
Recent Driver Education Evaluations
Non-Traditional DE Evaluations
Implications
The following topics will be discussed during this presentation.

3. Teen Crashes Road safety issue Public health concern
Teen drivers have an elevated crash risk
Those ages 16 to 19 are nearly three times more likely than drivers aged 20 and older to be in a fatal crash per miles driven
Public health concern
Motor vehicle crashes are the leading cause of death for teenagers
Young people ages represent only 14% of the U.S. population but account for 30% of the total costs of motor vehicle injuries among males and 28% among females
The following topics will be discussed during this presentation.

4. Solutions Graduated driver licensing Very popular
In every state and province
Proven to be an effective solution
The following topics will be discussed during this presentation.

5. Solutions Driver education and training Very popular
In every state and province
Not proven to be an effective solution
Traditional programs have generally failed to reduce crashes
Contemporary programs show promise but further evaluation needed
The following topics will be discussed during this presentation.

6. What is Program Evaluation?
Carefully collecting information about a program or some aspect of it
Determining whether programs are:
Appropriate
Adequate
Effective
Efficient
and, if not, how to make them so
The following topics will be discussed during this presentation.

7. What is Program Evaluation?
The goals are to PROVE (does it work?) and IMPROVE driver education
Does it work?
What are the lessons learned from the research that can be used to enhance the program?
The following topics will be discussed during this presentation.

8. What Program Evaluation is NOT?
A useless activity that generates lots of boring data with useless conclusions…
Only able to show the program’s failures
A proof of success or failure of a program
Complex and for experts only
A process that only produces what we expect
The following topics will be discussed during this presentation.

9. Program Evaluation: Key Questions
Does training work?
Do programs have logical objectives & structures?
Do some policies lead to more effective driver education?
Do they achieve their learning objectives?
The following topics will be discussed during this presentation.

10. Program Evaluation: Key Questions
Does training work?
Do they have the desired safety impacts?
reduce crashes
better or safer drivers
Do results vary between programs?
Why?
The following topics will be discussed during this presentation.

11. Why do DE Program Evaluation?
Tell the good news
Make the case for continued or expanded funding
Have an early warning system for problems
Monitor whether programs are producing desired results
Identify ways to improve the program
The following topics will be discussed during this presentation.

12. Why do DE Program Evaluation?
Measure progress objectively
Understand why or why not
Build evidence-based “know-how”
Learn whether programs have any unexpected benefits or problems
Demonstrate program effectiveness
Establish future benchmarks
Improve DE and make it more effective
The following topics will be discussed during this presentation.

13. Types of Program Evaluation
Formative Evaluation
Intended to improve program logic, content, delivery
Summative Evaluation
Intended to prove or “sum- up” program results & benefits
The following topics will be discussed during this presentation.

14. History of Driver Education Studies
Long history
NHTSA: Research Agenda for Driver Education (1994)
Jim Nichols: NTSB paper (2003)
Literature reviews – critical analysis
Mayhew and Simpson: Safety Value of Driver Education (2002)
Lonero and Mayhew: Large Scale Evaluation of Driver Education: Literature Review (2010)
States are encouraged to adopt and implement the standards contained in the NTDETAS to assist in their efforts to reduce teen driver crashes and fatalities.

15. History of Driver Education Studies
Literature reviews – critical analysis
Thomas et al.: A Fresh Look at Driver Education (2012)
DETA: Summary of Recent Reports (2017)
Almost all of these review reports are on the ANSTSE website:
ANSTSE.info
States are encouraged to adopt and implement the standards contained in the NTDETAS to assist in their efforts to reduce teen driver crashes and fatalities.

16. History of DE Evaluations
Mainly summative evaluations – 3 types
Experimental random controlled trials (RCTs) – records, surveys
Quasi-experiments – statistical control
Ecological – population effects
States are encouraged to adopt and implement the standards contained in the NTDETAS to assist in their efforts to reduce teen driver crashes and fatalities.

17. What had early studies done for driver education?
The following topics will be discussed during this presentation.

18. Early Findings Robertson and Zador study (1978), 27 states
No relationship between the drivers who completed a driver education and the fatality rates
Stock et al. (1983), DeKalb county, Georgia
Drivers who completed DE showed better skills only for the first six months
The following topics will be discussed during this presentation.

19. Early Findings Levy study (1990), 47 states
Only a small effect on fatal crashes
McKenna et al. (2000), Pennsylvania
No difference in crashes or convictions
The following topics will be discussed during this presentation.

20. Early Findings Pezoldt et al. (2007), Texas
An evaluation of the Parent-Taught Driver Education program in Texas
Parent-taught novice drivers had more traffic convictions and crashes than commercial or public school- trained drivers
The following topics will be discussed during this presentation.

21. Evaluating Evaluations
Not comprehensive – little or no “formative” evaluation
Unsystematic – “hit & run” research
Study design weaknesses
The following topics will be discussed during this presentation.

22. Evaluating Evaluations
Negative influence on beliefs & policy
Common belief that programs failed to reduce crashes
Support/resources evaporated
The following topics will be discussed during this presentation.

23. Recent Evaluations Shell et al. (2015), Nebraska
Small but statistically significant reductions in collisions
In year one of driving, the driver education group had fewer crashes than the non-driver education group (11.1% versus 12.9%)
The following topics will be discussed during this presentation.

24. Recent Evaluations Mayhew et al., (2014, 2017), Manitoba & Oregon
Modest, but important improvements in safe driving knowledge, attitudes, motivations, skills and behaviors
Level of safe driving knowledge improved
Significant improvements in self-rated driving skills
Driver education in Oregon was associated with a 4.3% statistically significant reduction in collisions
The following topics will be discussed during this presentation.

25. Nebraska and Oregon Studies
Pros
Efforts to improve on previous evaluation designs
Large study populations – census
Statistical controls to account for key factors – e.g. age, gender
Cons
Not true randomized, controlled experiments
Teens self-selected whether they took driver education or not
Statistical controls may not have fully compensated for the lack of random assignment
The following topics will be discussed during this presentation.

26. Nebraska and Oregon Studies
Bottom-line
Promising but not overvalued, in light of study limitations
Results should be treated cautiously
The following topics will be discussed during this presentation.

27. Non-Traditional DE Evaluations
teenSMART is a computer-based training program comprising three core elements:
computer-based driving tutorials;
parent-teen activities, including in-car driving sessions; and
a certification test at the end of the program.
Addresses the key factors that cause most teen driver crashes
hazard detection/risk perception
speed
situational awareness
peer pressure
The following topics will be discussed during this presentation.

28. Non-Traditional DE Evaluations
Developed by ADEPT Driver with the assistance of an Advisory Team of behavioral scientists, including experts in:
traffic safety
adolescent development,
risk perception/management, and
instructional design and technology.
The following topics will be discussed during this presentation.

29. Non-Traditional DE Evaluations
Evidence is compelling (Mayhew et. al 2016, review):
Improves specific driving skills taught in the program
Improves knowledge, driving-related skills and on-road driving performance
Statistically significant lower collision rates based on insurance claims data
The following topics will be discussed during this presentation.

30. Non-Traditional DE Evaluations
Limitations of the evidence:
Not experimental study designs with random subject assignment
other factors related to self-selection of teen drivers to take or not take the program may have also contributed to, or accounted for, their better safety performance
The following topics will be discussed during this presentation.

31. Non-Traditional DE Evaluations
Development and evaluation of a computer-based risk awareness and perception training (RAPT) for novice drivers
after they pass the road test
NHTSA funded research; Dr. Fisher and colleagues, University of Massachusetts
The following topics will be discussed during this presentation.

32. Non-Traditional DE Evaluations
Emerged from growing recognition that:
poor hazard perception skills contribute to teen driver crashes
These critical skills need to be improved through training
Part-task computer-based and/or commentary hazard perception training improves hazard detection and response, at least in a simulated driving environment
The following topics will be discussed during this presentation.

33. Non-Traditional DE Evaluations
Series of studies showed that RAPT:
Improves hazard detection on a driving simulator and on-road (Garay-Vega et al., 2007; Fisher 2008; Pradhan et al., 2006).
Crash-based evaluation of a modified version of RAPT produced mixed results (Thomas et al )
No overall main effect of the program
Trained males had a 24% lower crash rate that was statistically significant
Trained females had a 11% higher crash rate but this was not statistically significant
The following topics will be discussed during this presentation.

34. Non-Traditional DE Evaluations
Updated version of RAPT has undergone further research (Thomas et al., 2017) which showed:
Significant improvements in hazard detection on a driving simulator and on-road
Improved hazard detection using the computer assessment one month after training
Further updates to RAPT and an ongoing multi- year study to replicate the earlier crash-based study using a larger sample size
NHTSA funded
Conducted by TIRF
The following topics will be discussed during this presentation.

35. Implications: Driver Education
Existing DE programs:
may work but they need to be evaluated
May be unrealistic, however, to expect that the traditional form of DE by itself will lead to fewer teen crashes
Programs can be improved by adopting practices that are evidence-based
Adopt and implement the National DE standards (ANSTSE/NTDETAS)
Provides guidance as to how traditional DE should be substantially restructured to potentially achieve safety goals
The following topics will be discussed during this presentation.

36. Implications: Driver Education
A goal of improved programs and the focus of future evaluations should be to:
determine whether enhancements lead to better safety outcomes.
Critical to:
Identify and implement workable improvements
Evaluate program enhancements
Research and pilot test new programs that are evidence-based
The following topics will be discussed during this presentation.

37. Implications: Driver Education
Find resources & political will to support R&D
Evidence-based development is high priority
Support and conduct systematic & comprehensive evaluation
Avoid hit & run research: systematic evaluation is key
Small steps & continuous improvement
The following topics will be discussed during this presentation.

38. Advancing Quality in Driver Education!
Thanks for Listening
Thank you for your support and interest in Driver Education and Training!
Advancing Quality in Driver Education!