`+ Confidential Close Call Reporting System (C3RS) Final Lessons Learned Results C3RS User Group April 11, 2017 Dr. Joyce Ranney Volpe National Transportation Systems Center Cassandra Cantu, M.S. Volpe National Transportation Systems Center Dr. Maryam Allahyar Federal Railroad Administration Melinda Davey, M.S.E. Jacobs Dr. Jonny Morell Syntek Technologies Volpe The National Transportation Systems Center Advancing transportation innovation for the public good

Introduction and Methods General C3RS Information Results and Findings Agenda Introduction and Methods General C3RS Information Results and Findings Implementation Impact Sustainability Recommendations Next Steps

C3RS Evaluation Questions Implementation What are the characteristics of a successful C3RS implementation? Impact What is the impact of C3RS on safety and safety culture? Sustainability What are the conditions needed to make C3RS sustainable?

What is the Data Collection Method for the C3RS Evaluation?

Evaluation Status UP North Platte CP Central NJT Amtrak (West) Baseline Midterm Final UP North Platte 2007-2013 X CP Central 2008-2014 NJT 2009-2015 Amtrak (West) 2011-2016 Cross-Site Reports 2014-2017 In Reviews In Process TODAY

Introduction and Methods General C3RS Information Results and Findings Agenda Introduction and Methods General C3RS Information Results and Findings Implementation Impact Sustainability Recommendations Next Steps

Volume of C3RS Reports Monthly Average Number of Reports per 100 Eligible Employees All four sites received enough reports to generate corrective actions

Overall Categories Across Demonstrations: Excess Speed and Switching Most Reported Categories Percent of Cases

Distribution of Reported Categories Varies Across Sites Three of four sites reported excess speed All sites reported switching

More C3RS Cases are Unknown than Known Events A Close Call Event is a specific situation or occurrence, observed by a railroad employee, that has a potential for more serious consequences Unknown 56% Known 44% Reporters Unknown Event reported to C3RS and identity remains unknown 56% Reporters Known Minor incidents reported to C3RS and communicated directly to the railroad 44% *Sites 1, 2, 4

Corrective Action Examples Category Action Site Action Site Action Site Excess Speed Reformatting track bulletins 1 2 3 Reminder tags Modified signs 1 2 4 Consolidating slow orders 4 Dispatcher process improvements Switching Improved communications Improved switch visibility 12 Training Poster about problem switches Derailment Improved communications in yard (e.g. squawk box) Visual checks (e.g. camera, checking with yardmaster) 13 New policy about derail removal Collision Safety bulletin Barriers to prevent pedestrian crossing Communication improvements Contractor accountability Blue-Flag Maintenance procedure for flag removal Improve maintenance of blue lights   General Improved job safety briefings Departure checklist hang tag

Introduction and Methods General C3RS Information Results and Findings Agenda Introduction and Methods General C3RS Information Results and Findings Implementation Impact Sustainability Recommendations Next Steps

Overall Summary: How Far Did Each Site Progress? (Site Names Disguised) Implementation Successful C3RS implementation Impact Impact on safety and safety culture Sustainability Conditions to make C3RS sustainable   Implementation Impact Sustainability Site 1  Site 2 Site 3 Site 4

Implementation Findings Question Answer What are the characteristics of a successful C3RS implementation? Implementing C3RS as planned is possible within transportation departments in the railroad industry. While the demonstrations had varying degrees of implementation strength, each of them performed sufficient planned activities to justify an evaluation of their impacts High reporting rates were not associated with implementation strength, impacts, or sustainability Sites with strong implementation demonstrated significant impacts and had common characteristics Sites identified similar improvements for implementation

Implementation Finding While the demonstrations had varying degrees of implementation strength, each of them performed sufficient planned activities to justify an evaluation of their impacts. All demonstrations’ reporters submitted and PRTs received reports from the third party Demonstration Sites Implemented C3RS as Planned Third parties collected reports and provided them to carriers Labor, management, FRA participated at the carrier: Prioritized, grouped reports Applied problem analysis methodology Considered possible corrective actions Chose corrective actions to recommend Recommended corrective actions to management Implemented corrective actions Monthly Average Number of Reports per 100 Eligible Employees

 Implementation Finding High reporting rates were not associated with implementation strength, impacts, or sustainability. Monthly Average Number of Reports per 100 Eligible Employees Implementation Impact Sustainability Site 1  Site 2 Site 3 Site 4

Implementation Finding Sites with strong implementation had common characteristics. Carrier: local and senior management provide resources to the PRT and implement corrective actions Labor: promote detailed reporting, participate in PRT, implement corrective actions Stakeholders cooperate: Negotiate IMOU, maintain confidentiality, and support training Implement and monitor corrective actions Resolve disputes in a way that encourages continued future reporting All stakeholders agree that C3RS is valuable

Cause and Effect Diagram Main Factor A cause and effect diagram was created based on the evaluation results Each site was rated on each detailed factor Ratings were compared among sites Detailed Factor Detailed Factor Effect Detailed Factor Detailed Factor Detailed Factor Main Factor Diagrams and detailed ratings for each site will be available in final report

Successfully Implemented C3RS Implementation Factors Model Successfully Implemented C3RS

Implementation Finding Sites with strong implementation demonstrated significant impacts. Site 1 Site 2 Site 3 Site 4 The average rating for each of the main implementation factors for each site is shown above. You can see that Site 4 had the poorest implementation which led to low impact. Sites 1, 2, and 3 have more good and very good ratings which led to good impact. .

Implementation Finding Sites identified similar improvements for implementation. Communication: Support Team to PRT: status of corrective actions PRT and Support Team to workforce and managers wanted: C3RS accomplishments PRT Data Analysis and Efficiency: Analysis was done one case at a time, so the PRTs got behind Support Team wants more data on corrective action analysis to help with prioritization Process improvement experts in company not connected with PRT PRT members had to sit and wait for administrative tasks Tracking of Corrective Actions: Some corrective actions not recorded Status unknown Impact and effectiveness not measured

Impact Findings Question Answer What is the impact of C3RS on safety and safety culture? C3RS has demonstrated bottom-line impacts in 3 of the 4 sites in derailments, injuries, discipline hearings, equipment costs, and safety culture Derailments decreased at three sites in the presence of C3RS (1) corrective actions and (2) communication enabled by discipline protection Injuries decreased at one site in the presence of corrective actions such as improved job safety briefings Discipline hearings and associated costs decreased in the presence of C3RS protection Labor perceived that safety culture between supervisors and labor improved in the presence of C3RS-enabled communication

Impact Finding: Derailments Derailments decreased at three sites in the presence of C3RS (1) corrective actions and (2) communication enabled by discipline protection. Derailments  41%  30%  20%  -- Site 1 Site 2 Site 3 Site 4 Legend  = Reductions (improvement) -- = mixed results/ little to no change % of Derailment/ Switching cases 25% of cases on average across all sites Interview Quotes “Now if they have a run-though switch, they call it in and it is repaired, so a derailment is not caused. I see less equipment damage caused.” (Site 2) Corrective Actions Improved yard communication, policy about derail removal, reverse move training etc.

Impact Finding: Injuries Injuries decreased at one site in the presence of corrective actions such as improved job safety briefings. Transportation Personal Injuries ?  18% -- Site 1 Site 2 Site 3 Site 4 Legend  = Reductions (improvement) -- = mixed results/ little to no change ? = no data Interview Quotes “When you experience a mistake, you learn and become more cautious.” (Site 2) Corrective Actions Redesigned Job Safety Briefing procedure, trained workforce, used extensively (Site 2)

Impact Finding: Discipline Hearing Cost Discipline hearings and associated costs decreased in the presence C3RS protection. Disciplinary Hearings  90%  ?  39% Site 1 Site 2 Site 3 Site 4 Legend  = Reductions (improvement) ? = no data Interview Quotes “C3RS protection helps us avoid discipline hearings.” (Site 1) “A discipline hearing cost approximately $10,000 each in travel and work time.” (Site 3) “Reducing discipline in the dispatching office helped us find an error in the software and justified a budget to fix it.” (Site 3)

Safety Culture Survey* Impact Finding: Safety Culture Labor perceived that safety culture between supervisors and labor improved in the presence of C3RS-enabled communication. Interviews: Labor: “Definite impact on communication between labor and management. In the past, if you messed up, you would bury it.” Manager: “My workers feel more comfortable approaching me.” Safety Culture Survey* Site 1 Site 2 Site 3 Site 4 Safety Culture Organization Supervisor Coworker  - Safe Behaviors ? Legend  = Improvement - = mixed results/ little to no change ? = no data * 4000+ labor participants

Sustainability Findings Question Answer What are the conditions needed to make C3RS sustainable? C3RS is sustainable in railroad transportation when enlisting the support of labor, management, and FRA. Strong implementation, impact, and agreement between stakeholders are needed for a site to be sustainable FRA has taken steps to support sustainability in the railroad industry and has expanded the program

Sustainability Finding: Demonstration Sites Strong implementation, impact, and agreement on path forward between stakeholders are needed for a site to be sustainable. Implementation: Sustained sites had on average “good” implementation Impacts: Sustained Sites had safety and other positive impacts Agreement on Future: Sustained sites had agreement among stakeholders on a path forward with FRA and a continued focus on improving the C3RS process 1 2 3 4

Office of Railroad Policy and Development Sustainability Finding: Industry FRA has taken steps to support sustainability in the railroad industry and has expanded the program FRA actions: FRA funded a national program and established an office to support C3RS NASA was selected as the third party Evaluation Results: FRA used impact findings to: Obtain FRA/Congress funding Motivate industry to participate New Railroads: Five more railroads joined C3RS, including new crafts Challenges for future: Class 1’s are not participating at this time $$$ Human Performance Division $$$ Office of Railroad Policy and Development Derailments 41% Injuries 18% Discipline Hearings 90% Safety Culture Class 1

Summary of Findings Implementation Implementing C3RS is possible within the railroad industry. For implementation to be strong, need commitment and involvement by labor and management. Impact C3RS can demonstrate bottom-line impacts on safety and safety culture, if there is strong implementation. Sustainability C3RS is sustainable in the railroad industry with strong implementation by carriers and continued support by FRA.

Introduction and Methods General C3RS Information Results and Findings Agenda Introduction and Methods General C3RS Information Results and Findings Implementation Impact Sustainability Recommendations Next Steps

Recommendations Labor Management FRA NASA Encourage detailed reports, not just frequent reports Participate in corrective action implementation Management Resolve disputes in a way that encourages continued future reporting Monitor impact of corrective actions to determine if effective Determine impact measures based on details of corrective actions FRA Determine ways to measure impact in the future to ensure continued justification and funding for the program (Lessons Learned effort ending) NASA Consider ways to help sites share corrective actions with the industry

Introduction and Methods General C3RS Information Results and Findings Agenda Introduction and Methods General C3RS Information Results and Findings Implementation Impact Sustainability Recommendations Next Steps

Prepare and submit Final Report for publication TRB Next Steps for Lessons Learned Team Prepare and submit Final Report for publication TRB Article in Transportation Research Record Presentation at 2018 TRB Annual Meeting Articles in publication process at FRA C3RS Lessons Learned Midterm Report Research Result: "How an Active C3RS Program Can Impact Safety Results“ Research Result: "C3RS Reporting Trends and Implementation“ Research Result: “C3RS Demonstration Pilot Uncovers Dangerous Software Glitch and Saves $890K in Costs for Disciplinary Hearings”

For More Information – FRA eLibrary https://www.fra.dot.gov/eLib/Find 10 publications so far The Lessons Learned team has been creating publications for FRA’s website. There are 10 so far, and more are coming over the next couple of years. To find them: Go to the FRA eLibrary (the link on far right of the top menu) Type c3rs into the search field. Select Research Results and Technical Reports to filter the results 35

Volpe The National Transportation Systems Center `+ Contact Information Dr. Joyce Ranney Joyce.Ranney@dot.gov (617) 494-2095 Volpe National Transportation Systems Center Cassandra Cantu, M.S. Cassandra.Cantu@dot.gov Melinda Davey, M.S.E. Melinda.Davey@jacobs.com (734) 330-7957 Jacobs Dr. Jonny Morell jmorell@syntek.org Syntek Technologies Volpe The National Transportation Systems Center Advancing transportation innovation for the public good