1. TRIUMF Failure Investigation Tool
Violeta Toma
Accelerator Operations Manager, TRIUMF
October 17, 2017

2. Outline Introduce TapRooT® Incident investigation: Summary
Mild consequences
Moderate consequences
Summary

3. What is TapRooT® Incident investigation system: Helps solve problems:
Human errors
Equipment failure
Helps solve problems:
Reactively
Proactively
TapRooT is a root cause analysis methodology developed and owned by System Improvements Inc.
TapRooT was created by Mark Paradies who is the president of the System Improvements Inc. Mark started as a Admiral Rickover’s Nuclear Navy officer and worked for DuPont and Westinghouse before starting the System Improvements in The first TapRooT book was published in 1991 with a focus on human performance. It analyzed the human performance to find ways to correct human factors problems - not to assign blame.
In 2000 the equipment performance investigation was integrated in the TapRooT book.
System Improvements continues to improve the system, the written material, the software and training

4. TapRooT® Process Collect Information Understand what happened
Identify problems which caused the incident
Analyze each problem’s root cause
Search for systemic, cultural & organizational causal factors
Propose corrective actions
Present lessons learnt to management to obtain support for the proposed corrective actions
There are 7 steps in the TapRooT investigation process.

5. TapRooT® Best Practices
Your root cause analysis is only as good as the information you collect
Your knowledge (not only the lack of it) can get in the way of a good root cause analysis
You have to understand what happened before you can understand why it happened
Interviews are NOT about asking questions
TapRooT promotes 7 good practices
Don’t start an investigation by guessing what happened: this will bias your questions and the outcome of the investigation. Collect all information available which can help understand the incident, the sequence of events, etc. When needed, use a SnapCharT: helps organize information collected, identify information missing.
Previous experience can bias your investigation; Looking for cause/effect
Start an investigation by asking “what happened” not “why it happen”. The purpose of the interview is to collect info not justifications for what happened. Ask the interviewee to provide the smallest and apparently least important details – they can be very valuable info for the investigation
Listen: interviews are not only about asking questions - one must listen. Don’t interrupt to ask questions: it will break the remembering process and important info might be lost

6. TapRooT® Best Practices
You can’t solve all human performance problems with discipline, training, and procedures
Often, people can’t see effective corrective actions even if they can find the root causes
All investigations are NOT equal (some investigation steps can’t be skipped)
Statistically, most corrective actions for human performance problems are discipline, training and procedures. When these fail to improve performance it’s obvious that the CA were misused. The investigator should seek CAs based on the root causes which have the potential to safeguard against future human errors.
Investigator must be able to think out of the box. Doing the same thing in the same way for a long time prevents investigators to see effective corrective actions.
Root cause investigations must be tailored to the size of the problem. Simple incident investigation can be performed by one TapRooTer while more complex ones might need a team of 2, 3 or more. Some small incidents simply don’t need an investigation at all, just the implementation of a corrective action.

7. TapRooT® Human Performance Investigation
0) Incident: safety lock applied to a water pump electrical disconnect was cut in error by somebody other than the lock owner
Steps 1 -4:
1) These are old type pumps which comprise a motor connected to the pump via a drive shaft.

8. Collect Information & Understand what Happened
Water pumps
1) Cooling system has 2 water pumps: P-1 & P-2 - none are labeled.
2) Pumps’ electrical disconnects are just behind pumps in poor lit area
3) Electrical disconnect boxes are hand inscribed with felt pen – easy to read under good lighting
4) P-1 failed and was removed for repair by Plant Group (PG). A PG lock was installed on the pump electrical disconnect
5) Decision made to buy a new pump as repair was not possible – PG lock is left in place
6) As the 2 pumps are of the same type and age decision made to replace both (buy 2 pumps)
7) New pumps are delivered
8) Electrical Services (ES) placed a lock on P-1 disconnect and removed the feed wires from the pump motor. Wires are left on the floor with bare ends exposed
9) Defective P-1 removed & new pump is installed by PG – it is found that parts need to be made (adaptor to the water line)
10) Work is halted. Two locks (PG & ES) are left on the pump electrical disconnect
Left: old water pump
Right: new water pump with integrated motor

9. Collect Information & Understand what Happened
11) Parts are ready
12) Water is shut off
13) PG supervisor leaves for the day before work is completed. Mechanical Services (MS) group leader takes over work coordination
15) Keys are handed over
12) PG supervisor locks the second pump (P-2) to prevent its damage by running it without water
15) PG supervisor goes to his desk to pick up the P-2 lock key: 2 keys on the desk – one for P-1 (failed) the other for P-2. PG supervisor cannot remember which is which and hands in the 2 keys to the MS group leader saying “one will unlock the padlock’
16) PG supervisor does not show the MS group leader the padlock

10. Collect Information & Understand what Happened
17) Work finished, system brought to operations conditions
18) the MS group leader gives the 2 keys to an attendant to remove the lock
19) MS and PG supervisor communicate and PG advices MS to cut the padlock
17) Electrical connection to P-1 is finished, system is brought to operations conditions = valves opened
18) The attendant thinks there must be 2 padlocks on the disconnect because he was given 2 keys. He removes a padlock but cannot remove the second one

11. Collect Information & Understand what Happened

12. 22) After the loud bang the MS supervisor locks back P

13. TapRooT® Tool: SnapCharT

14. Search for systemic, cultural & organizational causal factors
Workers (PG & ES) unware of site lock-out procedure for multi-shift lock-out
Use of personal lock for a multi-shift lock
Padlocks hand inscribed, hard to read
PG supervisor didn’t check which key he needs to hand to MS supervisor
PG supervisor didn’t show MS supervisor the lock-out location
MS supervisor didn’t tell attendant that he must remove one lock-out only
MS supervisor didn’t tell attendant the name of the pump to be re-connected

15. Search for systemic, cultural & organizational causal factors
Attendant doesn’t relay that a padlock was already removed
Attendant doesn’t ask which pump to re-connect

16. Propose Corrective Actions
Training Programme and policy/procedure
Multi-shift lock-out
Padlock label
Clear policies & procedures
Audits and Evaluations
Inforce policy on lock-out through internal audits and disciplinary actions when applicable
Timely communication of policies and procedures
The online Lockout/Tagout training did not emphasize that OG padlocks are to be used when work lasts more than one shift and the importance of labeling the padlock with the owner’s name (not only the group)
Lock-out & Disable policies confusing
Had an audit been performed, it would have shown that none of the groups were aware of the device lock-out procedures ..
Analysis of previous NCRs showed that 38% of reported nonconformities were related to policies not being used or in need of improvement. This indicates that this failure of staff to be aware of, and competent in, the application of organizational policies is not unique.

17. Propose Preventative Actions
Ensure policies on Lock-out and on Device Disable are consistent
Group training plans to reference above policies when appropriate
Policy should be revised to include the acceptable methods of marking a lockout/disable padlock
All TRIUMF groups using shop padlocks as part of multi-shift device disables should be made aware of current policy and change all shop padlocks to padlocks from Operations Group

18. Propose Preventative Actions
Although not required by WorkSafe BC, BC Safety Authority or BC Electrical Code, it would be best practice for any electrical wiring that has been made safe (de-energised and device disable installed) and disconnected from a device to also have the device end wires be covered appropriately
Group supervisors to schedule periodic discussion sessions on policies and procedures and invite subject matter experts able to answer questions as needed

19. TapRooT dos and don’ts: Don’t assume Ask “what happened” not “why”
Summary
TapRooT dos and don’ts:
Don’t assume
Ask “what happened” not “why”
Expect more than one root cause
Human error is just the starting point of a root cause analysis
Search for root causes that can be fixed

21. Thank You! Merci! Questions? TRIUMF: Alberta | British Columbia|
Calgary | Carleton | Guelph | Manitoba |
McGill | McMaster | Montreal | Northern
British Columbia | Queen’s | Regina |
Saint Mary’s | Simon Fraser | Toronto |
Victoria | Western | Winnipeg | York
Thank You!
Merci!
Questions?