1. Air Carrier Continuing Analysis and Surveillance System (CASS)
Verifying Performance of the Maintenance Programs

2. Objectives
Determine how to assess the performance of the maintenance program
Identify “Performance” processes within the air carrier’s maintenance program
Describe the carrier’s procedures in its CASS for monitoring performance of its maintenance program

3. Overview Review: Audit types and procedures Audit plan Audit process
Air carrier’s responsibility for CASS
Maintenance program goals
Major CASS activities
Audit types and procedures
Audit plan
Audit process
Audit analysis

4. Maintenance Program Goals
As stated in CFR 14, Sections and :
All maintenance performed by the air carrier or by other persons is performed according to the air carrier manual. 
Competent personnel and adequate facilities and equipment are provided for the proper performance of maintenance.
Each airplane released to service is airworthy and has been properly maintained for operations in air transportation.

5. Air Carrier’s Responsibility
Maintain a system for the Continuing Analysis and Surveillance System (CASS) to measure the performance and effectiveness of its maintenance program
Culture of safety helps establish a properly functioning CASS
Properly functioning CASS should include an audit function to:
Review the actual work performed within its maintenance program

6. The CASS Cycle
CASS monitors maintenance program performance and effectiveness through a closed-loop system of four major activities.

7. Verifying Maintenance Program Performance

8. Surveillance: Audits Process
What do Audits ask:
Are they doing what they are
supposed to be doing?

9. For purposes of CASS, and in line with general audit methodology, an audit is defined as a formal examination of the activities of a department or area of an air carrier’s maintenance program based on compliance with the standard, which is the air carrier’s manual.
Remember that the one of the requirements of air carrier programs is that the program itself must ensure that all three of those regulatory objectives are being achieved. KEY POINT
Ask the class what the three regulatory objectives are?
What Is an Audit?
An audit is a formal examination of the activities of an air carrier’s departments or areas as compared to a standard, which is the air carrier’s program as written in its manual.  
Audits are intended to measure an air carrier’s compliance with maintenance program requirements.
The maintenance program itself must ensure that all maintenance activities are accomplished in accordance with the processes and procedures of the maintenance program. §§ and

10. Types of Audits
These are the four types of audits usually utilized by air carriers.
The four items are self-explanatory.
Generally, the differences are who accomplishes the audits and the different audit targets.
Internal Audits are performed by air carrier audit personnel on air carrier activities.
External Audits are performed by air carrier personnel on activities of outside entities or contract maintenance providers.
Self-Audits are performed “in-house” by a specific individual or position within a department or shop.
Third Party Audits are performed by third parties (FAA, C.A.S.E., DOD, etc.) on the carrier.

11. Audit Methods
These are the audit methods and their objectives that are used in one or more of the four audit types that we just discussed in the last slide.
The primary type of audit method that we expect the air carrier to use is the work-in-progress method. The purpose of these audits is to determine if the worker is following the manual. This is the § (a) and § (a) requirement. A negative finding is a program deficiency under the CASS rules and must be addressed. This follows the plain language meaning of the CASS regulation, i.e., the performance of the maintenance program; are they doing what the program requires.
The transaction method is used primarily for records review and again serves to see if the maintenance program standard for the record form, record procedures, record completion and record accuracy standards of the program are being achieved.
The Systems method is a high level, comprehensive, and documented examination of all of the activities, records, processes, and other elements of an air carrier’s various systems to determine their conformity with the requirements of a standard such as the air carrier’s maintenance program.
These audits also identify various latent faults in the air carrier’s overall systems. These audits are usually, but not always, accomplished by a professional audit agency and result in a written and detailed report that is used by senior management to make corrections to the systems.
Work-in-Progress Audits examine adherence to the execution of the maintenance program (“As the task is being performed”). This is a primary proactive audit method.
Transaction Audits check compliance against a standard or requirement, usually an after-the fact verification such as a records review.
Systems Audits are independent, periodic and documented examinations of all of the activities, records, processes, and other elements of an air carrier’s various systems to determine their conformity with the requirements of a standard such as the air carrier’s maintenance program.

12. Planning Audits
Risk-based resource allocation based on company experience
Software-supported risk analysis
Planning process should systematically make decisions compatible with the size and complexity of operations

13. Planning Audits (continued)
Air carrier’s audit plan should include:
What should be audited or collected
Who performs audits or data collection
How audits and data collection should be performed
How often audits and data collection should occur

14. Auditors
Thorough knowledge of auditing and the air carrier environment
Should be able to objectively identify and document deficiencies

15. Planning Priority Regulatory compliance (promotes safety)
Operational efficiency
Consider company experience and outside factors, such as:
Inspections – “OIG”
Reports – “NTSB”
Special Studies – “FAA”
DoD – “DOT”

16. Audit Verification Government Publications Program Description
Flight Operation
Ground Operations
Maintenance
Alterations
Conformity
Repairs
Government Publications
Industry Documents
Operator Documents
Manufacturer Documents
Program Description
Records
Reliability
Scheduling/Tracking
Aircraft Loading
Deicing
Taxiing and Towing
Fueling
e.g., regulations, airworthiness directives
e.g., ATA 100 index
e.g., General Maintenance Manual
e.g., maintenance manual, service manual

18. Red Rock Airline: D091 Quarterly Audit v 3.0

19. Effective Communication within CASS
Maintenance personnel should be able to discuss concerns they may have
Auditors should:
Ask questions not included in the checklists
Adopt a “what if?” proactive approach during surveillance activities

20. CASS Audits
Proactive – search for potential problems before undesired events
Responsive – procedures established to address unscheduled audits driven by findings and events (IFSDs, accidents, incidents, negative trends)
Productive – encourage open lines of communication throughout the organization
Priority – regulatory compliance, efficiency

22. Verifying Maintenance Program Performance22

23. Objectives of Audit Analysis
Identify the maintenance program deficiencies. To accomplish that:
Audit information should be factual
Who? What? When? How?
Make no prejudgments about root cause
Analysis should be:
Objective
Systematic

24. What to Watch For
Blame culture is an obstacle to improving safety.

25. What Is a Blame Culture? Blame Fixes the blame, move on…
Focuses on the individual who made the error
Stops short of identifying systemic problems
Stops short of identifying root cause
Never fixes the problem
Tends to allow mishaps/mistakes to recur
Blame

26. Inspector’s Oversight
Determine desired outcome – Review process
Does the process contain data points?
Do data points provide useful data for analysis (are data repeatable)?
Collect data – Procedures to perform the audit
Review data
How are data used?
How are data analyses documented?
Do the data support the air carrier’s finding?
How are data analyses results communicated?
(How do results get to the next step?)

27. Inspector’s Oversight (continued)
How does the carrier analyze audit results?
Identify deficiencies
Can either be a real or a potential hazard?
Are audit results used in part to allocate resources?
Do audit procedures include calling upon additional technical expertise if required?

28. Practice Exercise PE-01-PER

29. Practice Exercise PE-02-PER

30. Summary Review: Audit types and procedures Audit plan Audit process
Air carrier’s responsibility for CASS
Maintenance program goals
Major CASS activities
Audit types and procedures
Audit plan
Audit process
Audit analysis

31. Objectives
Determine how to assess the performance of the maintenance program
Identify “Performance” processes within the air carrier’s maintenance program
Describe the carrier’s procedures in its CASS for monitoring performance of its maintenance program

32. QUESTIONS ?