Management System Auditing Assessment of auditing methods © David Hoyle 2012

Objective To assess whether current auditing methods will hold back or stimulate developments to ISO 9001

Audit methods Clause based Department based Contract based Determine what you are going to LOOK AT and when you find it what you are going to LOOK FOR Select a method that fulfils your objectives and matches the information and resources available. (NAO) Audit methods Always know what you are attempting to establish relative to your objective Clause based Department based Contract based Process based Behaviour based

Approach Examine the method Identify assumptions Evaluate the method

Clause based approach Select clause of the standard Choose department ISO 9001 Select clause of the standard Choose department Assess conformity with requirements by ? Objective To determine the extent to which the QMS conforms to requirements of ISO 9001 Asking searching questions of each person Revealing evidence of conformity with clauses of standard Audit Report Produce report of nonconformities against clauses

Assumptions Departments are exclusively responsible for meeting certain clauses Evidence of conformity in one Dept is indicative of conformity in other Depts Conformity with clauses is evidence of capability Correcting non-conformity will improve system effectiveness People operate from the same causality as nature and machines

Evaluation Score on a scale of 0-5 Conformity? Inherent risks? Performance? Capability? Efficiency? Effectiveness? Improvements? Confidence? Use of resource?

Clause based approach Select clause of the standard Choose department ISO 9001 Select clause of the standard Choose department Assess conformity with requirements by ? Objective To determine the extent to which the QMS conforms to requirements of ISO 9001 Asking searching questions of each person Revealing evidence of conformity with clauses of standard Audit Report Produce report of nonconformities against clauses

Department based approach Choose department Select clauses of the standard that apply ISO 9001 Assess conformity with requirements by ? Objective To determine the extent to which the QMS conforms to requirements of ISO 9001 Asking searching questions in each area Following a trail through each department Revealing evidence of conformity with clauses of standard Audit Report Produce report of nonconformities against clauses

Assumptions Conformity with clauses is evidence of capability Correcting non-conformity will improve system effectiveness People operate from the same causality as nature and machines Departments are exclusively responsible for meeting certain clauses Evidence of conformity in one Dept is indicative of conformity in other Depts

Evaluation Score on a scale of 0-5 Conformity? Inherent risks? Performance? Capability? Efficiency? Effectiveness? Improvements? Confidence? Use of resource?

Department based approach Choose department ISO 9001 Select clause of the standard Assess conformity with requirements by ? Objective To determine the extent to which the QMS conforms to requirements of ISO 9001 Asking searching questions in each area Following a trail through each department Revealing evidence of conformity with clauses of standard Audit Report Produce report of nonconformities against clauses

Contract based approach Contract/ Order Select random sample of contracts/orders and critical characteristics Obtain or produce flow chart of activities from contract/order to fulfilment ? Assess conformity with contracts and clauses by ISO 9001 Asking searching questions at each stage Following a trail through each process from input to output Revealing evidence of conformity with customer requirements and with clauses of standard Objective To determine whether the organization has the ability to consistently provide product that meets customer requirements Audit Report Produce report showing the degree of conformity with requirements

Assumptions Conformity with clauses is evidence of capability Conformity with ISO 9001 will enable consistent provision of conforming products Correcting non-conformity will improve system effectiveness People operate from the same causality as nature and machines

Evaluation Score on a scale of 0-5 Conformity? Inherent risks? Performance? Capability? Efficiency? Effectiveness? Improvements? Confidence? Use of resource?

Contract based approach Contract/ Order Select random sample of contracts/orders and critical characteristics Obtain or produce flow chart of activities from contract/order to fulfilment ? Assess conformity with contracts and clauses by ISO 9001 Asking searching questions at each stage Following a trail through each process from input to output Revealing evidence of conformity with customer requirements and with clauses of standard Objective To determine whether the organization has the ability to consistently provide product that meets customer requirements Audit Report Produce report showing the degree of conformity with requirements

So what’s the problem? People don’t operate from the same causality as nature and machines We think we can design organizations in the same way we design machines A management system is not a set of documents The behaviour of nature and machines is based on its inherent design properties If you plant an acorn you get an oak tree. You don’t get a horse chestnut or a chestnut horse They are not distracted, they don’t prioritise or choose what to do and they react in a predictable manner.

The circles of influence that create management systems Outputs are the results of processes Satisfied Stakeholders Dissatisfied Stakeholders Produce Demands Business environment Influence Ordinarily an organization is influenced by its operating environment and if its response to that environment is effective its outputs will produce satisfied stakeholders The customers will be satisfied with the products and services The suppliers will be satisfied with the business arrangements The employees will be satisfied with the working conditions The investors/owners will be satisfied with the return on their investment and Society will be satisfied that the organization is creating wealth for the community and not harming it As outputs are the results of processes, we can consider this organization as a system of effectively managed processes If it ignores the business environment it will have ineffective processes and these will produce undesirable outputs that will dissatisfy some or all of the stakeholders maybe not all at once but over time. This shows the organization is an open system that must adapt to its operating environment to succeed System of effectively managed processes System of ineffectively managed processes Organization Desired Outputs Undesirable Outputs Organization is an open system Delivers Organization is a closed system

Reality check Conformity is only part of the picture People make choices People are influenced by images, events and their interaction with others Many invisible risks are hidden in behaviour The impact of behaviour is what effects outcomes and capability

Alternative Approaches Process management audit Behaviour assessment

Process based approach Mission, Objectives Measures Identify organization’s mission, objectives and success measures Identify processes and sub-processes that achieve these objectives ? Assess process effectiveness by Asking searching questions at each stage Revealing how processes are being managed Revealing evidence of capability against objectives and measures Objective To determine whether the organization’s processes are being managed effectively Audit Report ISO 9001 Produce report showing the effectiveness of the system of processes

A Process Audit Q S R T T T Outputs Demands Satisfied Inputs Resources Produce Product Deliver Product Support Product Plan Production Process objectives & Measures Improved efficiency Improved performance Process monitoring How do you know you are doing the right thing? How do you know you are doing it right? How do you make it happen? What are you trying to do? & What will success look like? How do you know you are doing it in the best way? Improved effectiveness Process review Process improvement

Assumptions The system as revealed through systematic enquiry is the system that is producing the results The information presented by the organization is legitimate and has not been fabricated

Evaluation Score on a scale of 0-5 Conformity? Inherent risks? Performance? Capability? Efficiency? Effectiveness? Improvements? Confidence? Use of resource?

Process based approach Mission, Objectives Measures Identify organization’s mission, objectives and success measures Identify processes and sub-processes that achieve these objectives ? Assess process effectiveness by Asking searching questions at each stage Revealing how processes are being managed Revealing evidence of capability against objectives and measures Objective To determine whether the organization’s processes are being managed effectively Audit Report ISO 9001 Produce report showing the effectiveness of the system of processes

Behaviour based approach Goals, Objectives Drivers Identify the system objectives and performance drivers Identify a range of behaviours as outcomes that reflect levels of maturities and cross ref clauses and apply weighting Maturity Grid ISO 9001 Identify people who experience what is happening Invite on-line 360° participation based on the part they play – confidential Objective To measure the inherent level of risk to optimum business outcomes and governance issues caused by the impact of everyday patterns of behaviour Run analysis engine to compute results Produce report showing scored risks to the achievement of objectives and drivers Audit Report

Fulfilment process results (part) # Outcome Av Score 1 We know the objectives that have to be achieved and how our performance will be measured 39.5 2 We know the activities that need to be carried out to achieve the objectives 31.7 3 The necessary physical and human resources are provided when needed 28.2 4 Provisions made to minimize risk are successful 28.5 5 Work commences on time 35.7 6 Work is executed in accordance with policies and plans 30.5 7 Work flows without unplanned interruption 41.6 8 When things go wrong we put them right 39.3 9 No work is released until found in conformity with all requirements 43.4 10 Outputs delivered on time 26.0

Maturity Grid for outcome 8 Which statement matches your experience the closest? When things go wrong we put them right. When things go wrong we put them right as best we can and get on with the job We normally correct mistakes as they occur following our procedures but we wouldn’t record these Records show our procedures have been applied to correct mistakes and prevent them happening again. We usually review all mistakes formally in line with our procedure and take effective action to prevent them recurring Records of all mistakes show that it is rare for the same mistake to happen twice Level 0.00 1.00 2.00 3.00 5.00 Clauses linked 8.5.2 Corrective action 4.2.4 Control of records

Graphical Representation By stakeholder 1 Process owner 2 Process workers 3 Process supplier 4 Process customer 2 3 40 20 1 4 By business process 40 20 1 4 1 Mission management 2 Resource management 3 Demand creation 4 Demand fulfilment 3 2 4 6 By clause 4 Quality management system 5 Management responsibility 6 Resource management 7 Product realization 8 Measurement, analysis & improvement 8 7 5 20 40

Assumptions The system as revealed through systematic enquiry is the system that is producing the results Patterns of behaviour are lead indicators of risk Correlation between clauses and business drivers relative to performance The statements don’t produce wildly different interpretations

Evaluation Score on a scale of 0-5 Conformity? Inherent risks? Performance? Capability? Efficiency? Effectiveness? Improvements? Confidence? Use of resource?

Behaviour based approach Goals, Objectives Drivers Identify the system objectives and performance drivers Identify a range of behaviours as outcomes that reflect levels of maturities and cross ref clauses and apply weighting Maturity Grid ISO 9001 Identify people who experience what is happening Invite on-line 360° participation based on the part they play – confidential Objective To measure the inherent level of risk to optimum business outcomes and governance issues caused by the impact of everyday patterns of behaviour Run analysis engine to compute results Produce report showing scored risks to the achievement of objectives and drivers Audit Report

Confidence level Clause based Not for 3rd Party Audits Department based Not for 3rd Party Audits Acceptable for 2nd Party Audits Contract based Acceptable for 3rd Party Audits Process based Acceptable when combined with other audit methods Behaviour based

Conclusion Which methods will hold back or stimulate developments to ISO 9001? Clause based Hold back Department based Hold back Contract based Hold back Process based Stimulate Behaviour based Stimulate

For Further Details: Behaviour Assessment contact ian.rosam@the-hpo.com Process Audit contact hoyle@transition-support.com Thank you and have a safe journey home