Automotive Metrology Laboratory The National Metrology Laboratory of South Africa Meeting the requirements of clauses 7.6 and 8.1 of ISO/TS 16949 NAAMSA Metrology Interest Working Group – 16 February 2005 Eddie Tarnow Metrologist Automotive Metrology Laboratory

Discussion overview Points for discussion • ISO/TS 16949: clause 7.6; Control of monitoring and measuring devices Selection of the correct test & measuring equipment Calibration of equipment Verification/validation of equipment Calibration intervals Measurement traceability Calibration status of equipment Calibration integrity Assessing the impact of “out of calibration” equipment Calibration records Validation of software used for testing measurement

Discussion overview Points for discussion continued • ISO/TS 16949: clause 7.6; Control of monitoring and measuring devices Laboratory requirements Internal Laboratory Laboratory scope Competency of personnel Testing of the product Ability to test according to national/international specifications/methods External Laboratory Acceptance of laboratory to the customer

Discussion overview Points for discussion continued • ISO/TS 16949: clause 8.1; Measurement, analysis and improvement, General Demonstrating conformity of the product ISO/TS 16949: clause 8.2.4; Monitoring and measurement of product Measuring the characteristics of the product Summary

ISO/TS 16949: Clause 7.6 Control of devices

Clause 7.6 Selection of appropriate test & measuring equipment Have the required tests/measurements been clearly identified? Has an analysis been conducted to determine the measurement accuracy required? What uncertainty is acceptable? What instruments have been identified that will perform the measurement/test? (Are they fit for the intended purpose?) Have the instrument accuracy specifications been correctly interpreted? (especially taking into account environmental tolerance) Has cost vs accuracy been adequately evaluated? (TUR) high accuracy – increased calibration costs, fewer calibration service providers, shorter calibration intervals, longer downtimes due to calibration lower accuracy – increased risk of non-conforming product, lower cal costs, many cal service providers, longer cal intervals, quicker cal turnaround time Is local technical support available?

ISO/TS 16949: Clause 7.6 Control of devices

Clause 7.6 Calibration of test & measuring equipment Definition: Calibration – comparison against a standard of known higher accuracy, adjustment to agree or noting of the difference. Is a full calibration conducted, or just a partial? Is the instrument calibrated according to the manufacturer’s recommended performance verification procedure to within specification or merely to within a stated uncertainty? Has the instrument been adjusted to get it to within spec? Or have correction factors simply been determined? Have the results been recorded/reported “as found” as well “as left? (before adjustment & after adjustment) Has the instrument been calibrated at the same points as previous calibration, thereby providing a useful history? Has the calibration service provider evaluated the results for conformance to accuracy specification and reported a statement to that effect?

ISO/TS 16949: Clause 7.6 Control of devices

Clause 7.6 Verification/validation of test & measuring equipment Definition: Verification – check that nothing has changed since the calibration Definition: Validation – check to see the instrument is indeed fit for purpose Has the required accuracy been proven by the calibration? (validation) Is there evidence that the instrument accuracy remains within the specification? (If compared to last cal & no adjustment performed, verification) Are measurement checks performed in between calibrations to maintain confidence in the original calibration status? Are the results of these checks recorded?

ISO/TS 16949: Clause 7.6 Control of devices

Clause 7.6 Calibration intervals Definition: Calibration interval – the interval between consecutive calibrations to ensure the required accuracy is maintained. Has a suitable calibration interval been scientifically determined taking into account the following factors? desired accuracy, stability of the instrument, risk analysis of the impact of an instrument NOT within the required accuracy, history of previous calibrations, how often adjustment is required and when last was it performed, (previous cal history destroyed after adjustment), influence of the cal service provider – use of the same calibration method & calibration at the same points Does the calibration service provider determine the cal interval?

ISO/TS 16949: Clause 7.6 Control of devices

ISO/TS 16949: Clause 7.6 Measurement traceability Definition: Measurement traceability – unbroken chain of comparisons between the UUT and the national measurement standard, each step of which has an estimated uncertainty associated with it. Can the last step of the traceability chain be identified/described? (Is the identity of the standard used known and unique?) Was the uncertainty of the standard used for the calibration acceptable? (Was the test uncertainty ratio valid?) Was the calibration service provider SANAS accredited & was the calibration performed within the laboratory’s published accreditation schedule best measurement capability (BMC)? Was the calibration procedure used validated and approved by the manufacturer of the instrument? (Were there any deviations from the manufacturer’s procedure and if so were the changes validated?)

ISO/TS 16949: Clause 7.6 Measurement traceability continued Is there evidence of the competence of the metrologist who performed the calibration? Is there an uncertainty of measurement reported with the results? Does the instrument provide “direct” or “indirect” traceability for the intended measurement/test? CALIBRATION POINT 100 mm ± 1 mm CALIBRATION POINT 200 mm ± 1 mm TRANSFERRED TRACEABILITY MEASUREMENT POINT 150 mm ± ?mm IMPORTED TRACEABILITY

ISO/TS 16949: Clause 7.6 Control of devices

ISO/TS 16949: Clause 7.6 To adjust or not to adjust during calibration Is there record of adjustments having been made during the calibration? If adjustment has been performed, are the results before AND after the adjustment reported? Does the calibration service provider have the technical ability to perform adjustments? (manufacturer’s technical procedure for performing adjustments as sequence, & influences of one adjustment on another, can be critical) Advantages of adjusting to within spec eliminates the need to apply corrections, facilitates ease of use for the operator, instrument can be used in other applications, can reduce the cost of calibration, simplifies the estimation of uncertainty of measurement when using the instrument

ISO/TS 16949: Clause 7.6 To adjust or not to adjust during calibration continued… Disadvantages of adjusting to within spec adjustment immediately invalidates (destroys) previous history, (this can be managed) can increase the cost of calibration (some calibration service providers refuse to perform adjustments or do not have the technical ability – typically a problem if not using the authorised agents as the calibration service provider) the natural drift of the instrument may be disturbed (the instrument may become more stable if left undisturbed for a long period of time)

ISO/TS 16949: Clause 7.6 Control of devices

ISO/TS 16949: Clause 7.6 Calibration status Can the calibration status, (valid or invalid calibration), easily be determined by the user? (How easily can the user inadvertently use an “out-of- calibration” instrument? Who’s responsibility?) On what basis is “Valid Calibration” status assigned to an instrument? Is the calibration expiry date indicated on the calibration label? If partially calibrated, is there indication to the user to prevent accidental usage for another application? Is there a unique link between the calibration status information and the instrument to avoid accidental applicability to the wrong instrument? How will the quality system prevent the use of an “un-calibrated” instrument? If the calibration was conducted by an accredited calibration service provider, does the SANAS logo appear on the associated documentation and calibration label?

ISO/TS 16949: Clause 7.6 SANAS accredited calibration laboratory logos Fig 1: Old Logo Fig 2: New Logo

ISO/TS 16949: Clause 7.6 Control of devices

ISO/TS 16949: Clause 7.6 Calibration integrity Is the calibration validity protected in some way? (Can the user accidentally or intentionally interfere with the calibration?) Who is responsible for ensuring calibration integrity? (Cannot always be the calibration service provider as the user may have access to adjustments) Which methods of calibration integrity protection are employed? Integrity seals over adjustment access points or chassis opening points? Password protection – who keeps the password? If an entire system is calibrated, does the protection apply to the entire system or only one instrument within the system? If in-house integrity seals are used, can they be accessed and applied by the instrument user? How is integrity sealing applied when more than one calibration service provider is used to calibrate the full capability of an instrument and one’s adjustment may affect the other’s calibration?

ISO/TS 16949: Clause 7.6 cont. Control of devices cont.

ISO/TS 16949: Clause 7.6 Evaluating the impact of an “out-of-calibration” instrument Is there communication from the calibration service provider regarding an “out-of-calibration” instrument and any subsequent adjustment? Was the error larger than the required uncertainty of the measurement or test? Was the instrument out of cal at a point directly applicable to the measurement function being used or was it on another range, function? Has a risk profile of the measurement been drawn up? Will an erroneous measurement have major consequences which cannot be rectified? What actions have been implemented to reduce the risk such as:- reducing the calibration interval, performing in-between-calibration verification checks, repeating the measurements using more than one instrument

ISO/TS 16949: Clause 7.6 cont. Control of devices cont.

ISO/TS 16949: Clause 7.6 Calibration records Have the records of calibration been reported on a calibration certificate? If only an in-house document is generated are at least the following recorded:- measurement results, identity of the metrologist & authorised signatory conferring validity to the calibration, estimated uncertainty of the measurements, before and after adjustment results, unique identity of the instrument, description of the calibration procedure/method followed, date & location of the calibration, measurement standards used, conformance statement if required

ISO/TS 16949: Clause 7.6 Verification records Does a verification policy exist defining verification intervals and measurement points? Is verification performed straight after calibration to establish a “signature”? Are the results of verification measurements recorded and analysed? Do they demonstrate the ongoing ability of the instrument to meet the requirements of its intended purpose? What action is implemented when the results obtained after verification differ markedly from the previous results? How and by whom are the verification measurement points chosen?

ISO/TS 16949: Clause 7.6 General instrument records Does each instrument have a history file? Are details of maintenance work recorded in the file? (In particular any possible invalidation of calibration resulting from maintenance) Are costs of maintenance and calibrations recorded to track lifecycle costs?

ISO/TS 16949: Clause 7.6 cont. Control of devices cont.

ISO/TS 16949: Clause 7.6 Software validation Is there evidence that the software is fit for its intended purpose? (proof that the software produces the correct measurement/test results) Has this been “validated” by comparing software generated results with results obtained by means of manual measurements? How is the integrity of the software protected? can a different version be inadvertently installed over the original version? is a copy of the original version kept in a safe place? is a particular version easily uniquely identifiable? (traceability) if changes are made, are these adequately documented and controlled? (can the user of the software make changes?) Is the integrity of the entire measurement process controlled including sampling, measurement and result storage?

ISO/TS 16949: Clause 7.6.1 Measurement system analysis

Definition of Repeatability Definition of Reproducibility ISO/TS 16949: Clause 7.6.1 Statistical analysis of measurement result variation Is the measurement result variability statistically analysed using a method agreed to by the customer Analysis of “repeatability” and “reproducibility” Definition of Repeatability “Closeness of agreement between successive measurement results obtained using the same system under the same conditions” Definition of Reproducibility “Closeness of agreement between successive measurement results obtained using the same system under changing conditions”

ISO/TS 16949: Clause 7.6.1 Estimation of uncertainty of measurement Is an uncertainty of measurement estimated for each measurement/test performed and is it recorded with the result? Is this uncertainty estimated according to the requirements of the ISO document entitled, “Guide to the expression of uncertainty in measurement” or ISO 14253-2, “Guide to the estimation of uncertainty in GPS measurement, in calibration of measuring equipment and in product verification”? Are decisions of product conformance based on ISO 14253-1, “Decision rules for proving conformance or non-conformance with specifications”?

ISO/TS 14253-1 Result of a measurement, y, and result of measurement, complete statement y’

ISO/TS 14253-1 Example of conformance proved

ISO/TS 14253-1 Example of non - conformance proved

ISO/TS 14253-1 Example of neither conformance nor non - conformance proved

ISO/TS 14253-1 Uncertainty of measurement reduces conformance range

ISO/TS 16949: Clause 7.6.2 Calibration/verification records

ISO/TS 16949: Clause 7.6.2 Calibration/verification records Highlighted points from above:- Do the calibration records reflect any engineering changes made to the instrument? (engineering changes can take place without the knowledge of the customer) Does the contract review, between a calibration service provider and the customer, record that special mention must be made in the reporting that an instrument’s performance is suspect for whatever reason?

ISO/TS 16949: Clause 7.6.3 Laboratory requirements.

ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Laboratory scope Is there evidence of a laboratory scope listing the methods or tests/measurements which can be performed by the laboratory? Are the uncertainties, (Best Measurement Capabilities), for these methods or tests/measurements quoted? Is the technically responsible person, who confers validity to the tests/measurement reports, identified?

ISO/TS 16949: Clause 7.6.3 Laboratory requirements.

ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Adequacy of laboratory procedures/methods Are documented test/measurement methods available? Have they been authorised for use in the laboratory by a person with appropriate signing power? Have they been declared technically capable of delivering the desired technical results, i.e. validated, by an appropriately technically competent person? (changes made since validation?) Do they reflect sufficient detail to ensure that the correct results will still be obtained by junior personnel if a senior personnel member is off sick or injured? Do the methods address the entire measurement process including sampling, performing of the measurements, capturing and storing of results, estimation of uncertainty of measurement as well as reporting the result? Are methods reviewed for ongoing relevance at planned intervals?

ISO/TS 16949: Clause 7.6.3 Laboratory requirements.

ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Competence of laboratory personnel Since the responsibility of assessing whether or not the metrologist is competent is an internal one:- Have criteria for the required competence been drawn up? Is there evidence that the metrologist’s competence has been assessed against these criteria? Was the metrologist’s competence assessed by means of physically witnessing a measurement/test AND comparing the results obtained with results of known accuracy? To what extent have audit sample measurements, or participation in proficiency testing schemes, provided proof of acceptable measurement capability? Does the metrologist have a training file containing records of relevant training and experience? Is there sufficient depth of staff and is a designated “stand-in” identified in the quality system?

ISO/TS 16949: Clause 7.6.3 Laboratory requirements.

ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Testing of the product Are suitable test methods available for testing of the product? Does the laboratory have access to the required technical specifications in order to declare conformance or not? Are the uncertainties of measurement in the test adequately estimated? Do decisions of conformance or non-conformance take the estimated uncertainty of measurement into account? Has an impact analysis been conducted to evaluate the risk of rejecting product which actually passes spec, and accepting product which fails spec?

ISO/TS 16949: Clause 7.6 Laboratory requirements.

ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Capability to perform the services correctly to the relevant process standard Are the latest revision applicable process standards available? Are revisions of process standards controlled? Do the relevant process standards apply in their totality, and if not, are deviations clearly identified and authorised by an appropriately competent person? Have all deviations from the process standard been validated where applicable? (e.g. lab using different equipment to perform the same test/measurement)

ISO/TS 16949: Clause 7.6 Laboratory requirements.

ISO/TS 16949: Clause 7.6.3 Internal Laboratory requirements Review of the related records Is there evidence of ongoing planned review of all relevant laboratory documentation? Is there evidence of methods being re-validated after a modification? Are instruments routinely calibrated according to a calibration schedule on which the calibration intervals have been scientifically determined?

ISO/TS 16949: Clause 7.6 Laboratory requirements.

ISO/TS 16949: Clause 7.6.3 Laboratory requirements.

ISO/TS 16949: Clause 7.6.3 External Laboratory requirements Acceptability to the customer If the external laboratory is NOT SANAS accredited, does it meet all the requirements of ISO/IEC 17025? Is there evidence of the customer having audited the external lab to confirm this to be the case? If it is SANAS accredited and, or, meets the requirements of ISO/IEC 17025, does it meet the technical requirements of the customer? (Does the scope of the laboratory cover the measurements/tests required by the customer) SANAS accreditation does not by default mean the laboratory technically meets the requirement of the customer!! Is there evidence that although the laboratory is SANAS accredited, it has performed tests/measurements outside of its accredited scope unbeknown to the customer?

ISO/TS 16949: Clause 8 Measurement to prove conformity of product.

ISO/TS 16949: Clause 8 Measurement to prove conformity of product continued.