1. An Evaluation of the EPA Quality Assurance Annual Report and Work Plan from the Perspective of a Traditional Cost of Quality Model Lora S. Johnson, Director of Quality Assurance Gary J. Foley, Laboratory Director USEPA, National Exposure Research Laboratory 18th Annual National Conference on Managing Quality Systems for Environmental Programs Cincinnati, Ohio April 16, 1999 NERL

2. OVERVIEW REPORTING ISSUES COST OF QUALITY CATEGORIES ELEMENTS OF THE QUALITY ASSURANCE ANNUAL REPORT AND WORKPLAN (QAARWP) QAARWP VS. COST OF QUALITY CATEGORIES ORD APPROACH FOR FY98/99 A POTENTIAL SOURCE OF COST DATA IDEAS FOR REFINING THE QAARWP NERL

3. REPORTING ISSUES Managers need information so that appropriate resources can be allocated to quality assurance activities. What information is needed? The Agency uses the Quality Assurance Annual Report and Work Plan (QAARWP) to provide this information. How does the EPA approach compare to a traditional private sector assessment of the “cost of quality?” How should it be measured? How should it be collected? How should it be represented? NERL

4. COST OF QUALITY CATEGORIES The costs of all activities specifically designed to prevent poor quality in products and services. Quality assurance project plans Training in quality QA review PREVENTION The costs associated with measuring, evaluating, or auditing products or services to assure conformance to quality standards and performance requirements. Calibration Performance evaluation samples Technical system audits APPRAISAL NERL

5. COST OF QUALITY CATEGORIES The costs resulting from products or services not conforming to requirements or customer/ user needs. FAILURE Costs that occur prior to delivery Reworking publications Too many signatures Failed Experiments INTERNAL FAILURE Costs that occur after delivery ??? EXTERNAL FAILURE NERL

6. ELEMENTS OF THE QAARWP QA Annual Report Total FTE’s Federal FTE’s - QA Management Federal FTE’s - QA Technical Support Contract FTE’s - QA Direct Support Contract FTE’s - QA Nontechnical Support Contract $ $ for Travel-Oversight Activities $ for Travel/Training about QA Resources adequate? Quality Management Resources FROM THE EPA QUALITY MANUAL, 7/98 Method used to assess training needs Course names and suppliers Attendees Course goals met? Training NERL

7. ELEMENTS OF THE QAARWP QA Annual Report (continued) Innovative Practices Technical Assessments Technical Assistance New/revised QA Guidance Publications Awards Management Accomplishments FROM THE EPA QUALITY MANUAL, 7/98 MSR’s - Internal MSR’s - External Management Assessments Work Plan (same elements as above)Quality Management Resources Activities NERL

8. QAARWP VS. COST OF QUALITY A, P Quality Management Resources P Training A, P Management Accomplishments A Management Assessments NERL CATEGORIES A = Appraisal Cost P = Prevention Cost

9. ORD APPROACH FOR FY98/99 ACCOUNT FOR RESOURCES IN A SIMILAR WAY TO PROMOTE COMPARABILITY IN THE ORD RESOURCE SUMMARY TABLE GOAL: DIRECTORS OF QA FOR NERL, NHEERL, NRMRL MET WITH QAD REPRESENTATIVE VIA TELECONFERENCE Quality Management Resources = Sum of the Division QA Managers and the Director’s of QA i.e. staff listed on EPA’s QA Community list, adjusted for percentage of time spent on the QA function METHOD: EXAMPLE: GROUP AGREED TO ACCOUNTING RULES FOR ELEMENTS IN QUALITY MANAGEMENT RESOURCES CATEGORY NERL

10. A POTENTIAL SOURCE OF COST DATA Vice-president’s “National Performance Review” seeks to “build a government that works better and costs less.” Customer service standards are developed across the Federal government. FEDERAL GOVERNMENT CUSTOMER SERVICE INITIATIVES Congress passes the Government Performance and Results Act of 1993 that seeks to improve Federal program effectiveness and public accountability by promoting a new focus on results, service quality, and customer satisfaction. ORD’s products are knowledge based therefore “failures” are hard to measure. Customer surveys are a potential tool for measuring both success and failure. Including customer survey data in the QAARWP would close the loop on cost of quality items. NERL

11. IDEAS FOR REFINING THE QAARWP Need to compare apples to applesBetter Definition of Terms: Need some measure of “cost of failure” Include customer survey data: NERL Is information helpful for making management decisions? Examine use of the QAARWP: Present data graphically QA FTEs/FTEs QA FTEs/$ QA FTEs/ # APMs Index QA resources to some measure of productivity:

12. HYPOTHETICAL SCENARIOS NERL QA Effect Positive Why are we bothering? QA Hurts QA Index Customer Satisfaction Time Customer Satisfaction Time QA Index Customer Satisfaction Time QA Index COMMENTS

13. CONCLUSION The QAARWP is a significant element of the EPA quality system. To make it a more useful management tool, some estimate of “failure” needs to be included. NERL Recent efforts in the area of customer satisfaction may provide a surrogate measure of external failure at least to the EPA R&D laboratory community.