Standard 4 Breakout Session Inspection Audit Program Russell Lilly Manager, Manufactured Food Program Missouri Department of Health and Senior Services Wednesday, March 12, :00 – 10:00 Elm Fork I Patrick Kennelly Chief, Food Safety Section California Department of Public Health Peter Haase Director, Bureau of Food Safety and Inspection Wisconsin Department of Agriculture, Trade and Consumer Protection Skya Murphy Program and Policy Analyst Wisconsin Department of Agriculture, Trade and Consumer Protection Angela Kohls Standards Implementation Staff FDA ORA, Office of Partnerships

Agenda Strengthening Compliance and Enforcement Welcome and Introductions Current Status of Standard 4 Implementation Presentations on Implementing Standard 4 Q & A/Group Discussion Summary and Challenge

Housekeeping Honor time schedule Stay on topic Get input from everyone Be willing to share Respect different program cultures Do not reference specific audits Be creative, ask questions – parking lot Facilitators – responsible for process Participants – responsible for content

Session Objectives Generate a list of resources, best practices, lessons learned and challenges/recommendations for developing, implementing and monitoring an inspection protocol. (Group Exercise Instructions) 1. Listen to presentations 2.Discuss and list suggestions on flip charts 3.Share your group’s suggestions with room

Standard Four

Breakout Session Focus Elements 4.2b &c, 4.3b & c, and 4.4b & c Implementation of the 3 Types of Audits – Field Inspection – Inspection Report – Sampling

Standard 4: Field Inspection Audits Patrick Kennelly, Chief Food Safety Section California Department of Public Health March 12, 2014

Field Inspection Audits Quality assurance review to assess the effectiveness of its inspection program. – 2 audits of each inspector every 36 months – Conducted by a qualified auditor – Documented on Audit form that is equivalent to the FDA Contract Audit form – Need an 80% audit rating to be considered a successful audit

Field Inspection Audits Then and Now – Better Defined Process – Uniform Documentation – Ability to Compare Ratings Across the Program

Field Inspection Audits

II. INSPECTION OBSERVATIONS AND PERFORMANCE (Continued) 6.DID THE INSPECTOR DEMONSTRATE THE ABILITY TO DISTINGUISH BETWEEN SIGNIFICANT VERSUS INSIGNIFICANT OBSERVATIONS AND ISOLATED INCIDENTS VERSUS TRENDS? ☐ Acceptable ☐ Needs Improvement COMMENTS (required for Needs Improvement) 7.DID THE INSPECTOR REVIEW AND EVALUATE THE APPROPRIATE RECORDS AND PROCEDURES FOR THIS ESTABLISHMENT’S OPERATION AND EFFECTIVELY APPLY THE INFORMATION OBTAINED FROM THIS REVIEW? ☐ Acceptable ☐ Needs Improvement COMMENTS (required for Needs Improvement) 8.DID THE INSPECTOR COLLECT ADEQUATE EVIDENCE AND DOCUMENTATION IN ACCORDANCE WITH STATE PROCEDURES GIVEN THE NATURE OF THE INSPECTIONAL FINDINGS? ☐ Acceptable ☐ Needs Improvement COMMENTS (required for Needs Improvement)

Field Inspection Audits Things to Consider – How do you select facilities for staff audits? – What constitutes a “qualified auditor”? – Do you want to vary or alternate auditors for each staff? – How do you address individual “Needs Improvement” Ratings?

Field Inspection Audits Scoring across the spreadsheet identifies areas of “Needs Improvement” ratings within a rated category. – Need 80% “Acceptable” ratings within each rated category across the program to be in conformance. – How do you address Non-Conformance ratings?

Field Inspection Audits Benefits – More Knowledgeable Workforce – Improved Inspection Procedures – Identifying Training Gaps – Findings Drive Quality Improvement

Contact Information Patrick Kennelly, Chief Food Safety Section California Department of Public Health (916)

Inspection Report Audits Russell Lilly Manufactured food program manager Missouri Department of Health March 12, :00 AM

Because… I had three inspectors and reviewed every inspection report It is wildly prescriptive – At least half of the widgets don’t apply If the shoe doesn’t fit it is uncomfortable

Our situation changed 18% of 22 inspectors A quality assurance program becomes necessary and valuable

We took the money… We owe an honest effort to comply with the standard

We did it… My best advice – Hire someone smart and assign it to them

Was it worth it? I think so – We have sent a number of “corrective s” Mark this here not there Be sure to put your “time out” on the form Turn in your reports on time

I would still like To have a quality system designed from the ground up to fit our program

Contact Information Russell Lilly Manufactured food program manager Missouri Department of Health

Steps to Implementing Standard 4 Sample Collection Record Audit Peter Haase Director, Bureau of Food Safety and Inspection Skya Murphy Program and Policy Analyst Wisconsin Department of Agriculture, Trade and Consumer Protection

Sample Collection Record Audit Outline Introduction and Challenges Steps to customization Conclusion Documents on flash drive

Intro: Wisconsin Department of Ag Trade & Consumer Protection Enrolled in MFRPS in 2007 Challenges: Full Standard 4 Implementation Planned for 2015 – Phase III Contract Audit since 2011 Currently Piloting Non-Contract Field Inspection Audit Inspection Report Audit 2013 SCR Audit 2014

Intro: Wisconsin Department of Ag Trade & Consumer Protection Current Surveillance Sampling Program for Manufactured Food Establishments >Sampling at 2,400 establishments annually: – 2 product samples and – 5-10 environmental swabs at each routine inspection

Intro: Wisconsin Department of Ag Trade & Consumer Protection Standard 10 Full Conformance – ISO Certified since 2007 Sampling Policies and Procedures Two Sample Collection Record Forms, Food and Dairy

Challenges of Auditing our Sample Collection Records: Step 1 Test drive of Appendix 4.7.

Challenges of Auditing our Sample Collection Record (SCR): Step 1 Results of Test drive of Appendix 4.7. – Findings: 4.7 is more easily managed in table format FDA Criteria: – Some not applicable or redundant for our reports – Very general: guidance specific to our SCRs required

Audited against our Policy Added guidance specific to our Sample Collection Record form and procedures

Audited against our SCRs

Customized 4.7 Form: Added guidance specific to our Sample Collection Record form and procedures

Piloted new form and guidance by auditing 71 SCR’s from Calendar 2012 Results: – Opportunities for improvement: Use of hash marks and down arrows, form completeness Redundant criteria: – SCR and sample always together

Challenges of Auditing our Sample Collection Records (SCR): Problem detected but not an audit criterion: Use of Correct Form Substituted alternate criteria: Use of correct form

Challenges of Auditing our Sample Collection Records (SCR): For redundant criteria, combined and substituted alternate criteria

Sample Selection (included in SOP for Standard 4) 75 SCR’s chosen per calendar year from MF establishments for auditing: – at least 1 per sampler, – random with regard to date, establishment

SOP for Standard 4 Cont. Entering and tabulating results: Excel Spreadsheet 1 for acceptable, 0 for needs improvement

SOP for Standard 4 Cont. SCR Audit completed in March for previous calendar year Needs Improvement Factors (<80%) become training topics at staff meetings Other improvements can be made immediately upon discovery: – Already identified from SCR 2013 audit: “Foodborne Outbreak” reason for sampling on new Dairy SCR

Conclusion – Adapting 4.7 to our Sample Collection Record Meaningful results for quality improvement – Documents on AFDO-MFRPA website: Lab SOP #2060-Receipt and Integrity of Samples SCR Policy and Procedures SCR Forms (Food and Dairy) Our Audit Form 4.7 and Data Entry/Calculations Sheet Description of sample selection and procedures for Standard 4 including Sample Collection Record Audits

Contact Information Skya Murphy Program and Policy Analyst Wisconsin Department of Agriculture, Trade and Consumer Protection

Contact Information Patrick Kennelly, Chief Food Safety Section California Department of Public Health Russell Lilly Manufactured Food Program Manager Missouri Department of Health Skya Murphy Program and Policy Analyst Wisconsin Department of Agriculture, Trade and Consumer Protection Peter Haase, Director Bureau of Food Safety and Inspection Wisconsin Department of Agriculture, Trade and Consumer Protection