Learning and Development Evidence Collection Planning January 2015

Objectives Building on the FVO-Auditing training course, to improve the audit preparation skills of FVO auditors through: Application of a systemic and systematic approach to audit-preparation, Applying the FVO SOP’s and WI’s, and Building a common understanding of audit preparation across the office. 2

Programme for the day Need for evidence Audit preparation: Breaking down the objective Audit preparation: Evidence Collection Planning Assessing an Evidence Collection Plan 3

Your expectations Quick round: What would you most like to take away from this course 4

The audit questions Is there a system in place? Can it work? Does it work? If not, why? So what? 5

Objective(s) Conclusions Findings Evidence Overall conclusion Questions (1 st level) Questions (2 nd level) Facts The Big Question Good preparation is … 6

1 st Assignment: need for evidence (30 min) In groups: Given the objective and legal basis Determine which evidence is needed If you have all the evidence and it’s all positive, what conclusion could you draw on this evidence, be as specific as you can Write your results on a flip chart. 7

Breaking down the objective Three levels: 1.Effectiveness: from the objective and scope, determine which (sub) processes are critical to the effectiveness of the system. Also determine which parts of the legal basis are applicable per (sub) process. 2.Compliance: determine which elements need to support the critical processes and which specific part of the regulations is applicable for each element. 3.Proof: last but not least determine what evidence will be needed to prove the existence and compliance of the elements. 8

Breaking down the objective, example Animal identification CA ControlsControl plan Risk based targeting Procedures for controls Et cetera Controls + results Et cetera FBO compliance 9

What is evidence? Evidence: “Records, statement of fact or other information, obtained through observation, measurement, test or other means, which are verifiable.” Evidence = information! 10

ECP-table or mind-map or … 11

Template for the final result … 12 Legal basis Reasons for the Big question Objective: the Big Question of effectiveness Compliance and effectiveness audit question (sub) section Compliance audit question (finding) Information question Corroboration question Detailed legal criteria Compliance audit question (finding) Information question Corroboration question Detailed legal criteria Legal articles

2 nd Assignment: breaking down the objective (30 min) In groups: Break down the objective with the help of lower level questions, what information do you need to gather Prioritize at each level Until you find yourself at the level where you will collect your evidence, the level of proof If you have all the evidence and it’s all positive, what conclusion could you draw now, be as specific as you can. Write your results on a flip chart. 13

What is persuasive evidence? Verified, i.e. corroborated Based on samples, consistent, sufficient, reliable and appropriate (the degree to which evidence can be considered trustworthy): – Relevance – Independence – Effectiveness of auditee's internal controls – Auditor direct knowledge – Qualifications – individual is a qualified source – Degree of objectivity – Timeliness 14

What is sufficient evidence? Examples of sampling standards: ISO 2859, applicable to inspection by attributes. CAC/GL , applicable to CODEX commodities. IAF MD1-2007, applicable to Certification of Multiple Sites Based on Sampling. Finally, it depends on the audience, you have to prove your point 15

Evidence: as much as needed and no more than necessary … 16 Competent Authority Suspicious stakeholder Compliance Non- compliance

Further attributes of evidence collection planning Sources: where are we going to get the evidence? Evidence collection methods: how are we going to get it? Evidence analysis methods: what will we do once we've got it? You have to bring back the evidence to answer up to the objective! 17

8 types of evidence Information obtained from: Documentation Physical examination Confirmation (cross-checking) Analytical procedures Inquiries to the auditee (both documentary and verbal i.e. interview) Re-calculation Re-performance Observation 18

Example of persuasive and sufficient evidence Individual animals identified: Sufficient: extremes, sampling Types: documentation, inquiries, (re)calculation, observation Sources: CA, farmer/industry representative, auditor, persons with overview and without conflict of interest Collection: observation, interviews Analysis: RCA, (re)calculation 19

3 rd Assignment: evidence attributes (30 min) In groups, to allow yourself to be targeted, flexible and to support verification and corroboration: For each lowest level audit question Determine source(s), collection and analysis methods for the evidence that will supply the answers Mind that the evidence is appropriate and sufficient so as to prove your point. Write your results on a flip chart. 20

Objective(s) Conclusions Findings Evidence Overall conclusion Questions (1 st level) Questions (2 nd level) Facts The Big Question Good preparation is half the work 21 Evidence Collection Planning

The steps in audit preparation Prepare/customise audit plan (SOP WI 003, 2.1) Research and analysis, consultation (WI 003, 2.2) Prepare/customise pre-audit questionnaire (SOP WI 003, 2.3) Consult Laboratories Guideline for FVO audits (WI 003, 2.3.5) Prepare/customise aide memoire(s) (WI 003, 2.4) 22

4 th Assignment: Assessing the ECP (40 min) In groups: Assess if correctly have been identified: The critical processes, sufficient to ensure the effectiveness of the system The elements needed to support the critical processes and required by legislation The evidence needed to proof the compliance or non-compliance of the elements Assess if there are any shortcomings. Which (sets of) questions are most important? Write your results on a flip chart. 23

Key messages Breaking down the objective: Effectiveness level Compliance level The level of proof Evidence: types, persuasiveness Sources, Collection and Analysis methods ECP table: a tool Good preparation is half the work! 24

Personal objectives, make a note for yourself: What’s your quick win? What will you try to implement first? For you For your Unit 25