QUALITY OF EVIDENCE FRCC Compliance Workshop September/October 2008

2 Evidence is essential The Registered Entity has the burden to prove compliance through presentation of quality evidence to a standard or requirement. “Provide Quality Evidence ”

3 Objective Objective – What is quality evidence? –Define evidence and the types of evidence. –Describe what constitutes quality evidence. –Define appropriate and sufficient evidence. –Provide a better understanding of evidence and its role in compliance monitoring activities.

4 Definition of Evidence Evidence is: – Data or information on which factual statements can be based. – A collection of relevant and sufficient information offered to verify a presented fact. – Obtained through examining and analyzing records, observing events, or conducting interviews.

5 Presenting Your Evidence Auditors gather evidence to draw conclusions on findings in a logical method based upon factual evidence. Facts are to be valid and relevant for the point needed to be proven. –A fact is something that can be objectively verified, is real and does exists. Facts must be verifiable. –A verifiable fact is one whose data or information can be substantiated by comparison, investigation or confirmation. Entities should consider presenting and submitting their evidence in the same manner.

6 Types of Evidence Evidence can be provided as: –Physical evidence –Direct observations –Corroborating evidence Corroborating evidence is evidence that tends to support a proposition that is already supported by some evidence. (Wikipedia 9/11/2008) To strengthen or support with other evidence; make more certain (the Free dictionary 9/11/2008)

7 Types of Evidence Physical Evidence is direct inspection of activities of people, property, or events: –Some examples include: Schedules Charts Maps Graphs One-lines Manuals Procedures and Policies Plans

8 Types of Evidence Documentation: –Physical forms of evidence which already exists –Can be both external or internal –Some examples include: Letters Contracts Assessments Agreements Spreadsheets Database Extracts s Studies

9 Types of Evidence Direct Observations: –What an auditor sees during your review. –Strengthened through corroboration with other evidence. –By itself is almost always unreliable. –Is achieved by monitoring of personnel, facilities, equipment, and tools.

10 Types of Evidence Corroborating evidence is: –Additional information from interviews, logs, etc provides clarity to make the appropriate compliance decision. –Used to establish the appropriateness of evidence. –Tests the reliability of evidence and provides confirmation. –Some examples include: Reliability Coordinator or Neighboring questionnaires Voice recordings Logs Interviews

11 An Example of Corroborating Evidence The auditor requests evidence that a Registered Entity has followed all directives from the Reliability Coordinator –The Registered Entity states it has not received any directives: Response is neither sufficient, nor adequate –Testimonial –Low quality –May not withstand scrutiny Corroborating evidence could be an from the Reliability Coordinator stating the Reliability Coordinator did: –Not send any directives to the Registered Entity during the specified time frame, or –Did send a directive and it was followed

12 Quality Evidence Quality evidence is evaluated by its: –Appropriateness relevant, valid and reliable to support findings. –Sufficiency enough information to lead another person to the same conclusion. –Quality high quality evidence to be used as proof of findings.

13 Appropriateness of Evidence Quality evidence must be appropriate: –Relevant to the reliability standard requirements: Is there enough to persuade the auditor his findings are reasonable? Is it logically related to the requirement? –Valid as a document: Is the evidence based upon sound reasoning and accurate information? –Reliable information: Can the evidence be substantiated when tested?

14 Example of Appropriate Evidence An agreement is offered as evidence. The auditor will seek to determine: –Is this the original document? –Does the agreement contain the Registered Entity letterhead, approval signature and date, revision date, etc? –Does the agreement address the subject of the requirement? –Is this version the current version?

15 Sufficient Evidence Quality evidence must be sufficient: –Is there enough evidence to lead an auditor to the same findings that you have reached. –Strong evidence may be better than a large volume of weak evidence.

16 Example of Sufficient Evidence The auditor is provided three documents to determine if a procedure exists and is coordinated with neighboring entities: –A procedure – s –Previous version of procedure The above evidence is sufficient as: –A valid procedure is provided which addresses all sub- requirements even if they do not apply to the audited entity. – s provide evidence the procedure was sent, received and confirmed as coordinated. –Previous version of procedure shows the procedure existed for the audit period.

17 Quality Evidence The evidence must be of the highest quality to be used as proof of findings. Quality evidence should contain: –Documents title –Purpose –Date approved –Revision level –Effective date –Authorizing signatures

18 Testimonial Evidence Written or oral statements in response to inquiries or interviews. –Examples are: Reliability Coordinator or Neighboring questionnaires Interviews Letters sent for confirmation

19 Computer Processed Evidence Computer evidence can be provided if: –The evidence can be determined to be sufficient and appropriate. Is the information considered quality evidence? Can the evidence be validated from electronic logs to confirm creation? Some examples include: –Printouts (titled, dated) –Screen shots –Logs (titled, dated) –Study results (titled, dated)

20 Professional Judgment All evidence reviewed requires some level of professional judgment. “Professional judgment of the auditors is based on their: –Knowledge –Skills –Experiences –Good Faith Application –Integrity

21 Final Thoughts Some things to remember: –Who you are trying to convince, the auditors. –Compliance is based upon the evidence provided. – Would you objectively make a determination of compliance or possible violation?

22 Conclusion At the end of the review, enough quality evidence must have been provided to: Soundly substantiate the teams findings and Support your expected findings

23 “Provide Quality Evidence ”

24 Questions ?