NSAA Peer Review Program: What to Know Before You Go! National State Auditors Association January 21,

2 Opening Remarks MODERATOR R. Kinney Poynter Executive Director, NASACT SPEAKER John Buyce Director of State Audits, Office of the State Comptroller (NY) SPEAKER William (Brad) Blake Chief Auditor, Center for Audit Excellence, Office of State Auditor (OH) SPEAKER Greg Fugate Performance Audit Manager, Office of the State Auditor (CO) SPEAKER Staci Henshaw Deputy Auditor, Virginia Office of the Auditor of Public Accounts (VA)

Agenda Overview, Objectives, and General Considerations Organization and Qualifications of the Review Team Performing Peer Reviews Finalizing the Peer Review Tips & Insights 3

Overview, Objectives, and General Considerations

Overview Purpose of peer review program – to provide independent assessments of state audit organizations to determine whether they have an adequately designed internal quality control system and are in compliance with that system 5

Overview Yellow Book began requiring peer reviews in 1989 NSAA program started in 1985 Currently, 54 eligible organizations participate in the program 6

Overview – Role of Committee Peer Review Committee has overall responsibility for the program –Provide guidance in the form of policies and procedures –Resolve potential disputes that may arise in the review process and ensure consistency of reviews –Coordinate with other groups to ensure the adequacy of the review process 7

Overview – Role of NASACT Administer the review process on a daily basis Coordinate and assign review teams Train review team members Update policies and procedures for review by the Peer Review Committee 8

Objectives of the Review Evaluate whether the State Audit Organization’s system of quality control is… –suitably designed, adequately documented and communicated; and –sufficiently complied with to provide reasonable assurance of compliance with government auditing standards 9

Objectives of the Review Upon completion of the peer review, the peer review team issues a report Types of peer review reports –Pass –Pass with deficiencies –Fail 10

General Considerations Maintain confidentiality of state’s working papers and peer review work papers Be independent of state audit organization being reviewed Possess adequate knowledge and proficiency to perform review 11

Organization and Qualifications of the Review Team

Organization of Review Team Concurring reviewer –Most senior member of team Team leader –Overall responsibility for planning and performing the review Team member –Responsible for performing tasks assigned by team leader 13

Organization of Review Team Team leader and concurring reviewer identified 4-5 months prior to the actual date of the review Team members identified 2-3 months prior to the actual date of the review 14

Qualifications – Concurring Reviewer Must have served as team leader on a least one review under NSAA’s program Must have been recommended by former concurring reviewer 15

Qualifications – Team Leader Must have served as team member on a least one review under NSAA’s program Must have been recommended by former team leader and concurring reviewer 16

Qualifications – Team Member Must be recommended by audit organization head Must be in a supervisory or managerial role Must review audit documentation as part of job responsibilities Minimum of three years in supervisory role 17

Qualifications for Review Team Member – Other Issues Ensuring a proper match –NASACT tailors team skills and experience to the SAO’s needs Type of work performed (Financial, Performance, Attest) Size of the SAO and number of audits Government Audit Quality Center (AICPA- GACQ) membership Federal reviewer for Single Audit Usually up to six reviewers selected 18

Performing the Peer Review

Preliminary phase Field work phase Completion phase 20

Preliminary Phase Understanding the Peer Review Program Obtaining/reviewing necessary SAO info including prior work papers Determining scope of review Sending audit staff questionnaire Selecting engagements Finalizing planning 21

Documents Used in a Peer Review Audit Organization Questionnaire Audit Staff Questionnaire Audit Organization’s Policies and Procedures and Review Guide Guide for Review of Audit Engagements Matters for Further Consideration (MFC) Findings for Further Consideration (FFC) Conclusions 22

Team Preparation: Understanding the Review Process Ensure each team member obtains an understanding of the process –NSAA External Peer Review Manual –Questions and Answers for All Team Members –Administrative Matters Timing of review procedures Travel arrangements General expectations 23

Team Leaders: Review of Prior Work Papers Issues identified in prior peer review report –Prior peer review rating Read notes from exit conference Review other issues identified (MFCs and FFCs) 24

Audit Organization Questionnaire Completed by the SAO and provided to the Team Leader approximately 12 weeks prior to the review Provides the population of audits and audit hours within the review period Provides information about the state office’s staff, structure, and quality control system 25

Determining the Scope of the Review Determine whether the SAO performs –Financial Audits –Attestation Engagements –Performance Audits Engagements the SAO has stated to have been performed in accordance with government auditing standards are subject to the review 26

Determining the Scope of the Review Review should cover a current period of one year to be mutually agreed upon by the SAO and the review team Covers the quality control policies and procedures in effect and compliance for the year under review Includes reports issued during the year under review, or immediately thereafter if the work was substantially completed during the period 27

Determining the Scope of the Review The scope of the review does not cover –Non-GAGAS engagements –Audits done by others (e.g., contracted) –Audit efficiency –State statute compliance –Administrative aspects, unless directly required to satisfy applicable professional standards (e.g., Independence, CPE) 28

Other Administrative Matters Team Leader –Work Program –Prepare and send the engagement letter –Work with the SAO to establish deadlines for receiving key documents –Initial communication with Concurring Reviewer and Team Members NASACT –Get signed contract to Team Leader 29

Pre-Visit Work Team Leader –Select Audit Engagements for Review –Audit Staff Questionnaire –Preliminary Site Visit –Policy & Procedure Review –Finalizing the Review Plan –Administrative Logistics 30

Selecting Audit Engagements for Review Based on listing provided by the SAO Factors to consider include: –Spread across various supervisors, regions, etc. –Spread across the review period –Different types of engagements (Cash Basis vs GAAP, Single Audits, CAFRs, Performance, Attest) Select one “surprise” audit 31

Other Sample Selections Select a sample of audit staff to complete the Audit Staff Questionnaire While onsite, select a sample of audit staff for verification of CPE records and independence documentation 32

Audit Staff Questionnaire Work with the SAO to administer the questionnaire about 8 weeks prior to the review –Responses submitted directly to the Team Leader about 6 weeks before the review Team Leader compiles the responses and communicates a summary to the team Evaluate responses for potential areas to concentrate review efforts 33

Preliminary Site Visit and Other Considerations Team Leader consults with the Concurring Reviewer to determine if a preliminary site visit is needed –Very rare –If one occurs, the preliminary site visit is 4-6 weeks in advance Team Leader has option to arrive on site at the SAO a few days prior to the rest of the team members 34

Team Members: Evaluation of the SAO’s Policies and Procedures Policies and Procedures Checklist –Separate checklists for financial audits, performance audits, and attestation engagements –Completed by the SAO and provided to the Team Leader along with the SAO’s policy and procedure manual, usually about 8 weeks prior to the review 35

Policies and Procedures Checklist 36

Policies and Procedures Checklist Team Leader assigns each team member a section or sections of the policies and procedures to review Assignments are made to the team members approximately 8 weeks prior to the review Team Members are responsible for completing this work before arriving on site for the review 37

Finalizing the Review Plan Team Leader discusses plan for the review with the Concurring Reviewer Team Leader informs the SAO about sampled audit engagements, usually about 4-6 weeks prior to the review Team Leader only informs the SAO about the sampled “surprise audit” when the team arrives on site 38

Sampled Audits/Engagements Team Leader assigns each team member one or more audits/ engagements to review Team Members are responsible for completing some work before arriving on site for the review –Download and review reports –Complete reporting sections of the Audit/ Engagement Review Checklist 39

Audit/Engagement Review Guide 40

Administrative Logistics Team Leader will communicate about administrative logistics in the 1-2 weeks leading up to the review –Confirm travel, car, and hotel –Set time/place for initial meeting –Work plan/schedule for the review –Dress code and other information If something is unclear, be sure to ask! 41

Field Work Phase Complete the study and evaluation of QC policies and procedures Review compliance with policies and procedures Identify matters, findings, deficiencies, and significant deficiencies Aggregate and evaluate matters Form conclusion on type of report to issue Communicate conclusions at exit conference 42

Overall Fieldwork Schedule Five Day Reviews –Generally run Monday - Friday Ten Day Reviews –May run Monday to Wednesday or Wednesday to Friday –Allows a few more days for review of sampled audits 43

Time Frames for Large Reviews 44

Time Frames for Small Reviews 45

Arrival – Day Before the Review Begins Team Leaders often hold a team meeting at the hotel in the evening –Answer any questions, logistics, etc. –Results of policy & procedure reviews Team Members –Discuss policies noted in their assigned area and their reviews of reports Some teams do a conference call during the week before the review 46

Day 1 – Let’s Get Started Entrance Conference Compliance testing for Policies & Procedures Get oriented to the audit workpapers, software, network, etc. Begin detailed review of first sampled audits 47

Team Leader – Day 1 Run the Entrance Conference Discuss any outstanding issues with SAO officials Compliance testing for P&P Get organized Assist Team Members in starting their audit reviews Status meeting at day’s end 48

Team Members – Audit/Engagement Reviews Standard review guide for each type of audit/engagement –Financial, Performance, Attest –Covers all phases of the audit/engagement from planning through reporting –Cross referenced to individual audit standards, AU citations, etc. Use to test compliance with procedures by teams 49

Audit/Engagement Review Guide 50

Audit/Engagement Reviews Complete checklist for each audit/engagement –Seek clarification from audit teams –Start your second review if waiting for information from others “NO” answers generate an MFC – given to Team Leader 51

MFC Form 52

MFC Form “Matters” generally will originate from “no” answers on checklist (either with design of, or compliance with, system of quality control) Can be cleared, discussed verbally with the audit organization, or carried forward to the Conclusions document 53

Day 2 – More Audit Reviews Team Leader –Answer questions, provide guidance, ensure consistency –Organize MFCs, review completed work, test items to workpapers –Get ready for the Concurring Reviewer to arrive Team Members –Discuss issues with audit teams –In most cases, you should be on your second audit in the afternoon 54

Day 3 – Start Wrapping it up Team Leader –Brief the Concurring Reviewer on progress and issues –Continue organizing and grouping MFCs –Assist Team Members in completing audit reviews 55

Day 3 – Start Wrapping it up Team Members –Continue completing review guides and MFCs –Second audits should be all but done –If there’s a third, it should be well underway in the afternoon Prioritize the areas you want to review Key on planning, execution, documentation and common issues 56

Day 3 – Start Wrapping it up End of the Day –Status meeting Initial discussion of issues that may warrant additional work or be potential Findings –Should have a sense of the overall rating Plan out work that needs to be completed and when –Reassign work and assist each other as necessary 57

Day 4 – Pull it all together Team Leader –Organize MFCs for discussion –Manage completion of reviews by mid- day –Lead team discussion to come to a consensus on disposition of MFCs, Findings and rating Contact and consult with NSAA if necessary 58

Day 4 – Pull it all together Team Members –Complete all review guides and MFCs –Assist others to tie up loose ends Afternoon – Team Discussion –Review all MFCs and discuss the ones you prepared –Work to consensus on items to elevate to Findings based on significance and pervasiveness 59

Day 4 – End of the Day Decide on form of report and issues to discuss at exit conference –Complete the relevant Conclusion documents and finalize a draft of the report –Consult with NSAA as appropriate if report rating is other than “pass” Brief officials about the general conclusions of the report and any findings 60

How do you decide what type of report to issue? Based on professional judgment considering nature, causes, pattern and pervasiveness of matters and their relative importance to the organization’s system of quality control taken as a whole 61

Forms Used in Documenting Results Matters for Further Consideration (MFC) Conclusions Document Findings for Further Consideration (FFC) 62

Reporting Matrix Section II of the Peer Review Manual includes a reporting matrix to provide guidance on various reporting considerations when evaluating results 63

Reporting Matrix 64

Types of Peer Review Reports Pass Pass with Deficiencies Fail 65

Pass Audit organization’s system of quality control has been suitably designed and complied with to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects 66

Pass Design –Adequate, or inadequate for parts of one or more standards Compliance –Sufficient, or insufficient for parts of one or more standards Severity – Insignificant or Moderate Frequency – Isolated, but considered a Finding if recurring and pervasive 67

Reporting Matrix 68

Pass with Deficiencies Audit organization’s system of quality control has been suitably designed and complied with to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects with the exception of a certain deficiency or deficiencies that are described in the report 69

Pass with Deficiencies Design –Adequate overall, but inadequate for substantially all of one standard or parts of several Compliance –Sufficient overall, but insufficient for one standard or parts of several Severity – Serious Frequency – Recurring & Pervasive 70

Fail Based on significant deficiencies described in report, audit organization’s system of quality control is not suitably designed to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects 71

Fail Or based on significant deficiencies described in report, audit organization has not complied with its system of quality control to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects 72

Fail Design –Inadequate to provide reasonable assurance – several standards Compliance –Insufficient overall – several standards Severity – Severe Frequency – Recurring & Pervasive 73

Conclusions Document Used to determine the appropriate reporting for matters –FFC Form –Deficiency –Significant Deficiency 74

Conclusions Document 75

Findings for Further Consideration Form (FFC) Used to report findings not rising to the level of a deficiency or significant deficiency This becomes part of the working papers, but not part of the reporting process 76

FFC Form 77

Completion Phase Finalize the peer review report If rating is other than pass, audit organization should respond 78

Tips for a Successful Peer Review

Communication A successful review depends on good communication at all levels before and during the review Avoid making assumptions Ask questions Be responsive when asked a question or information is requested 80

Preparedness Understand the peer review process Establish and adhere to deadlines for pre-review and on-site work Review the state office’s website and work products 81

Audit Reviews – Planning, Execution and Documentation Work from the end inward (documentation) –Examine the report and supporting work Also work from the start (planning) –Look at the application of general standards and the planning Meet in the middle (execution) –Did they follow procedures? –Do what they planned? –Properly report what they found? 82

Learn Something There is more than one way to comply with auditing standards The peer review process should be value-added –What can the state being reviewed learn from the review team? –What can the review team learn from the state being reviewed? 83

How can you get involved? Must have at least three years of supervisory experience Complete Team Member Qualifications form online –Info on your background, experience and the types of audits you supervise, as well as you availability and preferences for where you want to go Must update form each year Currently about 300 participants 84

Don’t Get Discouraged Factors can limit your chance of selection – especially the first time –Number of reviews scheduled –Type of audits those states perform –Usually only one person per state on a team –Usually no more than one first-time reviewer –Your state’s balance in time credits 85

Any Other Questions? Talk to your office’s peer review liaison or contact the Peer Review Coordinator, Kathleen Young, at or (859)

87 Questions MODERATOR R. Kinney Poynter Executive Director, NASACT SPEAKER John Buyce Director of State Audits, Office of the State Comptroller (NY) SPEAKER William (Brad) Blake Chief Auditor, Center for Audit Excellence, Office of State Auditor (OH) SPEAKER Greg Fugate Performance Audit Manager, Office of the State Auditor (CO) SPEAKER Staci Henshaw Deputy Auditor, Virginia Office of the Auditor of Public Accounts (VA)