1. NSAA Peer Review Program: What to Know Before You Go! National State Auditors Association January 21, 2015 1

2. 2 Opening Remarks MODERATOR R. Kinney Poynter Executive Director, NASACT SPEAKER John Buyce Director of State Audits, Office of the State Comptroller (NY) SPEAKER William (Brad) Blake Chief Auditor, Center for Audit Excellence, Office of State Auditor (OH) SPEAKER Greg Fugate Performance Audit Manager, Office of the State Auditor (CO) SPEAKER Staci Henshaw Deputy Auditor, Virginia Office of the Auditor of Public Accounts (VA)

3. Agenda Overview, Objectives, and General Considerations Organization and Qualifications of the Review Team Performing Peer Reviews Finalizing the Peer Review Tips & Insights 3

4. Overview, Objectives, and General Considerations

5. Overview Purpose of peer review program – to provide independent assessments of state audit organizations to determine whether they have an adequately designed internal quality control system and are in compliance with that system 5

6. Overview Yellow Book began requiring peer reviews in 1989 NSAA program started in 1985 Currently, 54 eligible organizations participate in the program 6

7. Overview – Role of Committee Peer Review Committee has overall responsibility for the program –Provide guidance in the form of policies and procedures –Resolve potential disputes that may arise in the review process and ensure consistency of reviews –Coordinate with other groups to ensure the adequacy of the review process 7

8. Overview – Role of NASACT Administer the review process on a daily basis Coordinate and assign review teams Train review team members Update policies and procedures for review by the Peer Review Committee 8

9. Objectives of the Review Evaluate whether the State Audit Organization’s system of quality control is… –suitably designed, adequately documented and communicated; and –sufficiently complied with to provide reasonable assurance of compliance with government auditing standards 9

10. Objectives of the Review Upon completion of the peer review, the peer review team issues a report Types of peer review reports –Pass –Pass with deficiencies –Fail 10

11. General Considerations Maintain confidentiality of state’s working papers and peer review work papers Be independent of state audit organization being reviewed Possess adequate knowledge and proficiency to perform review 11

12. Organization and Qualifications of the Review Team

13. Organization of Review Team Concurring reviewer –Most senior member of team Team leader –Overall responsibility for planning and performing the review Team member –Responsible for performing tasks assigned by team leader 13

14. Organization of Review Team Team leader and concurring reviewer identified 4-5 months prior to the actual date of the review Team members identified 2-3 months prior to the actual date of the review 14

15. Qualifications – Concurring Reviewer Must have served as team leader on a least one review under NSAA’s program Must have been recommended by former concurring reviewer 15

16. Qualifications – Team Leader Must have served as team member on a least one review under NSAA’s program Must have been recommended by former team leader and concurring reviewer 16

17. Qualifications – Team Member Must be recommended by audit organization head Must be in a supervisory or managerial role Must review audit documentation as part of job responsibilities Minimum of three years in supervisory role 17

18. Qualifications for Review Team Member – Other Issues Ensuring a proper match –NASACT tailors team skills and experience to the SAO’s needs Type of work performed (Financial, Performance, Attest) Size of the SAO and number of audits Government Audit Quality Center (AICPA- GACQ) membership Federal reviewer for Single Audit Usually up to six reviewers selected 18

19. Performing the Peer Review

20. Preliminary phase Field work phase Completion phase 20

21. Preliminary Phase Understanding the Peer Review Program Obtaining/reviewing necessary SAO info including prior work papers Determining scope of review Sending audit staff questionnaire Selecting engagements Finalizing planning 21

22. Documents Used in a Peer Review Audit Organization Questionnaire Audit Staff Questionnaire Audit Organization’s Policies and Procedures and Review Guide Guide for Review of Audit Engagements Matters for Further Consideration (MFC) Findings for Further Consideration (FFC) Conclusions 22

23. Team Preparation: Understanding the Review Process Ensure each team member obtains an understanding of the process –NSAA External Peer Review Manual –Questions and Answers for All Team Members –Administrative Matters Timing of review procedures Travel arrangements General expectations 23

24. Team Leaders: Review of Prior Work Papers Issues identified in prior peer review report –Prior peer review rating Read notes from exit conference Review other issues identified (MFCs and FFCs) 24

25. Audit Organization Questionnaire Completed by the SAO and provided to the Team Leader approximately 12 weeks prior to the review Provides the population of audits and audit hours within the review period Provides information about the state office’s staff, structure, and quality control system 25

26. Determining the Scope of the Review Determine whether the SAO performs –Financial Audits –Attestation Engagements –Performance Audits Engagements the SAO has stated to have been performed in accordance with government auditing standards are subject to the review 26

27. Determining the Scope of the Review Review should cover a current period of one year to be mutually agreed upon by the SAO and the review team Covers the quality control policies and procedures in effect and compliance for the year under review Includes reports issued during the year under review, or immediately thereafter if the work was substantially completed during the period 27

28. Determining the Scope of the Review The scope of the review does not cover –Non-GAGAS engagements –Audits done by others (e.g., contracted) –Audit efficiency –State statute compliance –Administrative aspects, unless directly required to satisfy applicable professional standards (e.g., Independence, CPE) 28

29. Other Administrative Matters Team Leader –Work Program –Prepare and send the engagement letter –Work with the SAO to establish deadlines for receiving key documents –Initial communication with Concurring Reviewer and Team Members NASACT –Get signed contract to Team Leader 29

30. Pre-Visit Work Team Leader –Select Audit Engagements for Review –Audit Staff Questionnaire –Preliminary Site Visit –Policy & Procedure Review –Finalizing the Review Plan –Administrative Logistics 30

31. Selecting Audit Engagements for Review Based on listing provided by the SAO Factors to consider include: –Spread across various supervisors, regions, etc. –Spread across the review period –Different types of engagements (Cash Basis vs GAAP, Single Audits, CAFRs, Performance, Attest) Select one “surprise” audit 31

32. Other Sample Selections Select a sample of audit staff to complete the Audit Staff Questionnaire While onsite, select a sample of audit staff for verification of CPE records and independence documentation 32

33. Audit Staff Questionnaire Work with the SAO to administer the questionnaire about 8 weeks prior to the review –Responses submitted directly to the Team Leader about 6 weeks before the review Team Leader compiles the responses and communicates a summary to the team Evaluate responses for potential areas to concentrate review efforts 33

34. Preliminary Site Visit and Other Considerations Team Leader consults with the Concurring Reviewer to determine if a preliminary site visit is needed –Very rare –If one occurs, the preliminary site visit is 4-6 weeks in advance Team Leader has option to arrive on site at the SAO a few days prior to the rest of the team members 34

35. Team Members: Evaluation of the SAO’s Policies and Procedures Policies and Procedures Checklist –Separate checklists for financial audits, performance audits, and attestation engagements –Completed by the SAO and provided to the Team Leader along with the SAO’s policy and procedure manual, usually about 8 weeks prior to the review 35

36. Policies and Procedures Checklist 36

37. Policies and Procedures Checklist Team Leader assigns each team member a section or sections of the policies and procedures to review Assignments are made to the team members approximately 8 weeks prior to the review Team Members are responsible for completing this work before arriving on site for the review 37

38. Finalizing the Review Plan Team Leader discusses plan for the review with the Concurring Reviewer Team Leader informs the SAO about sampled audit engagements, usually about 4-6 weeks prior to the review Team Leader only informs the SAO about the sampled “surprise audit” when the team arrives on site 38

39. Sampled Audits/Engagements Team Leader assigns each team member one or more audits/ engagements to review Team Members are responsible for completing some work before arriving on site for the review –Download and review reports –Complete reporting sections of the Audit/ Engagement Review Checklist 39

40. Audit/Engagement Review Guide 40

41. Administrative Logistics Team Leader will communicate about administrative logistics in the 1-2 weeks leading up to the review –Confirm travel, car, and hotel –Set time/place for initial meeting –Work plan/schedule for the review –Dress code and other information If something is unclear, be sure to ask! 41

42. Field Work Phase Complete the study and evaluation of QC policies and procedures Review compliance with policies and procedures Identify matters, findings, deficiencies, and significant deficiencies Aggregate and evaluate matters Form conclusion on type of report to issue Communicate conclusions at exit conference 42

43. Overall Fieldwork Schedule Five Day Reviews –Generally run Monday - Friday Ten Day Reviews –May run Monday to Wednesday or Wednesday to Friday –Allows a few more days for review of sampled audits 43

44. Time Frames for Large Reviews 44

45. Time Frames for Small Reviews 45

46. Arrival – Day Before the Review Begins Team Leaders often hold a team meeting at the hotel in the evening –Answer any questions, logistics, etc. –Results of policy & procedure reviews Team Members –Discuss policies noted in their assigned area and their reviews of reports Some teams do a conference call during the week before the review 46

47. Day 1 – Let’s Get Started Entrance Conference Compliance testing for Policies & Procedures Get oriented to the audit workpapers, software, network, etc. Begin detailed review of first sampled audits 47

48. Team Leader – Day 1 Run the Entrance Conference Discuss any outstanding issues with SAO officials Compliance testing for P&P Get organized Assist Team Members in starting their audit reviews Status meeting at day’s end 48

49. Team Members – Audit/Engagement Reviews Standard review guide for each type of audit/engagement –Financial, Performance, Attest –Covers all phases of the audit/engagement from planning through reporting –Cross referenced to individual audit standards, AU citations, etc. Use to test compliance with procedures by teams 49

50. Audit/Engagement Review Guide 50

51. Audit/Engagement Reviews Complete checklist for each audit/engagement –Seek clarification from audit teams –Start your second review if waiting for information from others “NO” answers generate an MFC – given to Team Leader 51

52. MFC Form 52

53. MFC Form “Matters” generally will originate from “no” answers on checklist (either with design of, or compliance with, system of quality control) Can be cleared, discussed verbally with the audit organization, or carried forward to the Conclusions document 53

54. Day 2 – More Audit Reviews Team Leader –Answer questions, provide guidance, ensure consistency –Organize MFCs, review completed work, test items to workpapers –Get ready for the Concurring Reviewer to arrive Team Members –Discuss issues with audit teams –In most cases, you should be on your second audit in the afternoon 54

55. Day 3 – Start Wrapping it up Team Leader –Brief the Concurring Reviewer on progress and issues –Continue organizing and grouping MFCs –Assist Team Members in completing audit reviews 55

56. Day 3 – Start Wrapping it up Team Members –Continue completing review guides and MFCs –Second audits should be all but done –If there’s a third, it should be well underway in the afternoon Prioritize the areas you want to review Key on planning, execution, documentation and common issues 56

57. Day 3 – Start Wrapping it up End of the Day –Status meeting Initial discussion of issues that may warrant additional work or be potential Findings –Should have a sense of the overall rating Plan out work that needs to be completed and when –Reassign work and assist each other as necessary 57

58. Day 4 – Pull it all together Team Leader –Organize MFCs for discussion –Manage completion of reviews by mid- day –Lead team discussion to come to a consensus on disposition of MFCs, Findings and rating Contact and consult with NSAA if necessary 58

59. Day 4 – Pull it all together Team Members –Complete all review guides and MFCs –Assist others to tie up loose ends Afternoon – Team Discussion –Review all MFCs and discuss the ones you prepared –Work to consensus on items to elevate to Findings based on significance and pervasiveness 59

60. Day 4 – End of the Day Decide on form of report and issues to discuss at exit conference –Complete the relevant Conclusion documents and finalize a draft of the report –Consult with NSAA as appropriate if report rating is other than “pass” Brief officials about the general conclusions of the report and any findings 60

61. How do you decide what type of report to issue? Based on professional judgment considering nature, causes, pattern and pervasiveness of matters and their relative importance to the organization’s system of quality control taken as a whole 61

62. Forms Used in Documenting Results Matters for Further Consideration (MFC) Conclusions Document Findings for Further Consideration (FFC) 62

63. Reporting Matrix Section II of the Peer Review Manual includes a reporting matrix to provide guidance on various reporting considerations when evaluating results 63

64. Reporting Matrix 64

65. Types of Peer Review Reports Pass Pass with Deficiencies Fail 65

66. Pass Audit organization’s system of quality control has been suitably designed and complied with to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects 66

67. Pass Design –Adequate, or inadequate for parts of one or more standards Compliance –Sufficient, or insufficient for parts of one or more standards Severity – Insignificant or Moderate Frequency – Isolated, but considered a Finding if recurring and pervasive 67

68. Reporting Matrix 68

69. Pass with Deficiencies Audit organization’s system of quality control has been suitably designed and complied with to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects with the exception of a certain deficiency or deficiencies that are described in the report 69

70. Pass with Deficiencies Design –Adequate overall, but inadequate for substantially all of one standard or parts of several Compliance –Sufficient overall, but insufficient for one standard or parts of several Severity – Serious Frequency – Recurring & Pervasive 70

71. Fail Based on significant deficiencies described in report, audit organization’s system of quality control is not suitably designed to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects 71

72. Fail Or based on significant deficiencies described in report, audit organization has not complied with its system of quality control to provide the audit organization with reasonable assurance of performing and reporting in conformity with applicable professional standards in all material respects 72

73. Fail Design –Inadequate to provide reasonable assurance – several standards Compliance –Insufficient overall – several standards Severity – Severe Frequency – Recurring & Pervasive 73

74. Conclusions Document Used to determine the appropriate reporting for matters –FFC Form –Deficiency –Significant Deficiency 74

75. Conclusions Document 75

76. Findings for Further Consideration Form (FFC) Used to report findings not rising to the level of a deficiency or significant deficiency This becomes part of the working papers, but not part of the reporting process 76

77. FFC Form 77

78. Completion Phase Finalize the peer review report If rating is other than pass, audit organization should respond 78

79. Tips for a Successful Peer Review

80. Communication A successful review depends on good communication at all levels before and during the review Avoid making assumptions Ask questions Be responsive when asked a question or information is requested 80

81. Preparedness Understand the peer review process Establish and adhere to deadlines for pre-review and on-site work Review the state office’s website and work products 81

82. Audit Reviews – Planning, Execution and Documentation Work from the end inward (documentation) –Examine the report and supporting work Also work from the start (planning) –Look at the application of general standards and the planning Meet in the middle (execution) –Did they follow procedures? –Do what they planned? –Properly report what they found? 82

83. Learn Something There is more than one way to comply with auditing standards The peer review process should be value-added –What can the state being reviewed learn from the review team? –What can the review team learn from the state being reviewed? 83

84. How can you get involved? Must have at least three years of supervisory experience Complete Team Member Qualifications form online https://www2.nasact.org/QCR/2015_Team_Member_Qualifications.asp –Info on your background, experience and the types of audits you supervise, as well as you availability and preferences for where you want to go Must update form each year Currently about 300 participants 84

85. Don’t Get Discouraged Factors can limit your chance of selection – especially the first time –Number of reviews scheduled –Type of audits those states perform –Usually only one person per state on a team –Usually no more than one first-time reviewer –Your state’s balance in time credits 85

86. Any Other Questions? Talk to your office’s peer review liaison or contact the Peer Review Coordinator, Kathleen Young, at kyoung@nasact.org or (859) 276-1147

87. 87 Questions MODERATOR R. Kinney Poynter Executive Director, NASACT SPEAKER John Buyce Director of State Audits, Office of the State Comptroller (NY) SPEAKER William (Brad) Blake Chief Auditor, Center for Audit Excellence, Office of State Auditor (OH) SPEAKER Greg Fugate Performance Audit Manager, Office of the State Auditor (CO) SPEAKER Staci Henshaw Deputy Auditor, Virginia Office of the Auditor of Public Accounts (VA)