1. 17 May 2011Utah Legislative Auditor GeneralSlide 1 Office of the Legislative Auditor General Performance Audits and Evaluation

2. Who We Are 17 May 2011Utah Legislative Auditor GeneralSlide 2 The Office of the Legislative Auditor General (OLAG) is constitutionally created by Article VI Section 33 of the Utah Constitution. The Legislative Auditor General has broad authority to conduct audits of any funds, functions, and accounts in any branch, department, agency or political subdivision of this state. The Legislative Auditor General is appointed by the Legislature for six year terms. The Auditor General reports directly to the bipartisan Audit Subcommittee of the Legislative Management Committee.

3. What We Do 17 May 2011Utah Legislative Auditor GeneralSlide 3 It is the mission of the Office of the Legislative Auditor General to serve the citizens of Utah and the Legislature by providing objective information, in-depth analyses, and useful recommendations that help Legislators and other decision makers. The Office of the Legislative Auditor General accomplishes its mission by:  Improving Programs  Reducing Costs  Promoting Accountability

4. Completing The Puzzle 17 May 2011Utah Legislative Auditor GeneralSlide 4 Resolving Issues Identified By Legislators

5. What Some People Say We Do 17 May 2011Utah Legislative Auditor GeneralSlide 5 “The Washington State initiative sent ripples through the nation's public-sector auditing community, and for a very simple reason: Performance auditing is an evolving and somewhat controversial practice.” October 2009 “They continue to insist that the exercise focuses more on catching governments in the act than on proactively working to improve key government operations.” October 2009

6. What Some People Say We Do 17 May 2011Utah Legislative Auditor GeneralSlide 6

7. What Some People Say We Do 17 May 2011Utah Legislative Auditor GeneralSlide 7

8. 17 May 2011Utah Legislative Auditor GeneralSlide 8 used with permission How We Cope

9. 17 May 2011Utah Legislative Auditor GeneralSlide 9

10. 17 May 2011Utah Legislative Auditor GeneralSlide 10 Audit Complexity Degree Of Difficulty Time & Dollars Expended Financial Compliance Efficiency (Operational) Effectiveness (Program)

11. What Is Performance Auditing 17 May 2011Utah Legislative Auditor GeneralSlide 11 Performance audit refers to an examination of a program, function, operation or the management systems and procedures of a governmental or non- profit entity to assess whether the entity is achieving economy, efficiency and effectiveness in the employment of available resources. The examination is objective and systematic, generally using structured and professionally adopted methodologies.

12. Examples of Performance Audits 17 May 2011Utah Legislative Auditor GeneralSlide 12 Completing of the Puzzle Via Legislative Audit Requests * Identify reasons why a heightened number of jail escapes and other inappropriate activities where happening in county jails. *Identify instances of fraud, waste, and abuse in Utah’s Medicaid program. *Review efficiency and effectiveness in Utah’s Medicaid managed care. *Examine the use of O&M funding for institutions of higher education. *Are criminal background check procedures in public education effective. *Are discrimination complaints being processed effectively and equitably.

13. 17 May 2011Utah Legislative Auditor GeneralSlide 13 OLAG Methodology inputsactivitiesoutputs Statute – Mission -- Objectives people facilities equipment supplies financial ability ACTUAL— service(s) product(s) EFFICIENCY THE MANNER IN WHICH AN AUDITEE USES ITS RESOURCES TO GENERATE SERVICES OR PRODUCTS

14. Efficiency Audits Compare How Well Inputs Are Used to Achieve Outputs Are they performing economically and timely? 17 May 2011Utah Legislative Auditor GeneralSlide 14 DistrictPlanned Elementary Utilization Actual Elementary Utilization Planned Secondary Utilization Actual Secondary Utilization Alpine62 %49 %127 %87 % Davis555010260 Jordan705013074 Murray53486754 Park City43339753 Salt Lake766112081 Washington65589483 Average62 %53 %100 %65 %

15. Smaller Buses Should Be Considered In 12 select school districts 194 Class D school buses could have been replaced with Class C buses For a potential savings of about $5 million Class C Bus:Capacity = 72Average Cost = $ 62,700 Class D Bus:Capacity = 84Average Cost = $ 88,100 17 May 2011Utah Legislative Auditor GeneralSlide 15

16. 17 May 2011Utah Legislative Auditor GeneralSlide 16 OLAG Methodology inputsactivitiesoutputs Statute – Mission -- Objectives ACTUAL— service(s) product(s) EFFECTIVENESS THE EXTENT TO WHICH AN AUDITEE ACCOMPLISHES ITS OBJECTIVES, AND ULTIMATELY, ITS GOALS INTENDED— service(s) product(s)

17. Effectiveness Audit Compares Intended Output to Actual Output Are they performing well? 17 May 2011Utah Legislative Auditor GeneralSlide 17

18. Effectiveness Audit Compares Intended Output to Actual Output Are they performing well? 17 May 2011Utah Legislative Auditor GeneralSlide 18 Audit Found:  Management did not listen to the jail reviews Action Taken:  New management took action instituted standards and consequences for not meeting standards Jail Contracting: Conducted annual reviews of county jails. They were performed for the most part efficient – but they were not effective?

19. Audit Phases Survey Phase - Assess for risk Fieldwork Phase – Audit tests Report – Fieldwork areas developed into chapters Presentation – Audit is made public 17 May 2011Utah Legislative Auditor GeneralSlide 19

20. 17 May 2011Utah Legislative Auditor GeneralSlide 20 Operating Methodology: “Attributes of a Finding” 1.Condition – What is happening? 2.Criteria – What should be happening? 3.Effect – What is the impact of the condition? 4.Cause – What created the condition? 5.Recommendation – What should be done? (corrective or preventative action)

21. 17 May 2011Utah Legislative Auditor GeneralSlide 21 Operating Methodology: Condition What is really happening? What is working well? What is defective or deficient? Is the problem isolated or widespread?

22. 17 May 2011Utah Legislative Auditor GeneralSlide 22 Operating Methodology: Tests For Condition Significant Resources Impact Upon the Public Management Controls Performance Measures Understanding of Mission and Goals Compliance with Statute and Rules

23. 17 May 2011Utah Legislative Auditor GeneralSlide 23 Medicaid Fraud, Waste, Abuse: Condition 25 Year Old Analytical Tool (since last update) Unclear Recovery Methodology Employee Productivity Employee A = 47,384 Employee B = 476,265 Inadequate Performance Measures

24. 17 May 2011Utah Legislative Auditor GeneralSlide 24 Operating Methodology: Criteria Examples: Laws, policies or rules Accepted industry standards Goals and objectives Practices in other states or private sector Expert opinion or auditor opinion Generally accepted practices Question Which one can you use? S t r o n g e s t — W e a k e s t

25. 17 May 2011Utah Legislative Auditor GeneralSlide 25 Medicaid Fraud, Waste, Abuse: Criteria Other state Medicaid programs (for general guidance on all areas) Federal laws and rules (for analytical tools) Accepted business practices (for employee productivity)

26. 17 May 2011Utah Legislative Auditor GeneralSlide 26 Operating Methodology: Effect What is the impact or consequence? What is the extent of difference between criteria and condition? Is the difference significant in terms of cost, performance, etc.?

27. 17 May 2011Utah Legislative Auditor GeneralSlide 27 Medicaid Fraud, Waste, Abuse: Effect Only 5% of Medicaid dollars were being reviewed for F,W,A National Statistics = $20.2 million

28. Weakest Criteria Weakest Effect Strongest Criteria Operating Methodology: Criteria & Effect Relationship Laws or Mission Statements Industry or Professional Standards Historical Data or Goals & Objectives Expert Opinion Prudent Business Practices Auditor Opinion No Significant Impact Implied Savings Small Dollar Impact Improved Services Large Dollar Impact Loss of Life or Public Danger Strongest Effect 17 May 2011Utah Legislative Auditor GeneralSlide 28

29. 17 May 2011Utah Legislative Auditor GeneralSlide 29 Operating Methodology: Finding the Underlying Cause Ask why? Need documentation Lends strength to the recommendation

30. 17 May 2011Utah Legislative Auditor GeneralSlide 30 Medicaid Fraud, Waste, Abuse: Cause Management Budget Constraints Lack of Adequate Training

31. 17 May 2011Utah Legislative Auditor GeneralSlide 31 Operating Methodology: Recommendations What Should Be Done to Correct the Problem or Issue: A direct relationship between the cause and the recommendation. Work with organization to get agreement. Change is the overall goal.

32. 17 May 2011Utah Legislative Auditor GeneralSlide 32 Operating Methodology: Recommendations

33. 17 May 2011Utah Legislative Auditor GeneralSlide 33 Principles For A Successful Audit Function Resist Intimidation –Understanding the source –Anticipating the act

34. 17 May 2011Utah Legislative Auditor GeneralSlide 34 Principles For A Successful Audit Function Exit for Understanding –How: Reveal findings often –Benefit: Strengthens audit findings Obtain Participation –Recommendations can be joint effort –Implementation is the key to success

35. 17 May 2011Utah Legislative Auditor GeneralSlide 35 Contact Information Kade R. Minchey Office of the Legislative Auditor General Utah State Capitol Complex 315 House Building Salt Lake City, Utah 84114 (801) 538-1033 Office Website: www.le.utah.gov/audit/olag.htm