Expect the Unexpected Planning the Scope of an IT Performance Audit Robin Garity, C.P.A., C.I.S.A. October 2014

 Standards  Importance  Audit Assignment #1 – Michigan Business One Stop System  Audit Assignment #2 – Branch Office System Agenda

 Generally Accepted Governmental Auditing Standards (GAGAS) states  6.07 Auditors must plan the audit to reduce audit risk to an appropriate level for the auditors to obtain reasonable assurance that the evidence is sufficient and appropriate to support the auditors’ findings and conclusions.  6.09 The scope defines the subject matter that the auditors will assess and report on, such as a particular program or aspect of a program, the necessary documents or records, the period of time review, and the locations that will be included. What do the standards say about Performance Audit Planning?

 Determines direction of audit (many possibilities)  Security  Accurate processing  Efficiency of system  Governance  Determines audit value  What will change if the conclusion is that the auditee/system is not effective?  Will recommendations be useful? Why is planning the audit scope important in a performance audit?

 Ensures that all significant risks are identified and addressed during the audit  Poor scope planning can result in a stressful audit  Inadequate resources  Inefficient testing  No pressure…But don’t mess up when planning the audit scope! Why is planning the audit scope important in a performance audit? (continued)

 Assignment based on criticality to audit entity  System mission - Create a one-stop shop for individuals or businesses doing business with the State of Michigan  No prior audits  Implemented in 2009  Known costs of $21.3 million to date for development and maintenance Audit Assignment Example #1 Michigan Business One Stop System (MBOS)

 Confidential and critical licensing information in the system.  Operating System Access and Configurations  Database Access and Configurations  Application Access  Monitoring Processes Scope Planning Ideas

 Interviewed project manager, DBA, and system administrators  Reviewed system documentation  Data dictionary  Network diagram  Development contracts  Reviewed policies and procedures for managing the system  Interviewed users/stakeholders Scope Planning Procedures

 Very few customers liked or used MBOS  Process was much more complex for customers  Applicant data must be reentered into secondary systems  New development projects on hold because of uncertainty regarding MBOS’s future  Departments unsure of what license information is available in the system What We Heard

 FROM: Operating System Access and Configurations Database Access and Configurations Application Access  TO: Project Planning - Is there a plan for making the system more effective? Governance - Is there leadership to make decisions on the future of the system? Updating of System - If departments are unsure of licenses in the system, are license applications really up to date in MBOS? Scope U-Turn

 Always interview users of the system during planning.  Keep in mind the future impact.  Be flexible. What We Learned About Planning the Audit Scope

 Findings  No strategic plan for continued development and use of the system.  No post-implementation review to determine if expected benefits were realized.  Lack of an effective governance structure.  No process to periodically review and update the content (out-of-date fees, applications, etc.)  Latest update – DTMB is shutting down the system because it is not providing the expected benefits. Outcome

 System used in branch offices for vehicle registrations, driver licensing, etc.  The Department of State collects approximately $2.2 billion per year through the various systems that process driver and vehicle related transactions.  Audit assignment based on revenue and criticality of system Audit Assignment Example #2 Branch Office System

Branch Office System  Application controls  Access/segregation of Duties  Proper input of licensing and registration data  Change management Scope Planning Ideas

 Interviewed project managers, DBA, and system administrators.  Reviewed system documentation  Data dictionary  Network diagram  Development contracts  System flows  Reviewed policies and procedures for managing the system.  Interviewed system users. Scope Planning Procedures

 Branch Office System scheduled for replacement.  Many systems process driver and vehicle related data on the back end and store confidential data. The Branch Office System is primarily data input.  Complex flow of information between departments for use in processing driver and vehicle-related data.  Prior non-IT audit of fee calculations (audited around systems) but no actual IT audits. What We Found Out

 FROM: Branch Office System Application controls  Access/Segregation of duties  Proper input of licensing, registration data  TO:  Excluding Branch Office System (being replaced)  Security for other driver and vehicle related systems that store confidential data  Operating System  Database  Reviewing actual processing of data outside of Branch Office System  Are matches and input of information proper to ensure no registrations to suspended licenses, deceased, stolen vehicles, etc.  Excluding fee calculations A New Focus

 Consider new development projects  Consider entire process  Understand in detail what has already been audited What We Learned About Planning the Audit Scope

 Security weaknesses  Access issues  Data processing inconsistencies Potential Audit Conclusions

 Be sure to:  Spend sufficient time in planning  Obtain complete understanding of business processes and flow of system data  Listen to what auditee and users think are the problems  Evolve your scope  To ensure:  Audit value  Impact on future processes  An efficient audit Final Suggestions For Planning the Audit Scope