1. Continuous audit: today and tomorrow Miklos A. Vasarhelyi KPMG Professor – Rutgers University Senior Consultant- AT&T Laboratories

2. Continuous Audit and Reporting Laboratory 2 Outline An evolving framework Some Key issues / the state of the art Some CARLAB experiences Six Steps in Implementing CA Organizational Context Opportunities and Challenges Conclusions

3. An evolving audit framework

4. Continuous Audit and Reporting Laboratory 4 An evolving audit framework Assurance of Data elements Data level Assurance Assurance of Key Processes Process level Assurance Assurance of Reports Report level Assurance XML/ XBRL datum Generated and modified by different processes Balkanization of data Control / Assurance tags Process reviews a la Systrust Internal or outsourced Third party processes are to become the norm Intra and Inter process controls an issue Compliance reports becoming commonplace Traditional audit is an instance of RLA Generated and modified by different processes

5. Continuous Audit and Reporting Laboratory 5 An evolving continuous audit framework Automation Sensoring ERP E-Commerce Continuous Audit Continuous Control Monitoring Continuous Audit Data CA = CCM+ C(D)A CA -> Continuous Audit CCM -> Continuous Control Monitoring C(D)A -> Continuous Data Assurance

6. Continuous Audit and Reporting Laboratory 6 Some Key Issues Two recent surveys (ACL and PWC) show that a large number of key companies are attempting to perform continuous audit like functions An industry of software is evolving with ACL, IDEA, APPROVA, and others growing rapidly Control Monitoring and Continuous Data Assurance are the main approaches The first recorded application was AT&T Bell Laboratories CPAS effort in the 1986-1991 period The Rutgers CarLab is working in leading applications

7. Continuous Audit and Reporting Laboratory 7 Continuous Auditing Value Proposition –Improved business performance Innovations in information technology & analytical modelling enable: –More frequent, timely, accurate & relevant business performance information –Lower compliance risk –Cost reduction

8. Continuous Audit and Reporting Laboratory 8 CAR-Lab Experiences Control monitoring at Siemens Transaction monitoring at Unibanco Continuous (data) assurance at HCA Other –Conceptual developments –Simulating Liberty –EBR work –KPMG projects

9. Overview of CaR-Lab examples

10. Continuous Audit and Reporting Laboratory 10 Siemens' – Project Value Proposition Expanded Audit Coverage Significant Cost Savings Automated Business Process Controls Monitoring Project

11. Continuous Audit and Reporting Laboratory 11 Siemens' – Project Features Formalize & automate internal audit procedures used for business process controls monitoring Conduct “man vs. model” assessments Calibrate “exception rules” to optimize model performance Scale up to all SAP instances Increase frequency of model application, where feasible Transition to Approva application and extend the model where optimal

12. Continuous Audit and Reporting Laboratory 12 A 3 pronged approach to audit automation Automate audit plan using delivered Rule Sets: Est 25% of a typical manual audit plan Automate using external data sets (Static & Variable): Est an additional 25% a typical manual audit plan Re-enginer manual controls into automated controls with improved control precision: Est an additional 25% a typical manual audit plan Total = Automation Opportunity ~75%!!

13. Continuous Audit and Reporting Laboratory 13 MCP A.A.S (audit Action Items) From Siemens Approva and other literature Class of Auditable Actions ---- of Audit Processes Audit Evidence Receptacle Master Audit Program Audit Parameterization Tool Other Static Parameters Evergreen Opinion Inference Engine Auditor Management Operating Alarm Flows CA Control Dashboard Deter- ministic Stocha- stic External Table comparisons Snapshot comparisons Other Remote Audit Communic. Tool Data Extraction Interactive Mail Management Tool Sustainable Object Verification Tool Other

14. Continuous Audit and Reporting Laboratory 14 IT / IA Continuous Auditing Program at Unibanco

15. Continuous Audit and Reporting Laboratory 15 Unibanco – Some CA Program Features Automated monitoring of over 5 million customer accounts on a daily basis using 25 automated procedures to : –Detect errors –Deter inappropriate events & behaviors –Reduce or avoid financial losses –Help assure compliance with existing laws, policies, norms and procedures Examples of “low hanging fruit:” –Customer advances –Excess over credit limit –Returned checks –Federal tax payment cancellations –TED emissions (should this be omissions?)

16. Continuous Audit and Reporting Laboratory 16 Unibanco – Advances to Clients Monitoring

17. Continuous Audit and Reporting Laboratory 17 Continuous Data Assurance (CDA) at a Major Health Services Provides (HSP) HSP is a large national provider of healthcare services, composed of locally managed facilities that include numerous hospitals and outpatient surgery centers. IT internal audit provided access to unfiltered extracts from their transactional databases, comprising all procurement cycle daily transactions from October 1st, 2003 through June 30th, 2004: Over 500,000 data points. Dataset mimics what a CDA system has to deal with: highly disaggregate data flowing through CA system in real time. Audit procedures have to be developed for this environment.

18. Continuous Audit and Reporting Laboratory 18 Analytical Procedures in CA Analytical procedures used in the planning, substantive testing, and reviewing stages of an audit. We focus on substantive testing. In conventional auditing first apply analytical procedures to identify potential problems, Then, focus detailed transaction testing on the identified problem areas. In CDA the sequence is reversed: 1.Use automated general transaction tests to all the transactions and filter out identified exceptions for resolution. 2.Apply automated analytical procedures to the filtered transaction stream to identify unforeseen problems. 3.Alarm humans to investigate anomalies.

19. Continuous Audit and Reporting Laboratory 19 Continuous Data Assurance Automation of Transaction Testing: –Formalization of business process rules as transaction integrity and validity constraints. –Verification of transaction integrity and validity  detection of exceptions  generation of alarms. Automation of Analytical Procedures: –Selection of critical business process metrics and development of stable business flow (continuity) equations. –Monitoring of continuity equation residuals  detection of anomalies  generation of alarms.

20. Continuous Audit and Reporting Laboratory 20 Enterprise System Landscape Ordering Accounts Payable Materials Management Sales Accounts Receivable Human Resources Business Data Warehouse Automatic Transaction Verification Exception Alarms Automatic Analytical Monitoring: Continuity Equations Anomaly Alarms Continuous Data Assurance System Responsible Enterprise Personnel

21. Continuous Audit and Reporting Laboratory 21 Establishing Data Integrity: A Procurement Example Referential integrity along the business cycle and identification of completed cycles: P.O.  Shipment receipt  voucher payment. Identification of data consistency issues and automatic alarms to resolve exceptions: –Changes in purchase order vendor numbers; –Discrepancies between the totals and the sums of line items; –Discrepancies between matched voucher amounts.

22. Continuous Audit and Reporting Laboratory 22 Detection of Exceptions Referential integrity violations –PO without matching requisition –Received item without matching PO –Payments without matching received items Data integrity violations –PO has zero order quantity –Received item has negative quantity –Invalid payment check numbers (e.g. All 0s) –Gross payment amount is smaller than net payment amount

23. Continuous Audit and Reporting Laboratory 23 Continuity Equation Based CDA Continuity Equations: –Stable probabilistic models of highly disaggregated business processes, uses as the expectation models for process based analytical procedures. –Originated in physical sciences (various conservation laws: e.g. mass, momentum, charge). Continuity equations are developed using statistical methodologies of: 1.Linear regression modeling (LRM); 2.Simultaneous equation modeling (SEM); 3.Multivariate time series modeling (MTSM) using various Vector Autoregressive Models (VAR).

24. Continuous Audit and Reporting Laboratory 24 Basic Procurement Cycle P.O.(t1) Receive(t2) Voucher(t3) t2-t1 t3-t2

25. Continuous Audit and Reporting Laboratory 25 Ideal Continuity Equations of Basic Procurement Cycle Receive(t2)= P.O.(t1) Voucher(t3)= Receive(t2) Aren’t partial deliveries allowed? Are all orders delivered after exactly the same time lag? Are there any feedback loops?

26. Continuous Audit and Reporting Laboratory 26 P.O.(t)= 0.24*P.O.(t-4) + 0.25*P.O.(t-14)+ 0.56*Receive(t-15) + ε PO Receive(t)= 0.26*P.O.(t-4) + 0.21*P.O.(t-6)+ 0.60*Voucher(t-10) + ε R Voucher(t)=0.54*Receive(t-1) - 0.17*P.O.(t-9) + 0.22*P.O.(t-17) + 0.24*Receive(t-17) + ε V Estimated Continuity Equations of Procurement Using VAR Model

27. Continuous Audit and Reporting Laboratory 27 Detection of Anomalies Anomalies are detected if: –Observed P.O.(t) < Predicted P.O.(t) - Var or –Observed P.O.(t) > Predicted P.O.(t) + Var Similarly for: –Receive(t) –Voucher(t) Var = acceptable threshold of variance. If there is anomaly  generate alarm!

28. Continuous Audit and Reporting Laboratory 28 Measuring Anomaly Detection False positive error (false alarm, Type I error): A non- anomaly mistakenly detected by the model as an anomaly. Decreases efficiency. False negative error (Type II error): An anomaly failed to be detected by the model. Decreases effectiveness. Detection rate is used for clear presentation purpose: The rate of successful detection of seeded errors. A good analytical model is expected to have good anomaly detection capability: low false negative error rate (i.e. high detection rate) and low false positive error rate.

29. Continuous Audit and Reporting Laboratory 29 Simulated Error Correction Access to highly disaggregate data in real time makes it possible for CA system to detect, investigate and correct anomalies also in (nearly) real-time. Real-time error correction enables utilizing the corrected rather than the erroneous data in revised continuity equation benchmarks. Real-time error correction is likely to benefit future anomaly detection. We investigate the magnitude of this benefit using simulation. Error correction raises important issues about auditor independence, and the line between auditing and monitoring of business processes.

30. Continuous Audit and Reporting Laboratory 30 Benefit of Real-time Error Correction: MTSM

31. Continuous Audit and Reporting Laboratory 31 Anomaly Detection Rate Comparison and Aggregation Levels of Metrics SEM and MTSM outperform the linear regression model when the error magnitudes are large, even though linear regression has slightly better detection rate when the error magnitudes are small. It is more important to detect material errors than non- material errors. The higher the aggregation level of BP metrics the more stable probabilistic relationships are likely to be and the fewer models have to be estimated and monitored. The higher the aggregation level of BP metrics the more work it will take on auditor’s part to investigate every anomalous event.

32. Continuous Audit and Reporting Laboratory 32 Takeaways from HSP Study Various statistical methods can be used to derive expectation models of acceptable quality. But key is access to highly disaggregate data, not which benchmark is used. With such data, most reasonable continuity equation models give usable results. Real-time error correction significantly improves error detection. More disaggregated models are not always better: weekly data can be more stable than the daily one. Alarms have to be managed – trade-off between Type I and Type II errors.

33. Implementation Issues in CA

34. Continuous Audit and Reporting Laboratory 34 Background –While technologies of continuous audit have been extensively discussed and are progressively emerging the more mundane issues of their implementation in a socio- technical environment have been neglected –http://www.theiia.org/itaudit/features/in-depth- features-2-10-08/feature-2/

35. Continuous Audit and Reporting Laboratory 35 2. Rule 5. Follow-up 1.Priority Areas 6. Action and Reaction 4. Parameterization 3. Frequency Audit Control Panel Six steps of process implementation

36. Continuous Audit and Reporting Laboratory 36 –1. Identification of Priority Areas Modularize risk areas, rate these risks and evaluate the cost x benefits Identify the basic audit objects Choose critical business processes that will be the focus of continuous audit (low hanging fruit) Identify key data in for the implementation of Continuous Audit in the mapped processes Political Considerations

37. Continuous Audit and Reporting Laboratory 37 Key Objective of Audit Procedure –Detective –Deterrent –Financial –Compliance

38. Continuous Audit and Reporting Laboratory 38 2. Rules of Monitoring and Auditing –Once an area of CA is chosen the “rules” of monitoring, alarming, and assurance must be established –These must take into consideration the legal and environmental issues as well as the objectives of the particular process –The CA process is established adopting certain rules, frequencies, and parameters. –e.g. we will monitor bank accounts in overdrafts or in excess limits

39. Continuous Audit and Reporting Laboratory 39 3. Frequency –The natural rhythm of the process Timing of computer processes Timing of business processes –Cost benefit considerations –Nature of procedure objectives Deterrence Prevention

40. Continuous Audit and Reporting Laboratory 40 –4. Parameterization Define parameter to analyze in accordance with the risk eg.: Monitoring all accounts in overdrafts in daily basis, that have a balance of debt 20% larger than its limit of loan and bigger than 1000 USD

41. Continuous Audit and Reporting Laboratory 41 5. Follow-up –Who will receive the alarm? Management? Audit leadership? Immediate superior of the responsible for the data The timing of the follow up –Pass the alarm along immediately –Reconcile the alarm prior to follow up –Wait for 3 sequential days of similar alarms to follow up Escalation guidelines –E.g. after three days send to the immediate superior’s superior or wait for 3 days prior to the re-escalation

42. Continuous Audit and Reporting Laboratory 42 6. Action and Reaction –Guidelines for dealing with auditees Lack of bias Consistency of response Guidelines for individual factor considerations Concern with collusion

43. Organizational Issues

44. Continuous Audit and Reporting Laboratory 44 Organizational Structure for CA –Is CA a part of the audit function or of management? –Its part of the audit function –Should there be a separate continuous audit group? –Yes, to facilitate its implementation progressively in the many areas of continuous audit

45. Continuous Audit and Reporting Laboratory 45 Workforce Effects –Progressively labor requirements for the traditional audits supported by CA will reduce and deeper audit will become possible –Rebalancing of workforces –High technological competencies needed

46. Opportunities and Challenges

47. Continuous Audit and Reporting Laboratory 47 Opportunities for business and research (1) Control system measurement –We are in a pre-paradigmatic stage of control documentation and measurement –We do not know how to monitor controls in large ERPs –We do not know how to provide a really supportable opinion on controls –We do not know how to rate combinations of controls

48. Continuous Audit and Reporting Laboratory 48 Opportunities (2) Business Process Monitoring and Alarming –Auditors have to carve a position on the new monitoring and control environment –Auditors can collect exception “alarms” as trusted parties and incorporate these into evidentiary matter –Auditors can be “trusted”

49. Continuous Audit and Reporting Laboratory 49 Opportunities (3) Automatic Confirmation Tools –Confirmations will have an increased evidentiary role with eventual elimination of population and integrity worries –Intelligent confirmatory tags can do much –Database to database hand-shaking will be medium –Business opportunity for auditors

50. Continuous Audit and Reporting Laboratory 50 Opportunities (4) Audit bots (agents) –Many of the basic audit functions can be emulated by software –These must be eventually developed by the profession to work hand-in- hand with human auditors in the new audit world –These agents will work on all areas including: 1) audit planning, 2) analytical reviews, 4) confirmations, and )5 evergreen opinions

51. Continuous Audit and Reporting Laboratory 51 Opportunities (5) Collecting forensic trails –Auditor “black” box Publishing real-time authenticated reports for different compliance masters Publishing FD independent compliance reports

52. Continuous Audit and Reporting Laboratory 52 Challenges Standards are needed for CA –Audit monitoring needs to be defined –Types of evidence are to change and must be reconsidered –Independence needs to be re-defined The billing model has to be restructured to bill on function not hours

53. Continuous Audit and Reporting Laboratory 53 Challenges Audit firms must put improved knowledge collection and management processes to feed their audit analytic toolkit Audit firms have to engage in auditor automation and pro-actively promote corporate data collection during-the-process Value added must be justified in terms of data quality

54. Continuous Audit and Reporting Laboratory 54 Conclusions –Attention must be paid to the organizational processes that implement continuous audit –There are 6 key steps to progressively implement a CA program module by module –The CA process is dynamic and CA management will change schedule and parameters of each process –The organization of the audit process must be evolved progressively