1. What Can We Learn From IT Control Weaknesses Reported under SOX 404? Efrim Boritz Louise Hayes Jee-Hae Lim University of Waterloo UWCISA 6th Bi-Annual Research Symposium

2. Motivation previous findings that IT control quality is associated with financial performance and other factors of interest to researchers and practitioners data availability little systematic, detailed study of specific IT weaknesses

3. Research Questions What are common clusters of IT control weaknesses? Is year of report, industry, company size, auditor type or internal control quality associated with these clusters? Is the pattern of non-IT weaknesses the same in companies with and without IT weaknesses?

4. Methodology use content analysis software to identify detailed IT weaknesses in the 2004-2006 SOX 404 audit reports 1,343 SOX 404 reports 2004-2006 20% with Information technology (IT) software, security or access concerns cross-tabulate and study frequency of association with non-IT weaknesses, year of report and company specific factors

5. Content Analysis Example... (iii) insufficient controls over the determination and application of generally accepted accounting principles with respect to revenue recognition for private label agreements and other revenue agreements, excluding those related to FindWhat.com Network revenue, (iv) the accounting function to resolve non-routine or complex accounting matters, (v) insufficient controls over and review of the quarterly and year- end financial statement close and review process, and (vi) insufficient segregation of duties whereby financial accounting personnel had access to financial accounting IT applications and data and also performed incompatible duties with respect to the authorization, recording, and control activities…. IT indicator Exclusionary phrase Less restrictive search: Keyword/phrase and IT indicator in the same sentence* More restrictive search: ignore keyword/phrases if exclusionary word/phrase in the same sentence*

6. Reliability Audit Analytics ‘code 20’ is a reliable indicator of IT weakness Automated content analysis is sufficiently reliable for the analyses performed in this paper

7. 17 IT Weaknesses Table 2 4 COSO Categories Table 3 – Panel B Statistical analysis requires grouping of low frequency IT weaknesses. Grouping used retains more detailed information than COSO categories. 8 Groups for Analysis Table 3 – Panel A

8. IT Weaknesses Frequency – Table 3

9. Year of Report Association? An interaction in 2004 –company size/auditor type –described later with auditor type otherwise, no association with year 2004-2006 –between 3 (restrictive search) and 4 (less restrictive search) IT weaknesses per company each year –5 non-IT weaknesses/yr for companies with IT weaknesses –3-4 non-IT weaknesses/yr for companies without IT weaknesses

10. Industry association? Selected industries have significantly lower numbers of IT weaknesses –Banks, insurance companies and real estate services had the fewest IT weaknesses (p<.03) –Fewest number of non-IT weaknesses depends on whether or not company has IT-weaknesses

11. Company Size Association? 60% of companies with IT weaknesses were smaller larger companies with IT weaknesses had more non- IT weaknesses (p=.013) the average number of non-IT weaknesses in companies without IT weaknesses did not differ with company size

12. Internal Control Quality Association? split on internal control quality differs from the number expected by chance (p<.001) 57% of companies with IT weaknesses, vs 18% without, had 5 or more non-IT weaknesses Fewer than 5 non-IT weaknesses? number of non-IT weaknesses same for companies with and without IT weaknesses. –More than 5? More if IT weakness

13. IT Weaknesses Cluster

14. Pairwise Associations of IT and non-IT weaknesses? Table 4: 6 IT weakness categories x 20 non-IT categories –Less than 1/6 of 120 possible associations are significant (p<.05) Significant associations across both restrictive and less restrictive search criteria for: –Accounting personnel resources, competency/training –Journal entry control issues –Untimely or inadequate account reconciliations –Management/Board/Audit Committee investigation(s) –Material and/or numerous auditor /year-end adjustments –Segregations of duties/ design of controls (personnel) –Scope (disclaimer of opinion) or other limitations –Senior management competency, tone, reliability issues –Ineffective or understaffed audit committee

15. Pairwise Associations of IT and non-IT weaknesses? IT weaknesses in decreasing order of association with the most non-IT weaknesses: –Monitoring (the most frequent IT-weakness) –End user computing (the third most frequent IT-weakness) –Other Information & Communication –Control Environment –Segregation & Access (the second most frequent IT-weakness) –Design, Change and Documentation the infrequent reporting of IT risk assessment weaknesses, surprising given the large number of weaknesses in other COSO categories, warrants additional investigation

16. Contribution “Dictionary” of keywords/phrases Demonstrate automated search feasibility More granular coding of IT weaknesses than Audit Analytics Identification of small number of frequently- occurring clusters of IT and non-IT control weaknesses –Researchers testing models using IT weaknesses as dependent or independent variables should take care not to omit non-IT weaknesses in their models

17. Future Research Track remediation of weaknesses and differences, if any, associated with persistent weaknesses reported for two or three years running Survey IT control and audit practitioners to obtain their reaction our coding and findings Relate the IT weaknesses and clusters of weaknesses identified to company financial performance