An Interactive Discussion: Contemporary Research on IS Auditors and Automated Controls Dr. Daniel Selby University Of Richmond ISACA VA January 20, 2011

Presentation Roadmap Interactive Judgment Exercises (20 minutes) Interactive Presentation of An Actual Research Study (20 minutes) Debriefing (10 minutes) Closing Comments/more Q&A

Key Terms ● Relevant: information that is useful for a specific judgment outcome (Nisbett et al. 1981; Hilton & Fein 1989;Macrae et al. 1992; and Young et al. 2001) ● Irrelevant: information that is of little value for a specific judgment outcome (Nisbett et al. 1981; Hilton & Fein 1989;Macrae et al. 1992; and Young et al. 2001) ● Key Risk Factor (KRF): factor that directly effects the successful completion of an objective ● i.e., KRFs affect the likelihood of a deficiency or material misstatement of a financial statement amount or disclosure

Motivation ● IS auditors’ sift through numerous pieces of information to target items that are relevant to understanding KRFs (Singleton 2007). ● Individuals have been found to lower their predictions of future outcome predictions when they are exposed to irrelevant information. ● Individuals tend to dilute their predictions by unintentionally overlapping characteristics of the irrelevant information with characteristics of the relevant information. ● Irrelevant information may influence IS auditors to reduce their assessments of KRFs when higher assessments would be more appropriate.

Research Question Can IS auditors ignore irrelevant information when they assess key risk factors?

Why is this study important? ● Irrelevant information in a KRF assessment setting should not be used to assess the KRF. ● If IS auditors decreased their KRF assessments during a financial statement audit as a result of their exposure to irrelevant information, too few resources may be allocated towards better understanding of the KRFs. ● Audit failure may occur

So What??? ● Investors Lose $$$ ● Litigation for IS auditors and their employing firms ● Diminished professional reputations ● i.e., window Lehman Brothers

Theory ● People subconsciously use available information whether relevant or irrelevant, to minimize their cognitive effort during unstructured tasks (Kahneman & Tversky 1972, 1974; Zukier & Jennings 1984). ● Salient features of irrelevant information distract attention away from diagnostic features between the target and the outcome to influence perceptions of similarity for the all available information [i.e., “representativeness” Tversky (1977); Nisbett et al. (1981)]. ● Irrelevant information weakens the relationship between the diagnostic information and outcomes suggested by diagnostic information (Young et al. 2001).

Hypotheses Ha: Irrelevant information will influence auditors to reduce their: KRF effectiveness rating likelihood estimate of the risk of material misstatement audit plan revision relative to the prior year’s clean audit

Methodology ● 37 IS audit specialists ● 38 minutes on average to complete ● Hypothetical integrated audit engagement of a large publicly-traded financial institution ● Pretested Cues (Hackenbrack 1992; Waller and Zimbelman 2003) ● Randomized order (Shaft & Vessey 1998; LaBella & Koehler 2004) ● Manipulation checks: separately rate the diagnosticity (abatement condition in Young et al. 2001) ● Knowledge Test/Background demographics

Participant Demographics (Std. dev.) number37 Avg. # months of experience49.43 (39.66) Avg. # of engagements20.76 ( 26.82) Avg. # of Professional IS Courses7.92 (8.14) Avg. # of IS courses in college2.95 (3.64) % professionally Licensed as CISA, CPA, or both 73%

Means (Std. Deviations) KRF Effectiveness Rating (-3 to +3) Likelihood Estimate of the Risk of Material Misstatement (0 to 100) Audit Plan Revision Relative to the Prior Year’s Clean Audit (0 to 10) Irrelevant Information and Relevant KRF -0.3 (1.27) (27.54) 7.86 (1.32) Relevant KRF only 0.97 (1.19) 57.7 (27.22) 8.46 (1.41)

Statistical Results based on one-tail t-tests ● KRF Effectiveness Rating (t = 6.473, p = <0.0001) ● Likelihood Estimate of the Risk of Material Misstatement (t = 3.948, p = ) ● Audit plan revisions relative to the prior year’s clean audit (t = 2.227; p = )

Knowledge of Automated Controls Mitigates the influence of irrelevant evidence Partitioned experimental participants based on knowledge test scores KRF Effectiveness Rating (p = 0.009) Likelihood Estimate of the Risk of Material Misstatement (p = 0.002) Audit plan revisions relative to the prior year’s clean audit (p = 0.019)

Conclusion If IS auditors are influenced by irrelevant evidence when they assess KRFs as they are in the highly simplified setting presented in this study, professional service firms may want to monitor this as a potential problem so that they can minimize the likelihood of audit failure. Next steps - Debriefing

An Interactive Discussion: Contemporary Research on IS Auditors and Automated Controls Dr. Daniel Selby