1. Improving System Development Project Success: How Internal Auditors Add Value Through Process Involvement & Measurement Glen L. Gray, California State University, Northridge, USA Anna H. Gold, VU University, The Netherlands Christopher G. Jones, California State University, Northridge, USA David W. Miller, California State University, Northridge, USA EAA 2011: Rome, Italy

2. 2 Overview Background –SDP failures and the dismal rate of SDP success –Control issues Research objective –Internal auditor’s role in SDP success Research questions, methods, and summary of findings

3. 3 Many SDP failures… December 2002: McDonald’s abandons major project after two years. Cost: US$170 million November 2004: Sainsbury (UK super- market chain) writes off a £260 million IT investment in its supply chain February 2008: Los Angeles Unified School District’s faulty US$95 million payroll system goes live. For months afterward, thousands are overpaid, underpaid, or not paid at all. November 2010: FBI spent $405 million of the $451 million budgeted for new Sentinel case-management system, but, as of September, it’s two years behind schedule and $100 million over budget

4. 4 Few SDP Successes… Standish Group Standish Group [2009]

5. 5 Costly Conundrum How do failing or challenged projects go undetected? Where were the ‘red flags’? –Missed, dismissed, or ignored all together? Who’s responsible for monitoring the controls and raising these red flags?

6. 6 Research Objective To explore how internal auditors currently do and potentially can provide value-added support to proactively help identify and monitor system development project controls to either:currently –Help get these projects back on track toward success or –Stop projects when the investment in the projects is still relatively low

7. 7 Post-SOX Changes? Pre-SOX: internal auditors usually came into a system development project after the project was completed to evaluate the internal controls—bayoneting the wounded Post SOX: internal auditors are more frequently active members of major system development projects, but— –auditor focuses on controls for the specific processes being automated, not the system development controls Gray [2004, 2007]

8. 8 Research Questions RQ1: When and how should internal auditors become involved in SDPs? RQ2: For which factors critical to system success can internal auditors add the most value? RQ3: What metrics should be used to monitor SDPs?

9. 9 Mixed-mode Research Method 1.Review IS and internal auditing literature CSFs and CFFs 2.Conduct internal auditor focus groups exploring RQ1 – RQ3. Qualitative 3.Develop CSF taxonomy from an internal auditing perspective Qualitative 4.Survey a sample of The IIA membership Quantitative

10. Critical Success Factors Literately, hundreds of success/failure factors –However, many different ways to say same things From both professional and academic literature Mostly opinions/observations vs. rigors analysis Mostly not stated as measurable factor/metric (e.g., adequate user involvement) Our next task: reduce factors to manageable set. 10

11. Critical Success Factor Taxonomy OrganizationProject Management Externalities People 11

12. Critical Success Factors Project Management 1.Systems Development Methodology 2.Quality Assurance 3.Change Management 4.Monitoring SDP Process 5.Financial Management 6.Tools and Infrastructure 7.Agile Optimization Project 8.System Requirements 9.Systems Interoperability People 10.Executive Support 11.Project Personnel 12.Project Management Expertise 13.Conflict Management Organization 14.User Involvement 15.Business Alignment Externalities 16.Vendor Relationship Management 12

13. Summary of Findings (1) RQ 1 Internal Auditor’s Role –Waiting until post-implementation review is too late. 13 Greenberg & Murphy, 1989

14. Summary of Findings (2) RQ 1 Internal Auditor’s Role –It’s OK to invite yourself to the party. 14

15. Summary of Findings (3) RQ 2 Where Internal Auditors Add Value –Some CSFs more critical than others. Criticality transforms. 15 Internal Auditing Adds Value Contributes to Project Success Critical Success FactorRankMeanRankMean Quality assurance (PM) 14.0454.54 Change management (PM) 24.0164.54 Monitoring SDP (PM) 33.93104.46 System requirements (P) 43.8514.72 Systems development methodology (PM) 53.8034.60

16. Summary of Findings (4) RQ 3 Monitoring SDP Success –Metrics abound but dashboards uncommon. –Conventional wisdom evolving. 16 Old Conventional Wisdom New Conventional Wisdom Internal auditing should primarily focus on application controls Internal auditing should also focus on SDP controls

17. Internal Auditor Involvement Three basic approaches to the auditor’s involvement in SDPs: –Auditor approach would be the more traditional auditing function by monitoring the SDP on a milestone basis to monitor how the project is progressing on behalf of management and the board. –Consultant approach where the internal auditors are advising the SDP team on an as-needed basis regarding controls. –Embedded approach where internal auditors are integrated in the SDP team functioning as the control experts. 17

18. Internal Auditor Involvement 18 [Large] Internal Audit Department Size [Small] Embedded Consultant Auditor [Audit] IT Skill Portfolio [IT]

19. The Final Survey Question Q: What is the one best way for internal auditors to improve the success rate of SDPs? A: “Be included, be involved, and participate regularly in the process from project inception.” 19

20. Questions? Thank You! Grazie Mille! Glen L. Gray [glen.gray@csun.edu] Anna H. Gold [a.h.gold@vu.nl] Christopher G. Jones [christopher.jones@csun.edu] David W. Miller [david.w.miller@csun.edu ]