1. the 22 nd meeting of the INTOSAI Working Group on IT Audit (WGITA) KPI Project Final Report — Key Performance Indicators Methodology for Auditing IT Programs Amy Young, IT Center China National Audit Office April, 2013Lithuania

2. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 2 CNAO Framework 1. Background 2. Project objectives 3. Project progress 4. Project products

3. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 3 CNAO 1. Background the 19th of the INTOSAI Working Group on IT Audit, April 2010 – three proposed projects: Development of IT Performance Indicators Performance measures of IT Solutions implemented in government organizations Index System about IT Performance Audit

4. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 4 CNAO 1. Background the 19th of the INTOSAI Working Group on IT Audit, April 2010 – The above three projects were put into a new one. “Key Performance Indicator Methodology for Auditing IT Programs”. – SAI China volunteered to be the team leader. – Team members : SAI Bhutan, China, Ecuador, Japan, Kuwait, Malaysia, Pakistan, Poland, Russia and USA.

5. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 5 CNAO Framework 1. Background 2. Project objectives 3. Project progress 4. Project products

6. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 6 CNAO 2. Project objectives two principal products – a set of specific and measurable IT-related indicators – guideline the ideas of KPI application feasible KPI evaluation methodology

7. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 7 CNAO 2. Project objectives Analysis, conclusions and decision making based on results of performance measurement exchange views, know-how, and information

8. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 8 CNAO Framework 1. Background 2. Project objectives 3. Project progress 4. Project products

9. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 9 CNAO 3. Project progress Milestones – By the end of 2011, the beta database of KPI indicators. – By the end of 2012, the draft Guideline. – In 2013, the final Guideline and KPI Database.

10. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 10 CNAO 3. Project progress the first team meeting on Apr 16, 2010 in Beijing. a new name for the project, “Key Performance Indicators Methodology for Auditing IT Programs”. the minutes of the first team meeting.

11. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 11 CNAO 3. Project progress SAI Russia, Japan, USA, Kuwait and Ecuador explained their own understanding about the project from April to June, 2010. The project initiation document to Chair India in July 2010. SAI Kuwait submitted the beta database of KPI indicators on Dec 29, 2010.

12. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 12 CNAO 3. Project progress The original framework based on SAI Kuwait’s and China’s PKI indicators in Feb, 2011. SAI Kuwait, Bhutan and Japan gave the comments in Feb and Mar, 2011. the Second team meeting on Apr 16, 2011 in South Africa. the minutes of the second team meeting.

13. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 13 CNAO 3. Project progress SAI Kuwait submitted the categorized KPI beta database on August 11, 2011. SAI Bhutan gave comments on August 18,2011. SAI China composed the beta indicators and send it to the team members for comments on November 30, 2011. SAI Japan and Kuwait gave the comments in December,2011.

14. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 14 CNAO 3. Project progress In the 21st meeting in Kuala Lumpur, Malaysia in January 2012, the team made the report about the KPI project. SAI China sent the content framework of guideline to the team members for comments on April 9, 2012. In April and May, SAI Kuwait and Bhutan gave the comments and SAI Kuwait applied to compose Part 1 and Part 2. SAI China began to compose the draft guideline in June, 2012.

15. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 15 CNAO 3. Project progress SAI China delivered the draft guideline for comments among the team members on December 30, 2012. SAI Japan, Kuwait and USA feedback in Jan and Feb 2013. In early March 2013, SAI China send the final guideline v1 to Chair India for comments among all the WGITA members.

16. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 16 CNAO 3. Project progress SAI Bangladesh, Brazil, Iraq and Russia gave the comments. Based on the comments, SAI China made the final guideline v1.1. Comments from SAI Malaysia, Lithuanian and Qatar. In Apr 2013, the final guideline V1.2 was delivered for approval in the 22nd meeting of WGITA.

17. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 17 CNAO 3. Project progress Communication Management Strategy – mainly based on email through Internet – email contact list for the team members – sometimes discuss trough telephone

18. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 18 CNAO Framework 1. Background 2. Project objectives 3. Project progress 4. Project products

19. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 19 CNAO 4. Project products Criteria for the indicator – The indicators: be widely accepted and measurable – the data source of the evaluating indicators : be accessible and analyzable – the calculation methods of the evaluating indicators: be relatively simple

20. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 20 CNAO 4.1- KPI database 11 parts, 367 indicators with three-level. 1. Decision (15 indicators) – Compliance with the laws – Feasibility study – Participation in decision 2. Requirement analysis (13 indicators) – Organization target – Core business coverage – Response/change

21. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 21 CNAO 4.1- KPI database 3. Design / Planning (45 indicators) – Requirement coverage – Time limit – Capacity Planning and Resource Provisioning – Cost estimation – IT risks

22. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 22 CNAO 4.1- KPI database 4. Procurement/Development (48 indicators) – Selection for partner or supplier – Cost control – Process Control – Code control – Outsourcing – Quality Control – Testing – Training – Upgrading

23. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 23 CNAO 4.1- KPI database 5. Product (26 indicators) – User satisfaction – Price – Delivery – Performance – Integration – Technology applicability

24. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 24 CNAO 4.1- KPI database 6. Maintenance (125 indicators) – Follow the management rules – Incident management – System Usability – Availability – Maintenance cost – Website – Monitoring – Change Management – Data Center

25. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 25 CNAO 4.1- KPI database 7. Security (26 indicators) – Security plan – Identity management – User account management – Security testing monitoring – Security incident definition – Malicious software prevention – Network security

26. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 26 CNAO 4.1- KPI database 8. Backup and Disaster Recovery (14 indicators) – Backup plan – Recovery plan – Backup operation management – Recovery operation management 9. Service (32 indicators) – Service request – Service Response – Service Satisfaction

27. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 27 CNAO 4.1- KPI database 10. Effectiveness (19 indicators) – Coverage of the core business – Benefit – Internal Management optimizing – Public Service 11. Others (4 indicators)

28. the 22nd meeting of the INTOSAI Working Group on IT Audit (WGITA) 28 CNAO 4.2 - Guideline Part 1. Preface Part 2. Audit Plan Part 3. Audit Implementation Part 4. Audit report Reference : INTOSAI training materials, ISACA, COBIT, ITIL and others.

29. the 22 nd meeting of the INTOSAI Working Group on IT Audit (WGITA) Thanks for your attention! China National Audit Office April, 2013Lithuania