1. INTOSAI IT Audit IT Methods Awareness

2. Outline Scope Overview It Methods Methods Description Methods Usage
Audit Reporting

3. Scope It Methods Described For: Project Selection, Control, Evaluation
Systems Development
Systems Acquisition
Enterprise Architecture Development
Security Assessment

4. Overview Methods Listed Here Are Generally Accepted in The Community
Methods Assess or Prescribe “What” Must Be Done Not “How” to Accomplish Activity
Methods Provide a Framework to Audit It Activity

5. It Methods

6. Methods Description
Module 1
Project Selection, Control, Evaluation

7. Project Selection, Control, Evaluation
Wisely Managed Investments in It Can Improve Organizational Performance
Internet and Local Area Networks Enable Data Sharing and Research
Data Warehouse Permits Organizations to Discover Unknown Fiscal or Physical Resources

8. Project Selection, Control, Evaluation
However, Along With the Potential to Improve Organizations, It Projects Can Become Risky, Costly, Unproductive Mistakes
In Response, Gao Developed Guidance, That Provides a Method for Evaluating and Assessing How Well an Agency Is Selecting and Managing Its It Resources

9. Project Selection, Control, Evaluation
The Select/control/evaluate Model Has Become a Central Tenet of the It Investment Management Approach

10. Project Selection, Control, Evaluation
During the Selection Phase the Organization
Selects Those It Projects That Will Best Support Its Mission Needs and
Identifies and Analyzes Each Project’s Risks and Returns Before Committing Significant Funds to a Project.

11. Project Selection, Control, Evaluation
During the Control Phase the
Organization Ensures That, As Projects Develop, the Project Is Continuing to Meet Mission Needs at Expected Levels of Cost and Risk
If the Project Is Not Meeting Expectations Steps Are Taken to Address the Deficiencies

12. Project Selection, Control, Evaluation
Lastly, During the Evaluation Phase,
Actual Versus Expected Results Are Compared to
Assess the Project’s Impact on Mission Performance,
Identify Any Changes or Modifications to the Project That May Be Needed, and
Revise the Investment Management Process Based on Lessons Learned

13. Project Selection, Control, Evaluation
Gao’s Information Technology Investment Model (Itim) Model Is Comprised of Five Stages of Maturity
Each Stage Builds Upon the Lower Stages and Enhances the Organization’s Ability to Manage Its It Investment Stages

14. Project Selection, Control, Evaluation
Five Stages Of Investment Maturity

15. Project Selection, Control, Evaluation
Progressing Through the ITIM Stages of Maturity

16. Project Selection, Control, Evaluation
Itim Is a Tool for Assessing the Maturity of an Organization
An Itim Assessment Can Be Conducted for an Entire Organization or For One of Its Lower Divisions
Itim Is Applicable to Organizations of Different Sizes

17. Project Selection, Control, Evaluation
Itim Allows Auditors to Assesses the Maturity of Organizations to Manage Investments
Itim Provides a Maturity Stage or “Level” for an Organization
Each Maturity Stage or “Level” Has Required Practices or Activities

18. Project Selection, Control, Evaluation
ITIM Required Processes

19. Project Selection, Control, Evaluation
Applying the Model Requires Assessing
Critical Processes, Such As the Processes Used to Create an It Investment Portfolio
Core Elements, (Purpose, Organizational Commitment, Prerequisites, Activities, and Evidence of Performance)

20. Questions / Discussion
Comments
Discussion
Etc.

21. Methods Description
Module 2
Systems Development

22. Systems Development Systems Development Includes Activities Such As
Project Management,
Requirements Management,
Configuration Management
Software Development, Testing, Etc.

23. Systems Development
Many Organizations Rely on Software-intensive Systems to Perform Their Missions
Software Quality Is Governed by the Quality of the Processes Used To Develop the Software
(Provide Reference)

24. Systems Development
The Software Engineering Institute Has Developed a Number of Models That Facilitate Assessing the Maturity of Organizations Developing Software
The Models Are Called Capability Maturity Models (Cmm)

25. Systems Development What Is the Cmm?
An Ordered Collection of Practices for the Acquisition, Development or Maintenance of Systems
Ordered by “Key Process Area”
Practices Determined by the Community Through Broad Peer Reviews
Defines the Stages Through Which Organizations Evolve As They Improve Their Acquisition Process
Identifies Key Priorities, Goals and Activities on the Road to Improving an Organization's Capability to Do Its Job

26. Systems Development The Cmm Provides a Framework for
Identifying an Organization’s Process Strengths and Weaknesses
Assisting an Organization Develop a Structured Plan for Process Improvement

27. Systems Development Who Uses the Sw-cmm?
Organizations That Develop or Maintain Products That Contain Software
Organizations Who Want to Improve Their Software Development Processes
Audit Organizations Who Want to Assess the Maturity Of Organizations Developing or Maintaining Software Products

28. Systems Development The Cmm Is Structured Into
Five Maturity Levels
Each Level Has Key Process Areas (Kpa)
Each Kpa Has Goals
Goals Require Certain Activities Be Performed
Management Provides Support and Verifies That Activities Are Being Performed

29. Systems Development

30. Systems Development The Five Levels Are
1. Initial: The Software Process Is Characterized As Ad Hoc and Few Processes Are Defined
2. Repeatable: Basic Project Management Processes Are Established; Improvement Activities Are Begun
3. Defined: Software Processes Are Documented and Standardized; All Projects Use an Approved, Tailored Version of the Organization’s Standard Software Processes

31. Systems Development The Five Levels Are (Contd.)
4. Managed/quantitative: Detailed Measures of the Software Processes, Products, and Services Are Collected; the Software Processes and Products Are Quantitatively Measured and Controlled
5. Optimizing: Continuous Process Improvement Is Enabled by Quantitative Feedback From the Process and From Piloting Innovative Ideas and Technologies

32. Systems Development
Software CMM Levels and KPAs

33. Systems Development Cmm Common Features Commitment To Perform
Ability To Perform
Activities
Measurement & Analysis
Verification

34. Systems Development Commitment To Perform
Describes What an Organization Must Do to ‘Set the Stage’ for Process Improvement / Implementation
Involves Establishing Policy
Assigning Responsibility

35. Systems Development Ability To Perform
Describes the Preconditions That Must Be Present to Facilitate Process Improvement / Implementation
Assignment of Duties to Groups
Providing Trained or Experienced Personnel
Ensuring Adequacy of Resources

36. Systems Development Activities
Describe the Activities, Roles, and Procedures That Are Necessary to Implement the Key Process Area
Requires Formal and Informal Planning Documents
Requires Formally Documented Procedures
Requires (Depending on Kpa) Coordination With Other Affected Groups, Tracking Contractor Performance, Etc.

37. Systems Development Measurement & Analysis
Describes the Practices That Must Be Accomplished to Enable the Group to Track the Status of the Kpa
Effort & Funds Expended by the Project Team in Conducting Its Activities
Tracking Their Schedule and Progress (for Developing Formal Plans, Requirements, Etc.)

38. Systems Development Verification
Describes the Practices That Must Be Performed to Ensure That Project and Senior Management Oversee the Activities of the Group
Includes Periodic or As Needed
Project Level Reviews
Senior Management Level Reviews

39. Systems Development
Example From Model

40. Questions / Discussion
Comments
Discussion
Etc.

41. Methods Description
Module 3
Systems Acquisition

42. Systems Acquisition Systems Acquisition Includes Activities Such As
Project Management,
Requirements Management,
Solicitation, Contractor Tracking
Evaluation, Risk Management, Etc.

43. Systems Acquisition
Many Organizations Rely on Software-intensive Systems to Perform Their Missions
Organizations Have Been Increasingly Contracting Out for Software or Engineering Services

44. Systems Acquisition
The Software Engineering Institute Has Developed a Number of Models That Facilitate Assessing the Maturity of Organizations That Acquire Software or Systems
The Models Are Called Capability Maturity Models (Cmm)

45. Systems Acquisition
Just As For Software Development There Is the Sw-cmm (or Just Cmm)
For Assessing or Improving Acquisition Related Activities, The Sei Has Developed the Software Acquisition Capability Maturity Model (Sa-cmm)

46. Systems Acquisition Who Uses The Sa-cmm?
Organizations That Acquire or Support Acquisition of Products That Contain Software, Including Software Support and Maintenance
Organizations That Are Responsible for Acquisition Life Cycle From Requirements Development Through System Delivery and Support
Audit Institutions That Want To Assess How Effectively Software or Services Are Being Acquired

47. Systems Acquisition The Sa-cmm Is Also Structured Into
Five Maturity Levels
Each Level Has Key Process Areas (Kpa)
Each Kpa Has Goals
Goals Require Certain Activities Be Performed
Management Provides Support and Verifies That Activities Are Being Performed

48. Systems Acquisition

49. Systems Acquisition The Five Levels Are
1. Initial: The Software Process Is Characterized As Ad Hoc and Few Processes Are Defined
2. Repeatable: Basic Project Management Processes Are Established; Improvement Activities Are Begun
3. Defined: Software Processes Are Documented and Standardized; All Projects Use an Approved, Tailored Version of the Organization’s Standard Software Processes

50. Systems Acquisition The Five Levels Are (Contd.)
4. Managed/quantitative: Detailed Measures of the Software Processes, Products, and Services Are Collected; the Software Processes and Products Are Quantitatively Measured and Controlled
5. Optimizing: Continuous Process Improvement Is Enabled by Quantitative Feedback From the Process and From Piloting Innovative Ideas and Technologies

51. Systems Acquisition

52. Systems Acquisition
Example From Model

53. Questions / Discussion
Comments
Discussion
Etc.

54. Methods Description
Module 4
Enterprise Architecture Development

55. Enterprise Architecture Development
An Enterprise Architecture (Ea) Establishes the Agency-wide Roadmap to Achieve an Agency’s Mission
Eas Are “Blueprints” for Systematically and Completely Defining an Organization’s Current (Baseline) or Desired (Target) Environment

56. Enterprise Architecture Development
Eas Are Essential for Evolving Information Systems and Developing New Systems That Optimize Mission Value
For Eas to Be Useful and Provide Business Value, Their Development, Maintenance, and Implementation Should Be Managed Effectively

57. Enterprise Architecture Development
An Ea Is A Strategic Information Asset, Which Documents the Mission And,
The Information , Technology, and the Processes Required to Perform the Mission
An Ea Includes a Baseline Architecture, Target Architecture, and a Sequencing Plan

58. Enterprise Architecture Development
Eas Typically Include
Business or Operational Architecture
Work Processes and Locations
Information or Data Architecture
Data or Information Needed to Perform Business
Technical or Systems Architecture
Technology Standards, It Systems Description

59. Enterprise Architecture Development
EA Process
Sections Refer to:
A Practical Guide To Federal Enterprise Architecture

60. Enterprise Architecture Development
Obtain Executive Buy-in and Support
Ensure Agency Head Buy-in and Support
Issue an Executive Enterprise Architecture Policy
Obtain Support From Senior Executives and Business Units
Establish Management Structure and Control
Establish a Technical Review Committee
Establish a Capital Investment Council
Establish an Ea Executive Steering Committee
Appoint Chief Architect
Define an Architecture Process and Approach
Define the Intended Use of the Architecture
Define the Scope of the Architecture
Determine the Depth of the Architecture

61. Enterprise Architecture Development
Develop the Baseline Enterprise Architecture
Collect Information
Generate Products and Populate Ea Repository
Develop the Target Enterprise Architecture
Develop the Sequencing Plan
Identify Gaps
Define and Differentiate Legacy, Migration, and New Systems
Planning the Migration

62. Enterprise Architecture Development
Use the Enterprise Architecture
Integrate the Ea With Cpic and Slc Processes
Train Personnel
Establish Enforcement Processes and Procedures
Maintain the Enterprise Architecture As the Enterprise Evolves
Reassess the Enterprise Architecture Periodically
Manage Products to Reflect Reality

63. Questions / Discussion
Comments
Discussion
Etc.

64. Methods Description
Module 5
Security Assessment

65. Security Assessment
Information and the Systems That Process It Are Among the Most Valuable Assets of Any Organization
Adequate Security of These Assets Is a Fundamental Management Responsibility

66. Security Assessment Agency Must
Ensure That Systems and Applications Provide Appropriate Confidentiality, Integrity, and Availability
Protect Information Commensurate With the Level of Risk and Magnitude of Harm Resulting From Loss, Misuse, Unauthorized Access, or Modification

67. Security Assessment Agencies Must Plan for Security
Ensure Appropriate Officials Are Assigned Security Responsibility
Authorize (or ”Certify") System Processing Prior to Operations and Periodically As Necessary

68. Security Assessment
The Federal It Security Assessment Framework Provides a Method for
Determining the Current Status of Their Security Programs
Establishing a Target for Improvements Where Necessary
The Framework May Be Used to Assess the Status of Security Controls

69. Security Assessment
The Framework Comprises Five Levels to Guide Assessments of Security Programs and Prioritization of Improvement Efforts

70. Security Assessment Level 1 Documented Policy
Level 2 Documented Procedures
Level 3 Implemented Procedures and Controls
Level 4 Tested and Reviewed Procedures and Controls
Level 5 Fully Integrated Procedures and Controls

71. Security Assessment Level 1 of the Framework Includes
Formally Documented and Disseminated Security Policy
Level 2 of the Framework Includes
Formal, Complete, Documented Procedures for Implementing Policies Established at Level One

72. Security Assessment Level 3 of the Framework Includes
Security Procedures and Controls That Are Implemented
Level 4 of the Framework Includes
Routinely Evaluating the Adequacy and Effectiveness of Security Policies, Procedures, and Controls
Level 5 of the Framework Includes
A Comprehensive Security Program That Is an Integral Part of an Agency’s Organizational Culture

73. Security Assessment

74. Security Assessment

75. Security Assessment

76. Security Assessment

77. Security Assessment

78. Questions / Discussion
Comments
Discussion
Etc.

79. Methods Usage

80. Methods Usage Most Methods Have Specific Activities to Be Performed
Can Be Applied to Specific Projects
Need a Team of About Auditors
Requires Training or Understanding of the Method

81. Methods Usage Since Methods Have Specific Activities
Questionnaires Can Be Generated
Results Can Be Tabulated
Analysis Can Be Formed Quickly From the Results

82. Methods Usage
The Sw-cmm and Sa-cmm Methods Require the Audit Lead Be Specifically Trained

83. Methods Usage
Sample Data Collection Instrument

84. Audit Reporting

85. Audit Reporting Audit Report Can Be Briefing Slides or Full Reports
Briefing Slides Can Contain Both Summary or Detailed Results

86. Audit Reporting
Sample SA-CMM Summary Results

87. Audit Reporting
Sample SA-CMM Acquisition Risk Management Detailed Results

88. Questions / Discussion
Comments
Discussion
Etc.

89. Contacts Keith Rhodes Madhav Panwar Phone 1 202 512 6412
Madhav Panwar
Phone