KPI Project Final Report

1
KPI Project Final Report Key Performance
Indicators Methodology for Auditing IT Programs

• Amy Young, IT Center
• China National Audit OfficeApril, 2013 Lithuania

2
Framework

• 1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products

3
1. Background

• the 19th of the INTOSAI Working Group on IT Audit
  , April 2010
• three proposed projects
• Development of IT Performance Indicators
• Performance measures of IT Solutions implemented
  in government organizations
• Index System about IT Performance Audit

4
1. Background

• the 19th of the INTOSAI Working Group on IT
  Audit , April 2010
• The above three projects were put into a new one.
  Key Performance Indicator Methodology for
  Auditing IT Programs.
• SAI China volunteered to be the team leader.
• Team members SAI Bhutan, China, Ecuador, Japan,
  Kuwait, Malaysia, Pakistan, Poland, Russia and
  USA.

5
Framework

• 1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products

6
2. Project objectives

• two principal products
• a set of specific and measurable IT-related
  indicators
• guideline
• the ideas of KPI application
• feasible KPI evaluation methodology

7
2. Project objectives

• Analysis, conclusions and decision making based
  on results of performance measurement
• exchange views, know-how, and information

8
Framework

• 1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products

9
3. Project progress

• Milestones
• By the end of 2011, the beta database of KPI
  indicators.
• By the end of 2012, the draft Guideline.
• In 2013, the final Guideline and KPI Database.

10
3. Project progress

• the first team meeting on Apr 16, 2010 in
  Beijing.
• a new name for the project, Key Performance
  Indicators Methodology for Auditing IT Programs.
  
• the minutes of the first team meeting.

11
3. Project progress

• SAI Russia, Japan , USA , Kuwait and Ecuador
  explained their own understanding about the
  project from April to June, 2010.
• The project initiation document to Chair India in
  July 2010.
• SAI Kuwait submitted the beta database of KPI
  indicators on Dec 29, 2010.

12
3. Project progress

• The original framework based on SAI Kuwaits and
  Chinas PKI indicators in Feb, 2011.
• SAI Kuwait, Bhutan and Japan gave the comments
  in Feb and Mar, 2011.
• the Second team meeting on Apr 16, 2011 in South
  Africa.
• the minutes of the second team meeting.

13
3. Project progress

• SAI Kuwait submitted the categorized KPI beta
  database on August 11, 2011.
• SAI Bhutan gave comments on August 18 ,2011.
• SAI China composed the beta indicators and send
  it to the team members for comments on November
  30, 2011.
• SAI Japan and Kuwait gave the comments in
  December,2011.

14
3. Project progress

• In the 21st meeting in Kuala Lumpur, Malaysia in
  January 2012, the team made the report about the
  KPI project.
• SAI China sent the content framework of
  guideline to the team members for comments on
  April 9, 2012.
• In April and May, SAI Kuwait and Bhutan gave the
  comments and SAI Kuwait applied to compose Part 1
  and Part 2.
• SAI China began to compose the draft guideline in
  June, 2012.

15
3. Project progress

• SAI China delivered the draft guideline for
  comments among the team members on December 30,
  2012.
• SAI Japan, Kuwait and USA feedback in Jan and Feb
  2013.
• In early March 2013, SAI China send the final
  guideline v1 to Chair India for comments among
  all the WGITA members.

16
3. Project progress

• SAI Bangladesh, Brazil, Iraq and Russia gave the
  comments.
• Based on the comments, SAI China made the final
  guideline v1.1.
• Comments from SAI Malaysia, Lithuanian and Qatar.
• In Apr 2013, the final guideline V1.2 was
  delivered for approval in the 22nd meeting of
  WGITA .

17
3. Project progress

• Communication Management Strategy
• mainly based on email through Internet
• email contact list for the team members
• sometimes discuss trough telephone

18
Framework

• 1. Background
• 2. Project objectives
• 3. Project progress
• 4. Project products

19
4. Project products

• Criteria for the indicator
• The indicators be widely accepted and measurable
• the data source of the evaluating indicators be
  accessible and analyzable
• the calculation methods of the evaluating
  indicators be relatively simple

20
4.1- KPI database

• 11 parts, 367 indicators with three-level.
• 1. Decision (15 indicators)
• Compliance with the laws
• Feasibility study
• Participation in decision
• 2. Requirement analysis (13 indicators)
• Organization target
• Core business coverage
• Response/change

21
4.1- KPI database

• 3. Design / Planning (45 indicators)
• Requirement coverage
• Time limit
• Capacity Planning and Resource Provisioning
• Cost estimation
• IT risks

22
4.1- KPI database

• 4. Procurement/Development (48 indicators)
• Selection for partner or supplier
• Cost control
• Process Control
• Code control
• Outsourcing
• Quality Control
• Testing
• Training
• Upgrading

23
4.1- KPI database

• 5. Product (26 indicators)
• User satisfaction
• Price
• Delivery
• Performance
• Integration
• Technology applicability

24
4.1- KPI database

• 6. Maintenance (125 indicators)
• Follow the management rules
• Incident management
• System Usability
• Availability
• Maintenance cost
• Website
• Monitoring
• Change Management
• Data Center

25
4.1- KPI database

• 7. Security (26 indicators)
• Security plan
• Identity management
• User account management
• Security testing monitoring
• Security incident definition
• Malicious software prevention
• Network security

26
4.1- KPI database

• 8. Backup and Disaster Recovery (14 indicators)
• Backup plan
• Recovery plan
• Backup operation management
• Recovery operation management
• 9. Service (32 indicators)
• Service request
• Service Response
• Service Satisfaction

27
4.1- KPI database

• 10. Effectiveness (19 indicators)
• Coverage of the core business
• Benefit
• Internal Management optimizing
• Public Service
• 11. Others (4 indicators)

28
4.2 - Guideline

• Part 1. Preface
• Part 2. Audit Plan
• Part 3. Audit Implementation
• Part 4. Audit report
• Reference INTOSAI training materials, ISACA,
  COBIT, ITIL and others.

29
Thanks for your attention!
China National Audit OfficeApril, 2013 Lithuania