Beyond Checklists: How to Effectively Audit Change Controls

1
Beyond ChecklistsHow to Effectively Audit
Change Controls

• June 2006

2
Acknowledgements

• IIA Change Management GTAG Change and Patch
  Management Controls Critical for Organizational
  Success
• Jay Taylor, General Director, IT Audit, General
  Motors Corporation
• Julia Allen, Carnegie Mellon University, Software
  Engineering Institute
• Glenn Hyatt, IT Security Manager, GMAC Mortgage
• Gene Kim, CTO, Tripwire, Inc.

3
Agenda

• Discuss common misconceptions about change
  management
• Discuss the importance of change controls
• Identify how to effectively and efficiently
  assess change management processes
• Compare characteristics of high and low
  performers according to the IT processes
  institute
• Discuss how to improve change management in 5
  easy steps
• Present results from the IT Controls Performance
  Study a 2.5 year quantitative study

4
Things You May Not Know About Change Management

• The only acceptable number of unauthorized
  changes in a change management system is

ZERO!

• Deficiencies in change management were widely
  cited in many SOX 404 reviews failure to
  address them effectively could elevate a
  significant deficiency to a material weakness
• Even high performing organizations may only be
  one change away from being a low performer

5
Things You May Not Know About Change Management

• High performing IT organizations would scream if
  someone threatened to take their change
  management process away
• Well controlled change management does not slow
  your IT organization down
• Some great change management metrics are now
  being used by IT and auditors
• Knowledgeable people can very quickly assess the
  health of an IT change management process

6
Things You May Not Know About Change Management

• Patches are just another type of IT change, and
  most best in class IT organizations patch less
  often than average IT organizations
• Changing a poor change management process into a
  great one is easier than you think sometimes,
  you can see significant improvement in under
  three months
• COBIT and ITIL used alone are not complete enough
  frameworks for defining effective change
  management processes in fact, even in
  combination, there are still big gaps

7
Why Should Auditors Spend More Time on Change
Controls?

• Management attention around the importance of IT
  controls is increasing
• Heightened focus from Audit Committee and Senior
  Management
• Internal auditors responsible for providing IT
  controls assurance to the Audit Committee and
  Senior Management

8
Four Types Of Audit Evidence

• Discussion and surveys
• Control self-assessments, asking what people do,
  
• Observation
• Emergency change being made, seeing late projects
  and unplanned work
• Re-performance and other sampling
• Samples of change requests, change
  reconciliations, break/fix tickets
• Examination of source documents
• Failure to meet Service Level Agreements for the
  business

9
Checklist Shortcomings How IT Auditors Are
Hoodwinked

• Checklists are static
• Auditors cannot easily integrate and use
  information gained by IT managements answers to
  ask better and follow up questions
• Checklists may be old
• Considerable new research has been developed on
  high-performing change management processes
• A good auditor is a thinking auditor
• You want to hire really good IT auditors that can
  understand the process and hone in on risk areas
  and look for the root causes of the symptoms
  that are surfaced in the audit

10
What were/are people worried about?

• IT controls dominate the deficiencies,
  significant deficiencies, and material weaknesses
  identified through the S-O 404 assessment.

• The estimated percentage of deficiencies
  identified show IT controls accounting for the
  most (34 percent), followed distantly by revenue
  (13 percent), procure to pay (10 percent), and
  fixed assets (10 percent). 
• The estimated percentage of significant
  deficiencies identified again shows IT controls
  leading the way (23 percent), followed by
  financial reporting and close (14 percent),
  procure to pay (13 percent), and revenue (12
  percent).  
• The estimated percentages of material weaknesses
  identified include IT controls (27 percent),
  revenue (18 percent), taxes (11 percent), and
  financial reporting and close (10 percent).  

It is important to note that the results
presented here are based on self-reporting by the
companies that participated in the survey.
Conclusions may be affected by the differing
methods companies use to report on various
elements of Sarbanes-Oxley compliance.
11
Change Management Failures Can Be Very Visible
12
Causal Factors of IT Downtime
System Outages 20 Incidents
Operator Error 60
5
Overall - 80 of System Outages are due to change
Security Related
Non-Security Related
15
Application Failure 20
Source IDC, 2004
13
Common Traits Of The Highest Performers

• Culture of change management
• Integration of IT operations and security
  processes via problem management and change
  management processes
• Processes that serve both organizational needs,
  as well as business objectives
• Highest rate of effective change (approved
  changes, change success rate)
• Culture of causality
• Highest service levels (MTTR, MTBF)
• Highest first fix rate (unneeded rework)
• Culture of compliance and continual reduction of
  operational variance
• Production configurations
• Highest level of pre-production staffing
• Effective pre-production controls
• Effective pairing of preventive and detective
  controls

Source IT Process Institute
14
Spectrum of Capability

• Continuously Improving
• lt5 of time spent on unplanned work

• Closed-Loop Process
• 15-35 of time spent on unplanned work

• Using Honor System
• 35-50 of time spent on unplanned work

• Reactive
• Over 50 of time spent on unplanned work

Effectiveness
Reactive
Using The Honor System
Closed-Loop Change Mgt
ContinuouslyImproving
Based on the IT Process Institutes Visible Ops
Framework
15
Comparing High- and Low-Performing IT
Organizations

• Auditor asks Change management is very
  important. How effective is our change management
  process here?

Ours is world class. We were even ready for
Sarbanes Oxley requirements when it came along
because all of the controls were already in
place. We have had to generate a few more
reports to show the control mappings, but were
in good shape.
We havent heard anything negative about our
change management process, especially not from
audit. We cant afford the overhead of a
burdensome process to fix something thats
already in place.
16
Comparing High- and Low-Performing IT
Organizations

• Auditor asks What number of unauthorized
  changes is acceptable here?

The only acceptable number of unauthorized
changes is zero. One rogue change can kill our
entire operation, and thats why we reconcile
changes daily. We trust, but verify.
Look, we dont get paid to not make changes.
Sometimes we need to fast-track the rules.
Thats how we really get work done here. Change
management is bureaucratic and you auditors just
want to slow things down.
17
Comparing High- and Low-Performing IT
Organizations

• Auditor asks How do you deal with change that
  happens outside our process?

We monitor for all change on an ongoing basis,
and we enforce reconciliation of all changes back
to an authorized, documented business
reason. Anything that cant be documented or
explained must be removed from the production
environment.
People know the policy and we expect them to
follow it. However, we dont usually know if
anything happens outside the process unless it
causes a problem. And lets just say I havent
had many complaints.
18
Metrics

• Number of changes authorized per week
• As measured by the change management log of
  changes
• Number of actual changes made per week
• As measured by detective controls such as
  monitoring software
• Number of unauthorized changes that circumvented
  the change process
• As measured by production changes that cannot be
  reconciled against authorized work
• Change failure rate
• As defined as unsuccessfully implemented changes
  (those that did not cause an outage, service
  impairment, or an episode of unplanned work) as a
  percent of actual changes made

Source IIA Change Management GTAG
19
Metrics, Continued

• Number of emergency changes (including patches),
• As defined by counting the number of changes that
  required an urgent approval during the week using
  the Change Review Board or Emergency Change
  process
• Percentage of time spent on unplanned work
• Planned work is time spent on authorized projects
  and tasks.
• Unplanned work includes break/fix cycles, rework,
  and emergency changes
• Unexplained outages and lots of unexplained
  changes
• The top leading indicators of risk when we look
  at an IT operation are poor service levels and
  unusual velocity of changes. -- Bill Philhower,
  Managing Director, IT Audit
• IT management is being awakened in the middle of
  the night regarding problems

Source IIA Change Management GTAG
20
Why Is Unplanned Work Such A Good Indicator?
of productionchanges
failed change or unauthorized changes
mean timeto repair
of time spenton unplanned work

X
X
Average 35-45 of OpEx spent on unplanned
work! Impact late projects, rework, compliance
issues, uncontrolled variance, etc
Source IIA Change Management GTAG
21
What Affects These Variables?
of productionchanges
mean timeto repair
of time spenton unplanned work
failed change or unauthorized changes

X
X
Behaviors that increase change success rate
Effective change testing Effective risk review
when approving changes Effective identification
of change stakeholders Effective change
scheduling
Behaviors that decrease MTTR Culture of
causality desire to rule out change first in
problem repair cycle Effective change
management process that can report on authorized
and scheduled changes Ability to distinguish
planned and unplanned outage events Effective
communications around scheduled changes
Effective monitoring of infrastructure for
production changes
Behaviors that reduce unauthorized changes
Culture of change management Management
ownership of change process Effective
monitoring of infrastructure with detective
controls to enforce change process Management
use of corrective action when change processes
are not followed
Source IIA Change Management GTAG
22
Look for Red Flags and Indicators

• Failure to complete projects and planned work
• Due to high amounts of firefighting and unplanned
  work
• Adversarial relationships between IT support
  staff and other parts of the business
• Business customers (internal or external) may
  complain of poor service quality, late delivery
  of functionality, etc.
• High IT staff turnover
• Due to sustained periods of urgent unplanned work
  and required heroics
• High amounts of time required for IT management
  to prepare for IT audits, and remediate the
  resulting findings
• High effort and unplanned work around audits
  indicates low degree of controls in daily
  operations

23
Good Questions To Ask IT

• Pick any piece of your infrastructure (router,
  server, firewall, application, etc.)
• If a change is made to this device right now, how
  will you know?
• How soon will you know it was made?
• How will you know if the change was good or bad?
• How long will that process take?
• What happens when the change is good?
• What happens when the change is bad?
• How do you verify that each change has been
  reconciled?
• How do you report on all of the above?
• Can you provide a historical report that provides
  an accounting for all changes in your
  environment?
• OK. Now show me...

24
Motivating and Sustaining Effective Change
Controls

• The IT Process Institute has been studying high
  performing organizations since 1999
• What is common to all the high performers?
• What is different between them and average and
  low performers?
• How did they become great?
• Answers to these questions has been codified in
  the Visible Ops Methodology
• The Visible Ops Handbook is now available from
  the ITPI

www.ITPI.org
25
Five Easy Steps To Improve Change Management
Processes

• Create the appropriate tone at the top,
  motivating the need for a culture of change
  management
• Support this with declaration from IT management
  Team, from here on out, the only acceptable
  number of unauthorized changes is
  zero. (remember hope is not a strategy)
• IT management must then develop and implement
  preventive and detective controls to help achieve
  and sustain this objective
• Regularly publish a list of authorized changes
• Regularly publish a list showing the
  reconciliations between all production changes
  and authorized work orders.
• Reports should show corrective actions taken
  because of unauthorized change
• Visible Ops describes this step as electrifying
  the fence

26
Five Easy Steps To Improve Change Management
Processes

• Continually monitor the number of unplanned
  outages
• Unplanned outages are an excellent indicator of
  unauthorized change and failures in change
  control
• Around unplanned outages, there should be a list
  of scheduled changes
• Reduce the number of risky changes by specifying
  well-defined and enforced change freeze and
  maintenance windows
• Reducing change to zero maximizes stability and
  productivity during production hours
• Again, unplanned outages serve as effective
  indicators that this change process is being
  circumvented

27
Five Easy Steps To Improve Change Management
Processes

• Use change success rate as a key IT management
  performance indicator
• When changes are unmanaged, unmonitored and
  uncontrolled, change success rates are typically
  less than 70
• Each failed change creates potential downtime,
  unplanned and emergency work, variance from
  plans, and business risk
• Increasing the change success rate requires
  effective preventive, detective and corrective
  controls.
• Use unplanned work as an indicator of
  effectiveness of IT management processes and
  controls
• High performing IT organizations spend less than
  5 of their time on unplanned work
• Average organizations often spend 45-55 of their
  time on unplanned (and urgent) activities.

28
Phase I Ungoverned Change
Unplanned work
(Unplanned work gt 100)
Failed changes orNum of unauth chgs
Change rate
time
29
Phase I Stabilized Patient
Unplanned work
Failed changes orNum of unauth chgs
Change rate
time
30
Benefit Improve Your Performance On Audits
Auditors perception of assurance
Control over change
Time spent on audit prep and liaising
of time spent on compliance activities
time
31
1. How Do You Electrify Fence?

• Must have a report that shows management that all
  production changes are authorized
• What changes map to authorized and approved work
  orders?
• What changes do not match expected changes?

32
2. What Happens When You Touch The Fence?

• All the high-performing IT organizations had some
  common processes for handling unauthorized change
• Making engineering team own the controls We
  just detected an unauthorized change you have
  four hours to retroactively document your cowboy
  change, otherwise we mobilize security.
• Deterrent and cultural controls E.g., wall of
  shame, two strikes and youre out
• Auditors love it when Management owns the
  controls
• Preventive policies
• Detective controls showing policies are being
  enforced
• Documentation of corrective actions, showing
  deterrent controls

33
Biggest Mistakes That IT Executives Make

• Not locking down change
• We cant we wont be able to get anything
  done.
• The business doesnt pay us to not make
  changes.
• Not electrifying fence
• We dont need to we trust our own people.
• Our people are professionals and dont need
  constant micromanagement.
• Not tackling culture issues
• Technology or process whiteboarding is easier
  to justify and implement than tackling people and
  culture issues

34
The 3 Cs and Enforcing Change Policy

• Culture
• Tone at the top
• IT Control starts with the CIO
• Defined processes
• Cant enforce what is not defined
• Controls
• All change must be auditable.
• All change must be authorized.
• All unauthorized change must be investigated.
• Credibility
• Accountability consequence
• What happens when someone goes around the
  process?
• Measured improvement
• Management by fact, not by faith

35
Greatness Is Possible Effectiveness and
EfficiencyServer/Sysadmin Ratio (2003 earlier
study)
High performers were not only effective, but also
efficient. They had server/sysadmin ratios
greater than 1001. FYI To compare, Google,
Akamai and other massively distributed systems
are around 80001.
36
Survey Identifies Foundational IT Controls

• Problem Statement
• Which IT processes and controls are the most
  important?
• Which have the highest rate of return?
• Where do you start?
• Conducted by the IT Process Institute
• Carnegie Mellon Software Engineering Institute
• Florida State University College of Information
• Survey conducted between August and October 2005
• 98 respondents

37
How Science Can Help Lean Manufacturing

• In the 1980s, a group of MIT researchers
  benchmarked the major auto manufacturing plants
  in the U.S., Japan, and Europe
• High performing plants shipped products with
• One-half the defects, using
• One-half the floor space, utilizing
• One-half the required staff, with
• One-half the cycle time, needing
• One-half the inventory
• Their research approach
• They studied the high performers
• Hypothesized how those processes led to their
  extraordinary results
• Conducted benchmarks of all of the participants
• Analyzed the results of the top performers
• Identified correlations and causal relationships
  between management behaviors and results (Roos,
  Womack, Jones1991)

38
ITPI Survey Demographics
39
ITPI Survey Controls Selection
1
The 6 leading BS15000 areas within ITIL that are
conjectured to be where to start were
selected. These were in the areas of Access,
Change, Configuration, Resolution, Release,
Service Levels
2
The 63 COBIT control objectives related to these
six areas were then selected for use in the
survey.
Source COBIT, IT Governance Institute/ISACA
40
ITPI Survey The 63 IT Controls
The resulting controls that we selected were in
the following control categories Access
Controls 17 controls Change Controls 13
controls Configuration Controls 7 controls
Release Controls 6 controls Service Level
Controls 8 controls Resolution Controls 12
controls
41
ITPI Survey Performance Measure Selection

• Measures that reflect the primary goals of IT
• Build and deliver projects for the business
• Operate and maintain existing IT assets

42
ITPI Survey Characteristics of the High
Performers

• High performers contribute more to the business
• 8 times more projects and IT services
• 6 times more applications
• When high performers implement changes
• 14 times more changes
• One-half the change failure rate
• When high performers manage IT resources
• One-third the amount of unplanned work
• 5 times higher server/sysadmin ratios

43
The 21 Foundational Controls
The full list of 21 foundational controls can be
found in the ITPI IT Controls Performance Study.
Please see the last slide for ordering
information.
44
ITPI Survey Foundational Controls Performance

• Most Low Performers do not have Foundational
  Controls
• Chaotic Reactive
• Access Resolution receives early attention
• Change Service Level are essentially missing

• Medium Performers have some Foundational Controls
• Still in a Reactive posture
• Release gains attention
• Access Resolution continues as an emphasis
• Change Configuration receive little attention

• High Performers have almost all the Foundational
  Controls
• High IT process maturity

45
ITPI Survey What Differentiates High Performers?
46
ITPI Survey Which Foundational Controls
Differentiate Top Performers?
Note that virtually every top performer monitors
their systems for unauthorized changes and has
defined consequences for unauthorized
changes! Organizations that have these controls
are almost always great.
47
Our Advice
Implement things that really work, such as

• Reconcile all changes that have occurred
• against
• changes that were authorized via the change
  management process.
• Electrify the fence to enforce the process and
  hold people accountable.

48
Conclusions

• Change management is a high risk area, and
  growing in importance to senior management and
  the audit committee
• Errors in change management can have significant
  business impact, for financial reporting,
  compliance with laws and regulations, and IT
  effectiveness/efficiency
• Ineffective change management processes are easy
  to spot, even for non-technologists
• Auditors can help management go from good to
  great
• The only acceptable level of unauthorized change
  is ZERO!

49
Resources

• ITPI Visible Ops Handbook
• Kevin Behr, CTO, IP Services, Inc.
• Gene Kim, CTO, Tripwire, Inc.
• George Spafford, Spafford Global Consulting
• IIA Change Management GTAG
• Jay Taylor, General Director, IT Audit, General
  Motors
• Julia Allen, Software Engineering Institute
• Glenn Hyatt, IT Security Manager, GMAC
• Gene Kim, CTO, Tripwire, Inc.
• ITPI IT Controls Performance Study
• Gene Kim, CTO Tripwire, Inc.
• Kurt Milne, ITPI
• Dr. Dan Phelps, Florida State University
• Dr. Grant Castner, University of Oregon
• IIA/ISACA 4 Hour Change Control Workshop
• Get your copy of VisOpsEmail tripwire.com/visibl
  eops
• More InfoEmail sales_at_tripwire.com

50
Sneak Peek Inside IT Controls Performance Study
IT Controls Research Report Now available at
itpi.org for 1,695. (This is the price of
sending one person to ITIL Foundations
training.)
51
QUESTIONS?