IS 2540

1
IS 2540

• IT Governance
• A Practical Transition Strategy
• Based on
• The Visible Ops Handbook, Behr, Kim Spafford,
  2005
• All figures from Visible Ops

2
Visible Ops

• Based on extensive observation / data collection
  from hundreds of IT organizations
• Goal was to discover what IT practices
  distinguish high-performing organizations
• Employs a benchmarking approach

3
Visible Ops

• Main IT success factors
• Pervasive change management practices
• Understanding of cause-effect relationships
• Use of effective auditable controls
• IT management based on facts not intuition or gut
  feel
• 80 of outages due to operator or application
  errors

4
Visible Ops

• IT organizational culture problems
• Bureaucratic CM
• End runs around CM
• Delusional agility of cowboy culture
• Control isnt possible, so page the IT
  firefighters
• Firefighting in lieu of fire prevention
• Auditors see chaos, so they push for more
  controls
• IT doesnt know which controls to implement
• Does COBIT have the right controls?
• In which order should they be implemented?

5
Visible Ops

• Symptoms of effective IT
• High service levels / availability
• High MTBF and low MTTR
• Lots of changes, successfully implemented
• 100s-1000s / week
• gt 99 successful
• Invest in early phases of IT processes
• Lowers cost of defect repair
• Sound familiar?

6
Visible Ops

• Process integration between organizational units
• Leads to collaborative working relationships
• Compliance-oriented
• Relevant controls in place working
• Controls documented easily verified
• Low of unplanned work
• lt 5 spent on unplanned / urgent work
• Frees up resources for fire prevention
• Sound familiar?

7
Visible Ops

• Huge leverage WRT IT assets human resources
• ServerSysAdmin gt 1001 (5X the average)
• Process effectiveness leads to higher
  productivity
• Sound familiar?

8
(No Transcript)
9
Visible Ops

• High performing common cultures
• Change management
• Viewed as absolutely critical
• Not viewed as bureaucratic
• All changes must be successful
• Causality
• 80 of outages due to changes
• 80 MTTR finding change that caused outage
• By analyzing changes, first fix 90 effective

10
Visible Ops

• Continual optimization
• Discover root causes of IT problems
• Prevent problems before they happen
• Highest level of compliance with least effort to
  maintain compliance
• Sound familiar? (Quality is free!)

11
Visible Ops

• Common IT processes
• None succeeded due to COBIT or ITIL!
• Each rediscovered good practices
• Each developed their own terminology
• Causes communication problems
• Visible Ops standardizes process terminology

12
Visible Ops

• Visible Ops standardizes processes wrt ITIL
  process framework
• Release
• Control
• Resolution
• Relationship
• Service delivery

13
(No Transcript)
14
Visible Ops

• Successful companies used 3 out of 5
• Release
• Invest your efforts in pre-production activities
• Plan, build, design, configure before release
• Control
• Control to prevent service disruptions
• Effective controls allow greater agility, not
  less
• Resolution
• Minimize rework efforts downtimes
• Requires cause-effect knowledge
• Frees resources for release control activities

15
(No Transcript)
16
Visible Ops

• Other success factors
• Controls are visible to management, security
  auditors
• Effective CM must address human factors
• Rebuilding is easier than repairing

17
Visible Ops

• Visible Ops Key to success
• Make transition short, easy practical
• Multi-year death marches dont work
• Could lose management sponsorship
• Staff will circumvent
• Use fewest of processes possible
• Implement 3 of 4 processes within 90 days

18
Visible Ops

• Process projects are
• Definitive w/ clearly defined objective
• Ordered to build on previous phase
• Catalytic to free up more resources than it uses
• Auditable to create ongoing documentation of
  controls
• Sustaining by creation of value to enterprise

19
Visible Ops

• 4 Visible ops phases
• Stabilize The Patient
• Catch Release and Find Fragile Artifacts
• Establish Repeatable Build Library
• Enable Continuous Improvement

20
(No Transcript)
21
Visible Ops

• Stabilize The Patient
• Medical triage for IT
• Goal
• Reduce unplanned work to lt 25
• Frees resources for more productive work

22
Visible Ops

• Symptoms
• Unplanned work 35-45 on average, can exceed 65
  ..Sound familiar?
• IT creates most of their own problems
• Most of downtime spent diagnosing cause of
  problem, only 20 spent in actual repair
• Dont know who made change or why
• Changes undo other changes
• Lack of confidence in IT

23
Visible Ops

• For each fragile IT asset
• Reduce / eliminate access
• No changes unless explicitly authorized
• Communicate change lockdown to stakeholders
• Allow change only during specified time window
• Enforce / reinforce CM process

24
Visible Ops

• Effective CM plays critical role in stabilizing
  IT
• Responsibility accountability for everyone
• Use automated detection tools like Tripwire to ID
  unauthorized changes
• For each unauthorized change
• Who did it?
• What was changed?
• Can it be rolled back?
• How to prevent reoccurrence?

25
Visible Ops

• CM key to success is
• Create culture of accountability
• Enforce maintenance windows
• Manage by facts, not beliefs
• of acceptable unauthorized changes 0

26
Visible Ops

• Eliminating changes decreases outages reducing
  amounts of unplanned work
• Frees up resources for productive work
• Create a Change Advisory Board (CAB) to manage
  changes
• Accept that business events cause IT change
  events
• All major IT groups on CAB Senior Mgmnt.
• Create emergency change procedure use it
  sparingly

27
Visible Ops

• Implement change request tracking system
• Document track all changes throughout their
  lifecycle
• Automated tools are available
• Collect change control metrics generate reports
• CAB weekly meetings to authorize changes
• Goal is maximum effectiveness with minimum
  bureaucracy
• Use meeting agenda template (p. 33 - 34)

28
Visible Ops

• For each change request do complete analysis of
  impacts
• Who, What, When, How, What IF questions
• Rank requests by priority
• Identify change dependencies
• Major risks involved
• Rollback strategy

29
Visible Ops

• Effective CM
• Post-implementation reviews
• Measure success rate learn from it
• Everyone attends meetings
• Document all change outcomes
• Ineffective CM
• Authorize changes without rollback plan
• Rubber stamping
• Outright waivers

30
Visible Ops

• Primary reason for any process failure is
• Lack of accountability
• Lack of strong management support
• General perception of nimbleness agility is a
  delusion

31
Visible Ops

• Stabilize The Patient Benefits
• Higher availability
• Less firefighting
• Higher change rate success
• CM process thats efficient effective
• Increased MTBF due to change windows
• Decreased MTTR due to CM
• Increased individual accountability
• Improved organizational communication

32
Visible Ops

• Phase 2 Catch Release / Find Fragile Artifacts
• Create maintain inventory of IT assets (esp.
  production assets)
• Symptoms
• How to start building a CMDB?
• Knowledge is individual, not organizational
• Uncontrolled changes cause unknown configuration
  states
• Explosion in of configurations

33
Visible Ops

• Tasks
• Senior staff to inventory all managed assets
• Thoroughly document all assets (P. 42 has
  checklist of questions)
• Tag the fragile assets
• ID those requiring most unplanned work
• Do Not Touch
• Focus efforts on unstable assets
• Prevent new builds until inventory completed
• Exceptions only via CAB

34
Visible Ops

• Benefits
• Service catalog documenting most critical
  services being supported
• CMDB containing all CI
• Supports queries / ad hoc reporting based on
  metrics
• Prioritized list of projects to replace fragile
  assets
• More organizational learning

35
Visible Ops

• Phase 3 Repeatable Build Library
• Create library of repeatable builds focusing
  first on fragile configurations
• A datacenter of Golden Images
• Enables replace instead of repair

36
Visible Ops

• Symptoms
• Configurations are unique, irreplaceable works of
  art
• Production configurations evolve rendering
  release configuration obsolete
• More configurations require more specialized
  knowledge about each configuration
• Patches cause crashes
• Patches not incorporated into builds

37
Visible Ops

• Create release management team
• Operate earlier in cycle to reduce defects in
  production
• Engineer repeatable builds
• Require constant time to rebuild
• Reduces configuration variance
• Junior staff does the builds
• Frees senior staff for more proactive tasks
• Goal is to reduce of configurations while
  increasing their shelf life

38
(No Transcript)
39
Visible Ops

• Create repeatable build process
• Generates Golden Builds (master images)
• Thoroughly planned, tested and approved
• Kept current with new patches upgrades
• Stored in definitive software library (DSL)
• Along with associated assets (documentation,
  licenses, keys, etc.)
• DSL is SW Fort Knox

40
Visible Ops

• Creating a DSL
• ID lowest common IT asset denominators
• Operating systems, applications, business rules
  data
• Create build catalog of components that must be
  standardized
• Create a repeatable build process for each item
  in catalog
• Isolate build network from other networks
• Place master builds in DSL
• Keep master builds current

41
(No Transcript)
42
Visible Ops

• Designate a DSL manager
• Create a DSL approval process for submitting
  master builds
• Keep all copies under revision control
• Initial 1 year amnesty for all running
  applications
• Replace with certified master builds as they
  become available
• Weed out unnecessary master builds

43
Visible Ops

• Establish acceptance process between production
  and release teams
• Release team designs and builds configurations
• Production teams accepts and deploys
• Production must get CAB approval prior to
  deployment
• Cant put any configuration into production
  unless accepted by production team
• Production only tests configurations in DSL
• For security reasons, developers not part of
  build process
• Could insert malicious code

44
Visible Ops

• Patching
• Belongs in release management
• Patch and Pray to be avoided
• Successful IT organizations patch less often
• Apply / test patches before releasing to
  production
• Patch to production system may be overwritten by
  subsequent build
• Use detective control tools to ensure build
  integrity

45
Visible Ops

• Benefits
• Build library cuts unplanned work to lt 15
• Release management team with well defined roles
• Process for repeatable builds
• Can repair by rebuilding
• Free up senior staff resources
• Tighter integration between release production
• Reduced of configurations
• Reduced patch risks

46
Visible Ops

• Phase 4 Continual Improvement
• Goal is to collect use metrics to improve
  performance
• Simply adopting best practices competitive
  paritynot good enough

47
Visible Ops

• Cant manage what you cant measure
• Sound familiar?
• Key IT metrics are availability
• MTBF MTTR
• Affected by factors in release controls

48
Visible Ops

• Release
• Are we efficient at provisioning infrastructure?
• Controls
• Are we making good change management decisions?
• Resolution
• Are we quickly diagnosing and fixing problems?
• IT needs metrics for all 3 process areas

49
Visible Ops

• Release metrics
• Time to provision good build
• of build revisions before accepted
• Build shelf life
• systems that are good builds
• builds with security sign-off
• builds rushed into production
• Release Engineers SysAdmin ratio
• Higher is better

50
Visible Ops

• Controls metrics
• authorized changes / week
• actual change / week
• Should equal authorized
• unauthorized changes
• Should be zero
• Change success rate
• Should be gt 99
• outages
• emergency changes per CAB

51
Visible Ops

• special changes outside CAB
• business as usual changes
• CM overhead in man-hours
• Changes submitted vs. Reviewed
• Resolution metrics
• MTTR
• MTBF