Chapter 10 Change Management and Revision Control

1
Chapter 10Change Management and Revision Control
2
Change ManagementDefinition

• The process of managing a server upgrade or any
  other type of update/upgrade/changes being made
  to a network. Change management ensures
• Smooth changes
• Successful changes
• Reduced oops-quotient
• Fast recover from an oops

3
Change Management

• Must match the process with
• Type System(s) being changed
• Type of Change being implemented
• The process for moving a user folder
• very different from
• Upgrading server OS
• What is a significant change?

4
Elements of Change Management

• Planning/preparation
• Communication
• Scheduling/timing
• Avoid conflicting with business schedules
• Implementation
• Documentation/audit trail
• Testing
• Post-event analysis
• Continuous Process Improvement

5
Communication Requirements

• Do not flood your customers with too many
  messages
• Use push communications for important messages
• Use pull communications for FYI

6
Communication Requirements

• Inform customers of new software
• How to use new software
• Is old software still available/how long
• Hard cutover
• Soft cutover
• Can changes be phased in?
• Inform customers when change is complete/how to
  report problems

7
Scheduling Issues

• Routine update
• No scheduling
• Fix problems that arise
• Local desktops/invisible to most users
• Major updates
• Affect many systems
• Require significant service outage

8
Scheduling Issues

• Sensitive updates
• altering router configurations
• global access policies
• firewall configurations
• making alterations to a critical server

9
Process and Documentation

• Need for Revision History
• Following the processes
• Producing documentation
• Force SAs to consider each and every step of the
  upgrade through the use of a change
  control/proposal form
• Check Syntax
• Verify Application usage

10
Change Proposal Form

• Changes to be made
• What Systems and services affected
• Reasons for the change
• Risks involved
• Test procedures
• Back-out plan and time to implement
• Time required to implement change
• Who will be making the chnage

11
Quiet Times

• This is quiet times for the SAs, not the
  business
• When the business is really busy, worse time to
  jeopardize the services they need.

12
ICING

• Automation
• Provides more stability
• Easier to track problems
• Keeps procedures consistent
• Change Control Meetings
• Keeps everyone informed

13
Server UpgradesChapter 11
14
Upgrade Process

• Develop a service checklist
• What services are provided by the server
• Who are the customers of each service
• What software package provides which service
• Verify that each software package will work with
  the new OS or plan an upgrade path
• For each service, develop a test to verify that
  it is working
• Write a back-out plan
• Select a maintenance window

15
Upgrade Process(continued)

• Announce the upgrade as appropriate
• Get customer buy in
• Execute the tests developed earlier to make sure
  they are correct
• Do the upgrade with someone watching/helping
  (mentoring)
• Repeat all the tests developed earlier. Follow
  the usual debugging process
• If all else fails, rely on the back-out plan
• Communicate completion/ back-out to the customers

16
Develop A Service Checklist

• Spreadsheets or other software
• Meetings
• Check current machine state
• Determine Dependencies
• Develop Shutdown/Boot Sequence checklist

17
Verify Software Compatibility

• List all software
• My require application upgrade
• Supports both old and new OS
• Supports only new OS
• May be no longer supported
• Check software documentation
• Test current software
• Regression testing
• Use the grapevine
• Dont always trust the vendor
• If not compatible
• Upgrade
• Back-Out
• Do With Out

18
Verification Tests

• Develop tests to verify working
• Manually or by Scripts
• Advantages
• Disadvantages

19
Write A Back-Out Plan

• Written steps to revert back
• How long?
• Be cautious of little problems
• Do the math to figure out when to start back-out
• Back up or Clone??

20
Select A Maintenance Window

• How Long?
• Upgrade time upgrade test Back out Time
  Back Out test
• When?
• What to do in 24/7 operation
• Document when back-out plan to be initiate
• End-Time Back out Time Time to test Back Out
  
• Follow Scottys advice (James Doohan) from Star
  Trek

21
Announce the Upgrade

• Decide on a format and use it consistently
• Email
• Voicemail
• Desk-to-desk paper memo
• Newsgroup posting
• Web page
• Note on door
• Smoke signals

22
Execute the Tests

• Do last minute tests
• Fail before or after upgrade?

23
Do The Upgrade

• Too critical to be done alone
• Unique mentoring possible
• Not everyday routine
• Ask for help

24
Test Your Work

• Run tests
• Involve customers

25
If All Else Fails

• Begin Back-Out at appointed time
• Taking too long
• Testing failing
• Critical dependency not available
• Business critical software fail?
• Retest again

26
Communicate

• Completion or Back-Out
• What got accomplished
• Service available?
• What changes?
• How to report problems

27
Other Considerations

• Adding new software
• Remove and add software
• Only make one change at a time
• Scripted tests
• Log, Log, Log, Log
• Two versions on same machine
• Redundancy
• Dress rehearsal
• Checklist

28
Important Elements

• Good Communications
• Attention to small details
• Well thought out test plan

29
Chapter 12Maintenance Windows
30
Maintenance WindowDefinition

• A maintenance window is basically a short period
  of time used to do work on the system that cannot
  be performed while the system is in normal use.
  The system must be partially or fully shut down
  to do the necessary work.

31
Work Requiring AMaintenance Window

• Hardware upgrades
• Software upgrades
• Replace broken parts
• Network changes

32
3 Stages of aMaintenance Window

• Preparation
• Execution
• Resolution

33
Other Considerations

• Scheduling
• Flight Director
• Change proposal

34
Change Proposals

• What changes are going to be made
• What machines will be worked on
• What are the pre-maintenance window dependencies
  and due dates
• What needs to be up for the change to happen

35
Change Proposals (cont)

• What will be affected by the change
• Who is performing the work
• How long will it take
• What are the test procedures and what equipment
  is needed for these tests
• What is the back-out procedure and how long will
  it take

36
Disable Access

• Post notices on doors etc.
• Disable all remote access
• Announce on Public Address system
• Change helpdesk voicemail message
• Emails in advance
• Advertise on Intranet/web page
• Memos

37
Shutdown/Boot Sequence

• Some machines cannot boot properly unless other
  machines are running
• If many interdependencies, have documented list
  of services and machines each machine depends on

38
The Morning After

• Similar to sensitive update
• All SAs available in case of unforeseen problems
• Postmortem
• Improve the process

39
ICING

• Training for the next Flight Director
• Keep good records
• Provides input for planning
• May need to provide limited availability
• Isolate the needed network and services
• Can redundant services be provided
• 24/7 availability
• Shorter more frequent Maintenance Windows
• Communication is key