Rapid Server Deployment

1
Rapid Server Deployment

• Presented by Matthew Bulat

2
Objectives

• New server considerations
• Redundancy options
• Site preparation
• Deployment documentation
• Deployment tools
• System Testing

3
Background

• Recent Server Deployment Project
• 42 old NT4 servers being replaced with Windows
  2003
• Genesys Call Centre Software upgraded
• 16 Sites around Australia
• Most deployed in February 2006
• Team group of 4 persons
• No Active Directory

4
My Role

• Hardware Choice
• Site Scope
• Standard Operating System
• Implementation Documentation
• Design deployment tools
• Deployments in Sydney, Brisbane, Penrith and
  Canberra

5
Server Considerations

• Software Needs
• Processing
• Storage
• Memory
• Network Connections
• Redundancy Options
• Future growth potential
• Physical Environment

6
Redundancy Options

• Dual Processor
• Dual Power Supply
• Multiple Redundant Cooling fans
• RAID hard drives
• Multiple memory slots
• Multiple network sockets
• Same model servers to have whole server backup

7
Site Equipment

• Keyboard Video Mouse Switch
• Network Switch connections
• Power Requirements
• Cooling
• Backup methods
• Racking

8
Site Checklist

• Space for server/s
• Rack ventilation for higher powered servers
• Power points available (more for redundant power)
• UPS capacity for new load for servers
• Keyboard Video Mouse cable lengths
• Patch Panel / Network Switch space
• Network Cable Lengths

9
Server Choice

• 2 RU Server
• Dual 3.4 GHz Xeon
• Mirrored 73 GB drives
• 4 x 512 Mb RAM
• 4 4 redundant cooling fans
• Redundant Power 600W
• Dual Gigabit network
• DVD burner on 1 server per site

10
Server Layout
11
Windows 2003 Server

• Windows 2003 Standard allows 4 CPU 4GB Ram
• Server roles added as required
• Roles have high level of security by default
• DNS name resolution standard
• Support IPv6 and IPv4 TCP/IP addressing
• Distributed File System Allow a logical file
  structure that can be replicated at many sites

12
Windows 2003 continued

• Remote Desktop has unlimited connections
• Windows 2003 Enterprise allows Cluster setups
• Internet Information Server 6 more resilient to
  crashing with multiple pool processes
• Pools can have set properties and are recycled
• IIS6 has read only static web pages by default
  initially

13
.NET framework

• Installed on Windows 2003 / XP or can be
  downloaded on Windows 98
• Provides standard objects for programming with
  the operating system. E.g. Win32, IO, Threading,
  System Management, Drawing, Forms, Data, Web
  Services, Messaging and Directory Services.
• Dynamic Link Library use is reduced using .NET
• Finished programs are much smaller as a lot of
  the work is achieved by referencing .NET
• .NET framework can be used by multiple languages

14
Standard Operating Environment

• List all software used
• Design of the Operating System Configuration
  e.g.. Unattend.sif
• List all media required
• Anti Virus selected
• Security Policy
• Server Management tools e.g. MOM, ESMPRO

15
Microsoft Baseline Security Analyser

• Free download (complete process in 1 day)
• Make sure patches are up to date
• Load on laptop
• Crossover to target computer
• State server name to test
• Wait for report
• Review deficiencies and alter security policy

16
Set Server Unattend.vbs

• INPUTS
• Input Server Name
• Unattend source
• Unattend variables, Server Name, IP Address,
  Subnet Mask, Gateway, DNS, Time Zone, License Key

• OUTPUTS
• Winnt.sif Unattend Output
• Set_Network.bat
• Set_DNS.bat
• Who_Am_I.txt

17
Configuration Floppy

• Set Server Unattend.vbs
• Local Security Policy
• Post Operating System Batch file for Drive
  letters, directories, shares, Accounts, Security
  policy, time sync source
• SCSI / RAID drivers
• Server Check tools
• Host file

18
Server Checks.vbs

• Uses Windows Management Instrumentation
• Configuration file used to display select HTML
  output
• Confirms Server name, hardware specifications,
  hardware operational, display settings,
  individual service status, user accounts etc.
• Result in form Parameter, expected value,
  returned value (green match, red not match)
• Win32_DiskDrive,Status,OK,, OK

19
Media List

• Windows 2003 CD
• NEC Express Builder CD
• Software CD
• SQL CD
• Configuration Floppy
• Management Virtual Machine
• SQL Virtual Machine

20
NEC Express Builder CD

• It has all drivers for server
• You can create a Management Utilities partition
• You can run diagnostics from the CD
• Has ESMPRO server monitoring and control software
  including Wake on LAN, reboot ability,
  utilization monitoring, temperatures, voltages,
  fan speeds, CPU load, drive usage
• Has Diana Scope which can read a standard
  hardware monitoring card within the server
  including log files held in flash memory

21
Management Virtual Machine

• Imitate central management server
• Deploy Windows Server Update Service connection
  with recent Microsoft Service Packs, Patches /
  Security updates
• Deploy Anti Virus with recent definitions
• Use ESMPRO to test server hardware early

22
SQL Server Virtual Machine

• Imitate central SQL Server structure
• This allows software installation with a known
  SQL server structure present
• Allow running of DTS packages to set up databases
• Allows the setting up and testing of
  authentication structures

23
Windows Server Update Service

• Provides a single point for updates on your
  network
• Free software, needs IIS setup, Web Interface
• Update information to WSUS, you choose updates to
  download and computer groups
• 8 GB of updates available
• Script to configure PC to use of WSUS
• Uses Background Intelligence Transfer Service
• WSUS knows all updates deployed on each client
• Can export WSUS data and import it into a remote
  network WSUS

24
Microsoft Operations Manager 2005

• Group Management of computers
• Real time analysis of issues, performance and
  security
• Health of network can be seen in single view
• Alerts can be drilled down to computer group,
  computer, process
• Automatic / manual responses can be done locally
  or centrally
• Computer types can have their own rule set
• Standard rule sets available free or via 3rd
  party e.g. Cisco

25
MOM continued

• Alerts can be logged in/out to person
• Microsoft knowledge base available with context
  of alert
• Company knowledge base built as alerts handled
• History of actions in similar alert available
• Network connections constantly monitored
• Test transactions can be used to monitor
  performance
• WMI style computer discovery ability
• 100 standard reports for resource availability,
  capacity planning, alert response time, alert
  type and frequency

26
MOM representation
27
MOM Requirements

• MOM on SQL Server
• Clients can report to up to 4 MOM servers
• Suggest lt 10 Admin consoles per MOM (30 max)
• Clients can be XP / 2000 / 2003
• 100 Clients generate approx. 250 MB / day
• Maximum 2000 clients per MOM
• 2 minutes response time to Admin Console
• Approximately 2 Proc. client load, 100Mb space,
  10 Mb RAM

28
MOM Admin Console
29
Computer Telephony Integration
30
Genesys Software

• Translator matches PABX phone activity and PC
  activity to stay in sync. Common API independent
  of PABX model. Genesys allows a whole extra layer
  of customization. E.g. Route by skill, client
  representative, client team, client region.
• SQL Server Defines rules for routing calls,
  caches Solution Configuration, keeps call logs
  and statistics
• Management Configure PABX, configure Genesys,
  monitor and control Genesys
• IVR Receives call, collects key data to pass on
  to Genesys for call handling purposes

31
Genesys Monitoring

• Each server runs a Local Control Agent
• LCA can start, stop and monitor applications
  under its control.
• LCA reports to Solution Control Server
• You use Solution Control Interface to look at SCS
  for current status and to control the solution.
• Network connections are monitored constantly
• Alarms can be sent in various forms

32
Server Redundancy Styles

• Have applications loaded on more than 1 server
  with 1 active and the others disabled
• Move mission critical hard drives from one server
  to another to continue operations
• Point applications to offsite server to take up
  role
• Hot swap RAID drives in case of single drive
  failure
• Hot standby RAID drive to take up activity faster
• Fault Tolerant Server for LAN, HDD, PCI, CPU,
  memory, SCSI redundancy

33
Deployment Risks

• Team skills
• Time available and possible distractions
• Money versus performance
• Delivery of equipment
• Applications fail in new environment
• Network traffic changes which cause solution
  degradation

34
Staff Options

• NEC Professional Services
• Efficient
• Fast
• Easier on team resources

• In house Team
• Know solution fine details
• Learning Experience
• Ongoing Maintenance benefits
• Quicker to refine requirements

35
Team Skills

• Electrical Engineering skills
• Microsoft Operating System skills
• Software / Programming skills
• Genesys skills
• Cabling abilities
• Testing ability
• Fault finding
• Project Management

36
Time Management

• Allocate staff to handle deployment with some
  spare time for contingencies
• Allocate resources to take care of teams normal
  duties
• Raise any project plan issues early
• Weekly meeting to confirm project progress
• Central Document Register with Versions and owners

37
Project Implementation Testing

• 1 server was used for Standard Operating
  Environment testing
• 3 servers were used in Development testing
• 3 servers were used as the first site Pilot
• Pilot ran for 1 month and full rollout signed off
• Network monitoring was performed before and
  during the pilot phase
• Implementation documentation was refined

38
Logistics

• NEC server division having knowledge of server
  deployment schedule
• Exact server configuration known including drive
  placement, Memory sticks and slots, RAID setup
• NEC Server division can assemble all options,
  create RAID and test servers before delivery
• This saves over 1 hour per server

39
Implementation Documentation

• Large Word document with linked Spreadsheet for
  all site parameters
• Includes initial preparation, equipment
  inspection, racking, Standard Operating
  Environment, Standard Application Environment,
  testing, milestones and expected completion time
  for each stage
• Each item checked off when completed
• Milestones broadcast to project team by SMS

40
Site Documentation

• Technology Map
• Hardware Specification
• Software Specification
• System Monitoring
• System Processes
• Regular Operations

• New Server Configuration
• Diagnostic Operations
• Disaster Recovery Plans
• Document References
• Contacts

41
Initial Site Preparation

• Confirm all hardware is present on site
• Setup and test KVM switch, monitor, keyboard and
  mouse
• Label all cables, patch panel connections and
  hardware
• Setup server Configurations disks
• Confirm operation of setup Virtual Machines
• Confirm security arrangements for extended hours
  of operation

42
Speed Improvements

• Try to acquire no activation Operating System
• Run deployments in parallel
• Allocate other task suggestions while waiting for
  software to install such as cabling
• Have breaks during longer install sections
• Use a network switch between the Virtual Machine
  and the servers to set up concurrently

43
Software Deployment Style

• There were 5 server types to setup
• All server had the same SOE
• Software installed depending on role and site
• Script used to prompt for site name and role
• Script the used silent command options to install
  applications
• Script finished with text file noting each
  application as being installed successfully or
  not
• Manual application installation method also noted

44
Testing Points

• Confirm RAID present
• Confirm Operating System Installed basic
• Confirm WSUS deploys updates
• Confirm Anti Virus present
• Confirm ESMPRO says hardware is OK
• Confirm Server Check.vbs says Operating System is
  OK
• Confirm applications all successfully installed

45
Connection to LAN WAN

• Test network connections to and from server
• Test main applications end to end
• Test support utilities with sample files
• Initial connection to system tested achieved in
  less that 1 hour
• Allow for client testing when you as satisfied

46
Useful designed tools

• Find Latest File Checks a list of directories
  for the newest file to confirm logs and backups
  are being created. Results are in a colour coded
  relative to age HTML page.
• Search Logs Can scan log files for errors or
  confirm connection activity
• Nightly zipped logs Conserves drive space
• Delete Old Logs Nightly remove zips older than
  x days

47
Utilities

• AutoIt Scripting to imitate keyboard and mouse
  activity to perform regular tasks. It can work by
  itself or as an ActiveX with another language.
• Word Viewer
• Excel Viewer
• PSTools including Pspasswd
• VNC
• Winzip
• Adobe Acrobat Reader

48
Post Implementation Review

• Little things can slow progress down at sites
• Implementation plan is a good stating point but
  advanced skills are needed for contingencies
• Personal preferences in desktop presentation
  should not interfere with the implementation
• Some testing may identify new issues outside of
  server scope such as WAN latency
• New servers take a short period of time to be
  acknowledged on the WAN

49
The Future

• Call centre voice recording on a very large scale
• At least 16 servers, interfacing equipment and a
  large Storage Area Network
• Central collection overnight
• Attached data with audio files
• Search for call audio by extension, caller, date
• Very useful for compliance / quality purposes

50
Storage Array Network NEC S2400

• 4 to 120 Hard drives
• 27 Terabytes maximum
• 8 GB RAM maximum
• 4 x 2Gbps fibre channels connections
• Supports RAID 0, 1, 5, 6, 10
• RAID 6 has 2 parities which allows 2 drives to
  fail