Virtual Systems Monitoring and Capacity Planning

1
Virtual Systems Monitoring and Capacity Planning
An UpdateDemand Technology Windows Symposium
CMG 12/05/2005

• Phil Henninge
• Demand Technology Software
• 1020 Eighth Avenue South, Suite 6, Naples, FL
  34102
• phone [phone] fax [phone]
• e-mail philh_at_demandtech.com
• http//www.demandtech.com

2

• Charley August 13, 2004

• Cape Coral

• Naples

3

• Wilma - October 24, 2005

• Cape Coral

• Naples

4
Agenda

• Review
• Whats New
• Virtual Machine Timing
• What Lies Ahead
• Sessions Available at CMG
• Questions

5
Introduction

• What is a virtual machine?
• An abstract machine for which an interpreter
  exists. Virtual machines are often used in the
  implementation of portable executors for
  high-level languages.
• - Melinda Varian, Princeton University
• Java VM
• SAS
• Visual Basic for Applications VBA
• A software emulation of a physical computing
  environment
• See http//computing-dictionary.thefreedictionary.
  com
• e.g., VM/CMS

6
Virtual System Overview
7
Introduction

• Who Are the Major Players?
• VMWare (wholly owned subsidiary of EMC)
• Workstation - powerful virtual machine software
  for developers and system administrators
• GSX Server -enterprise-class virtual
  infrastructure for departmental server
  consolidation and streamlining development and
  testing operations
• ESX Server -virtual infrastructure software for
  partitioning, consolidating and managing systems
  in mission-critical environments
• Microsoft (formerly Connectix).
• Virtual PC - a software virtualization solution
  that allows you to run multiple PC-based
  operating systems simultaneously on one
  workstation.
• Virtual Server Standard Edition run on one
  server with up to 4 processors.
• Virtual Server Enterprise Edition run on one
  server with up to 32 processors.

8
Introduction VMWare Virtualization
9
Introduction - Microsoft Virtualization

• From the bottom of the stack
• The host operating system Windows Server 2003
  manages the host system.
• Virtual Server 2005 provides a VM virtualization
  layer that manages virtual machines, providing
  the software infrastructure for hardware
  emulation.
• Each virtual machine consists of a set of
  virtualized devices, the virtual hardware for
  each virtual machine.

10
Roles of Virtual Systems in an Enterprise

• Consolidate multiple server workloads.
• Underutilized Servers
• Disaster Recovery
• Environmental and TCO (total cost of ownership)
• Re-host legacy applications on newer hardware.
• NT 4.0
• W2K
• Linux under Windows/Windows under Linux
• Enterprise software test and development.
• Technology arose for ISV test and development.
• Technology demos.

11
Performance Monitoring of VSs

• At the system level we look at the system
  resources
• CPU Utilization
• Memory Utilization (memory consumption and
  paging)
• Disk Utilization
• Network Utilization (NIC traffic and topology)
• At the software level we look at specific
  objects.
• Process (what are the VMWare and Microsoft
  specific processes)
• Network Interface (what virtual network adapters
  are defined)
• Other Performance Objects

12
Monitoring VMware

• VMWare object
• One instance for each Virtual Machine
• Virtual Disk (8 Counters)
• Disk operations (R-W-Total) performed by the
  guest OS
• Guest Locked Memory Bytes
• The number of bytes of simulated physical memory
  that is locked by the guest OS
• Guest Virtual Physical Memory Bytes
• The number of bytes of simulated physical memory
  in the virtual machine
• Percent Guest Physical Memory Touched
• The percentage of simulated physical memory
  recently used by the guest OS

13
Monitoring VMware

• VMWare object (Continuted)
• Network Counters (9 Counters)
• Network Transfers/sec
• Network Bytes Transferred/sec
• Network Transfer Errors/sec
• Network Packets Sent/sec
• Network Bytes Sent/sec
• Network Send Errors/sec
• Network Packets Received/sec
• Network Bytes Received/sec
• Network Receive Errors/sec

14
Monitoring VMware

• Host OS
• Processes
• Vmnat, vmnetdhcp,vmware,vmware-authd,vmware-vmx
• Network Interface
• Vmware virtual Ethernet adapter VMNet1, ..
  Adapter VMNet8
• VMWare object
• One instance for each Virtual Machine
• Guest OS
• Processes
• VMWareService, VMWareTray, VMWareUser
• Network Interface
• AMD PCNET Family Ethernet Adapter

15
Monitoring Virtual Server

• Virtual Processors Object (Virtual PC)
• One instance for each Virtual Machine
• Guest External Interrupts
• Number of virtual interrupts delivered
  to guest OS.
• Host-to-VMM Context Switches
• Number of context switches between
  Windows and the guest
• (VMM) context.
• Cumulative Guest Run Time
• The guest run time represents the
  number of microseconds
• the guest processor has run on a
  host processor. With
• the default scaling, the graph
  represents guest run time
• percentage.
• VMM Exceptions
• Number of processor exceptions handled
  by the VMM.

16
Monitoring Virtual Server

• The Virtual Server WMI Class contains two objects
• VirtualMachine - CPU, disk, and network usage
  counters an instance for each virtual machine
• VirtualNetwork - monitor the usage of each
  virtual network (must be attached to a physical
  NIC an instance for each virtual network
• For detailed information on these objects
• http//www.microsoft.com/technet/prodtechnol/virtu
  alserver/2005/proddocs/vs_tr_tools_WMI.mspx

17
Monitoring Virtual Server

• Host OS
• Processes
• Vssrvc (one for each guest machine)
• Virtual Processors Object (Virtual PC)
• One instance for each Virtual Machine
• WMI Objects (Virtual Server)
• VirtualMachine
• One instance for each virtual machine
• VirtualNetwork
• One instance for each virtual network

18
Monitoring Virtual Server

• Guest OS
• Processes (after Virtual Machine Additions)
• VMSrvc, VMUSrvc, VMPCMap, Interface
• Network Interface
• Virtual Server - Intel 21140-Based PCI Fast
  Ethernet Adapter (Generic) Packet Scheduler
  Miniport
• Virtual PC Intel DC21140 PCI Fast Ethernet
  Adapter

19
Performance Monitoring of VSs

• HALT/Idle Loop Measurement Anomaly
• When a machine is idle, its operating system will
  either issue a HALT instruction or repeatedly
  execute an idle loop of NOP instructions
• Idle loop is the default for most server machines
  
• Idle loop is a function contained in hal.dll
• When a virtual machine executes an idle loop, it
  is actively executing instructions which run on
  the host machines physical processor. Thus
  performance tools in the guest machine will show
  inactivity, while the host machine will appear
  fully utilized.
• Virtual machines running Windows operating
  systems having the wrong HAL (Hardware
  Abstraction Layer) installed will make the guest
  operating system spin in its idle loop, instead
  of HALTing when there is nothing else to do.

20
Performance Monitoring of VSs

• Halt/Idle Anomaly
• Even when the correct HAL is installed, some
  guest operating systems HALT more aggressively
  than others.
• The multiprocessing HAL favors using the Idle
  loop, instead of HALTing a processor.
• VMWare reports that W2K frequently spins, whereas
  Windows 2003 HALTs whenever it is idle. See
  AnswerID 1077 in WMWares KB
• http//www.vmware.com/support/kb/enduser/std_adp.p
  hp?p_faqid1077

21
Virtual Systems Sizing

• VMWare Planning Tools
• VMware P2V Assistant
• http//www.vmware.com/products/vtools/p2v_features
  .html
• VMWare Virtual Infrastructure Methodology
• http//www.vmware.com/pdf/vim_datasheet.pdf

22
Virtual Systems Sizing

• Microsoft Planning Tools
• Microsoft Virtual Server Migration Toolkit
• http//www.microsoft.com/windowsserversystem/virtu
  alserver/overview/vsmtdatasheet.mspx
• Solution Accelerator for Consolidating and
  Migrating LOB Applications
• http//www.microsoft.com/technet/itsolutions/techg
  uide/msa/solacc/lobsa/default.mspx

23
Virtual Systems Sizing

• Sizing destination servers requires first
  understanding the performance of the applications
  running on the source servers.
• The VM Host machine must contain sufficient
  capacity (Processor, Memory, Disk and Network)
  to handle the peak loads of guest machines
• accumulate measurement data over long term
  periods that include seasonal peaks
• compute PeakAverage ratios and understand when
  peak periods occur to ensure they do not overlap
  on the same host
• compute 90-95th percentiles

24
Virtual Systems Sizing
25
CPU Capacity

• The processor requirements of a source server
  should not exceed the processor capacity
  available to a virtual machine on the destination
  server.
• Normalize based on MHz
• CPU requirements number of CPUs x CPU speed x
  CPU utilization
• The ? Processor Time for all virtual machines
  running on a destination server should be lt 90
  of the available CPU capacity
• 10 reserved for the host OS and I/O for virtual
  machine threads.
• CPU capacity number of processors x CPU speed

26
Memory Capacity

• The total amount configured for all virtual
  machines cannot exceed the size of physical RAM
• Guest Memory sizeof(RAM) Available Bytes
  (95th percentile)
• Every virtual machine requires an additional 32
  MB of physical memory
• The host operating system requires exclusive use
  of at least 384 MB of memory.
• Host Memory Capacity gt
• 384 (SizeofVM132MB)(SizeofVM232MB)(Sizeo
  fVMn32MB)

27
Disk Capacity

• The disk must be sized to support ? Physical
  Disk\Transfers/sec for all guests I/Os.
• A single drive can sustain 100-200 random I/Os
  per second.
• Faster disks with 15,000 RPMs and 6 ms seek time
  may be able to do better.
• See Friedmans A simplified approach to Windows
  disk tuning on Tuesday.

28
Disk Capacity

• The following are best practices for performance
  optimization on virtual hard disks
• Use a hard disk solution that allows fast access,
  such as a locally-attached SCSI hard disk, RAID,
  or SAN.
• Put each virtual hard disk on a dedicated volume,
  SCSI hard disk, RAID, or SAN disk. It is easiest
  to put virtual hard disks together with their
  associated virtual machine configuration files on
  a RAID or SAN because this keeps everything in
  one place.
• Reduce disk fragmentation. As a dynamically
  expanding virtual hard disk increases in size, it
  becomes increasingly fragmented. You can
  defragment the host operating system to make the
  virtual hard disk more contiguous. If disk
  performance is important, consider doing this.
  Fixed size virtual hard disks are allocated a
  contiguous block of reserved space on the
  physical hard disk. Therefore, there is no
  overhead created by the growing disk.
• Compact the virtual hard disks to create more
  physical disk space.

29
Network Capacity

• Provide a dedicated network adapter in the
  destination server for each network adapter that
  existed in the source server.
• Configure at least one additional network adapter
  for managing Virtual Server itself and remote
  access to virtual machine consoles.

30
Network Capacity

• Load balance.
• You can load-balance virtual machines for
  networking. To do this, run a mix of
  network-intensive and non-network-intensive
  applications on a single physical computer.
• Add network adapters.
• For best performance, you should allocate a
  physical network adapter to each virtual machine.
  
• Note Virtual machines cannot take advantage of
  software-based network load balancing (NLB)
• The Virtual Server network driver runs below the
  network load balancing driver in the host
  operating system network stack.
• This isolates each host guest operating
  systems.

31
Questions?

• Resources
• VM and the VM Community Past, Present, and
  Future Melinda Varian, Princeton University
• http//pucc.princeton.edu/melinda/25paper.pdf
• Microsoft Virtual Server
• http//www.microsoft.com/windowsserversystem/virtu
  alserver/default.mspx
• http//www.microsoft.com/virtualserver
• VMWare
• http//www.vmware.com
• Planning Guide for the Virtual Server 2005
  Solution
• http//www.microsoft.com/technet/itsolutions/tec
  hguide/msa/solacc/lobsa/lobsaplg.mspx
• VMWare Capacity Planning
• http//www.askewview.net/lxy/VMware/Capacity_Plan
  ning.html

32
Agenda

• Review
• Whats New
• Virtual Machine Timing
• What Lies Ahead
• Sessions Available at CMG
• Questions

33
Whats New with VMWare

• ESX Server 3 and VirtualCenter 2
• Distributed Availability Services
• Distributed Resource Scheduling
• VMware Workstation 5.5
• 64-bit Guest Support for AMD64 and EM64T Systems
  with VT Support
• Two-way Virtual SMP (experimental)
• VMware Player
• Enables Anyone to Easily Run, Share or Evaluate
  Software in a Virtual Machine on a Windows or
  Linux PC

34
Whats New with Microsoft

• Virtual Server 2005 R2
• High Availability
• Clustering virtual machines across hosts
• Host clustering support
• Greater Scalability
• 64-bit (x64) host support
• Improved Performance
• Up to 50 drop in CPU utilization
• Improved Manageability
• PXE network boot support
• Better Interoperability
• Linux guest support

35
Agenda

• Review
• Whats New
• Virtual Machine Timing
• What Lies Ahead
• Sessions Available at CMG
• Questions

36
Virtual Machine Timing

• Whitepaper that discusses timing mechanisms
  between operating systems and hardware.
• Processes within a virtual machine interact with
  the virtual machine monitor instead of the
  hardware
• http//www.vmware.com/pdf/vmware_timekeeping.pdf

37
Virtual Machine Timing

• Windows keeps track of time by counting timer
  interrupts or timer ticks. When the operating
  system starts up, it reads the current time to
  the nearest second from the computer's
  battery-backed (CMOS) real time clock or queries
  a network time server to obtain a more precise
  time.
• To update the time from that point on, the
  operating system sets up one of the computer's
  hardware timekeeping devices to interrupt
  periodically at a known rate (say, 100-200 times
  per second).
• This is timekeeping mechanism is known either as
  the periodic interrupt or the quantum in Windows.

38
Virtual Machine Timing

• Types of Hardware Timers
• PIT - Programmable Interval Timer
• CMOS RTC - Real Time Clock
• Local APIC - Advanced Programmable Interrupt
  Controller
• ACPI - Advanced Configuration and Power Interface
  or Chipset
• TSC - Time Stamp Counter
• HPET - High Precision Event Timer

39
Virtual Machine Timing

• Using a hardware interrupt to track time leads to
  problems in the guest virtual machine
• At the moment a virtual machine should generate a
  timer interrupt, it may not actually be running.
  In fact, the virtual machine may not get a chance
  to run again until it has accumulated a backlog
  of many timer interrupts.
• Timer interrupts queued up for a single timer
  device cause a scalability issue as more and more
  virtual machines are run on the same physical
  machine.

40
Virtual Machine Timing

• What does this mean for Windows analysts?
• Microsoft Windows has an additional time
  measurement feature accessed through the
  QueryPerformanceCounter system call. This name is
  a misnomer, since the call never accesses the
  CPU's performance counter registers. Instead, it
  reads one of the timer devices that have a
  counter, allowing time measurement with a finer
  granularity than the interrupt-counting system
  time of day clock.

41
Virtual Machine Timing

• Lets look at the counter PERF_100NSEC_TIMER
• Description - This counter type shows the active
  time of a component as a percentage of the total
  elapsed time of the sample interval. It measures
  time in units of 100 nanoseconds. Counters of
  this type are designed to measure the activity of
  one component at a time.
• Formula - (N1 - N0) / (D1 - D0) x 100, where the
  denominator (D) represents the total elapsed time
  of the sample interval, and the numerator (N)
  represents the portions of the sample interval
  during which the monitored components were
  active.
• Example - Processor\ User Time

42
Virtual Machine Timing

• Lets Look at another counter PERF_PRECISION_100NS
  _TIMER
• Description - This counter type shows a value
  that consists of two counter values the count of
  the elapsed time of the event being monitored,
  and the "clock" time from a private timer in the
  same units. It measures time in 100 nanosecond
  units. This counter type differs from other
  counter timers in that the clock tick value
  accompanies the counter value eliminating any
  possible difference due to latency from the
  function call. Precision counter types are used
  when standard system timers are not precise
  enough for accurate readings.
• Formula - Nx - N0 / D1 - D0, where the numerator
  (N) represents the counter value, and the
  denominator (D) is the value of the private
  timer. The private timer has the same frequency
  as the 100 nanosecond timer.
• Example - PhysicalDisk\ Disk Time

43
Virtual Machine Timing

• Counter types and example counters in Windows
  Server 2003 using timers
• PERF_100NSEC_TIMER Processor\ User Time
• PERF_100NSEC_TIMER_INV Processor\ Processor
  Time
• PERF_AVERAGE_TIMER - PhysicalDisk\ Avg. Disk
  sec/Transfer
• PERF_ELAPSED_TIME - System\System Up Time
• PERF_PRECISION_100NS_TIMER - PhysicalDisk\ Disk
  Time

44
What Lies Ahead?

• Hardware enabled virtualization
• Intel - VT Virtualization Technology formerly
  Vanderpool
• Now shipping in desktops server support in 2006
• AMD - Pacifica first half of 2006
• Software
• VMWare ESX will most likely be modified to
  support hardware virtualization
• Microsoft Longhorn Server will have
  extensions after it is released
• XenSource Open source consortium targeting VT
  and Pacifica
• Virtuozzo from SWSoft uses a customized HAL
  to virtualize OS

45
Agenda

• Review
• Whats New
• Virtual Machine Timing
• What Lies Ahead
• Virtual Machine Sessions at CMG 2005
• Questions

46
Virtual Machine Sessions at CMG

• 314 Virtualization Concepts, Applications, and
  Performance Modeling
• TUESDAY 915 AM - 1015 AM OSCEOLA 34
• 324 To V or not to V A Practical Guide to
  Virtualization
• TUESDAY 1030 AM - 1200 PM OSCEOLA 34
• 334 Measuring Up for Server Virtualization
• TUESDAY 200 PM - 300 PM OSCEOLA 34
• 344 Virtual Performance Wont Do Capacity
  Planning for Virtual Systems
• TUESDAY 345 PM - 445 PM OSCEOLA 34
• 546 Modeling VMware ESX Performance
• THURSDAY 345 PM - 445 PM OSCEOLA 12

47
Questions?

• Resources
• Timekeeping in VMware Virtual Machines
• http//www.vmware.com/pdf/vmware_timekeeping.pdf
• Microsoft Virtual Server
• http//www.microsoft.com/virtualserver
• VMWare
• http//www.vmware.com
• Others
• http//www.xensource.com
• http//www.swsoft.com/en/products/virtuozzo
• http//www.run-virtual.com
• http//about-virtualization.com