Alleviating Constraints with Resource Pools

1
Alleviating Constraints with Resource Pools
Live Migration with Enhanced VMotion

• Breakout Session 2823
• Raghu Yeluri
• Sr. Architect
• Intel Corporation
• Date 09/16/2008

Alan Bumgarner Program Manager Intel
Corporation
Jim Blakley Director, DC Virtualization Intel
Corporation
2
Contents

• Problem
• VMotion Overview
• VMotion CPU Compatibility Constraints
• Why we care
• Solution Enhanced VMotion Intel VT
  FlexMigration
• Case Study Enabling Enhanced VMotion with
  Multi-Generation Intel Processors
• Wrap Up

3
VMotion Overview
Capability provided by VMware to migrate a live
Virtual Machine (VM) from one physical server to
another.

• Core Enabling Technology for Virtualization 2.0
  Use cases
• Dynamic Load balancing, planned
  maintenance/server upgrades, High Availability,
  Power-aware load balancing
• Key Requirements for Migration
• Part of the same Datacenter (should have the same
  subnet)
• Connected to the same storage (SAN, NAS, iSCSI)
• Recommended dedicated Gigabit network for
  VMotion
• Destination host must have enough resources
• Source and destination hosts must have compatible
  CPU models.

Key Requirement Source and destination hosts
must have compatible CPU models
4
VMotion Steps
VM running on server esx01 to be moved to esx02
Resume VM on esx02
1
4
Provision VM on esx02, pre-copy memory CPU
state from esx01 to esx02, log changes to a
bitmap
Delete virtual machine from esx01 VMotion
Successfully Completed
2
5
Suspend the VM on esx01 and copy changed memory
to esx02
3
Memory and CPU State copied to the target host!
5
Why Compatible CPUs?

• VMotion requires all processors in Live VM
  Migration Pool be compatible
• Compatible same ring 3 visible instructions
  (e.g. visible to all application software)
• E.g. 64-bit, SSE3, SSE4.1, SSE4.2, etc
• Caches, speeds, of cores, of sockets can vary
• CPU Vendor has to be same (not the OEM)
• Why?
• User mode features are not intercepted execute
  directly on the CPU from the application.
• If an application starts using an instruction
• it wants to keep using it even when an
  application is migrated to a new server
• If the instruction suddenly isnt presentthe
  application breaks
• Example 1 SSE4.1 (Penryn) instruction support
• Running application which uses SSE4.1
• Migrate to a CPU which does not support SSE4.1
  (Merom) gt crash

6
Why we care DC Environment for VMotion
Compatibility
Pool of Compatible Machines
VMotion
VMotion
4S 2S 1S
4S 2S 1S
4S 2S 1S
Dual-Core, Quad-Core Multi-Core
Quad-Core Dual-Core
Multi-Core 45nm
45nm (SSE4.1) Intel Core2 Micro architecture
(Penryn)
Future Intel (SSE4.2. Nehalem Beyond)
65nm (SSE3) Intel Core2 Micro architecture
(Merom)
Multiple Resource Pools. Data Center Fragmentation
Shows a DC environment with Intel-based
Processors.
7
Why we care Impact of Multiple Resource Pools

• Islands of Clusters/Pools, Limited Fungibility of
  resources
• In-effective utilization of resources Cannot
  leverage available capacity in a different Pool
• Reduced benefit of VMotion DRS
• Cannot add newer servers incrementally to a
  VMotion compatible pool
• Cannot take advantage of the faster performance
  and lower power processors..
• Customers need simpler ways to determine if new
  hardware are compatible with current environment

8
Solution Enhanced VMotion

• What
• Capability in VMWare ESX3.5U2 to increase
  VMotion Compatibility
• Automatically ensures hosts in a cluster present
  same CPU feature set, even with different
  generation CPUs on hosts. (CPUs must be from the
  same vendor)
• Will enable resource pools to include hosts with
  multiple generation of processors
• How
• Defines extension to a resource cluster called
  Enhanced VMotion Compatibility (EVC).
• Defines baseline feature/instruction set for
  cluster that host processors present to VMs.
• For Intel Processors
• Baseline feature set Intel Core Micro
  Architecture (Merom-based) - SSE3
• Leverages Intel VT FlexMigration Assist to
  present baseline feature set to VMs.
• Hosts with out baseline feature set or VT
  FlexMigration hardware assist cannot be added
  to the EVC cluster.
• EVC capability can be enabled/disabled from
  Virtual Center

9
Solution Intel VT FlexMigration

• Built-in hardware assist starting in 45nm Intel
  Core Micro architecture that increases live
  migration compatibility pool
• VT-based VMMs (64bit VMWare ) Intel VT-x
  execution control
• Non VT-based VMMs (BT 32Bit VMWare) CPU-ID
  Masking
• Allows VMWares ESX to report a consistent set of
  available instructions to guest software running
  within a VM.

Investment Protection and Future Proofing
Resource Pools
10
Case Study Enhanced VMotion with
multi-generation Intel Processors
vConsolidate Consolidate Stack Unit (CSU)

• Proof of Concept Goal
• Test ride and demonstrate Enhanced VMotion
  Intel VT Flex Migration
• Identify integration configuration issues to
  use Enhanced VMotion
• Approach
• Realistic VMs workloads vConsolidate
  benchmark
• VConsolidate 1.1
• 2 CSUs (10VMs)
• CSU 5 VMs (DB, Java, Web, Mail, Idle)
• EVC Cluster with multi-generational Intel Xeon
  processor hosts
• Intel Core (merom) and Intel Core2 (penryn)
  micro architecture servers
• Manual Automatic Migration with VMware DRS

11
Case Study EVC Cluster setup
Baseline Feature set SSE3 (Merom)
2
2
Intel Xeon 7400 (4S/6C, Penryn-based)
Intel Xeon 3200 (1S/4C, Merom-based) (Second
OEM)
1
1
1
Intel Xeon 5365 (2S/4C, Merom-based)
Intel Xeon 5460(2S/4C, Penryn-based)
Intel Xeon 7350(4S/4C, Merom-based)
EVC Cluster with Intel Core (Merom) and Intel
Core2 (Penryn) architecture Servers
12
EVC Cluster setup Adding physical hosts

• Issue 1 In compatible hardware.
• Certain CPU features not enabled on physical host
• Execute Disable (XD) Bit
• VT flag
• Fix Enable XD Bit and VT flag from the BIOS.
• Need an AC recycle for the VT flag to take into
  effect.

13
EVC Cluster setup Adding physical hosts -2

• Issue 3 Incompatible host being added into a
  EVC Cluster.
• All the hosts in a EVC cluster should have the
  ESX3.5U2 software.
• Fix Upgrade the host to ESX3.5 U2

14
Case Study Migration Results
After Enhanced VMotion
Before Enhanced VMotion
Intel Xeon 5460 (2S/4C, Penryn-based)
Intel Xeon 5365(2S/4C, Merom-based)
Intel Xeon 7350(4S/4C, Merom-based)
Intel Xeon 5460(2S/4C, Penryn-based)
Manual Migration
DRS Conservative
DRS Medium
DRS Aggressive
Successful migration between multi-generation
Intel-based servers within the same pool!!
15
EVC Requirements - Summary

• All hosts in the cluster should have ESX3.5 U2.
• Intel VT and Execute Disable (XD) flag needs to
  be enabled in BIOS for systems in EVC enabled
  clusters.
• Hosts should have Intel Xeon 3200, Intel Xeon
  5300, Intel Xeon 7300 or later processors.
• No VMs should be active on a host when being
  added to the EVC cluster.
• EVC can be enabled/disabled from Virtual Center.
• Need the latest version of Virtual Center (VC2.5
  U2) to enable EVC.

16
Wrap Up

• VMotion enabling new usages HA, Power-aware
  load-balancing, etc.
• CPU compatibility requirements of VMotion
  creating Islands of clusters/pools DC
  fragmentation
• Enhanced VMotion addresses Multiple Resource
  Pool Constraints in Data Center.
• For Intel Processors Enhanced VMotion leverages
  Intel VT FlexMigration hardware assist.
• Intel VT FlexMigration provides Investment
  Protection and future proofing of resource pools

17
Backup
18
Creating Enhanced VMotion Compatible clusters
Scenarios
1 Creating a brand new EVC cluster, and add physical hosts
2 Enabling an existing resource cluster for Enhanced VMotion with active VMs on the physical hosts
2 Enabling an existing resource cluster for Enhanced VMotion with hosts running prior version of ESX server software
19
Create a new cluster and enable for Enhanced
VMotion

• Create a regular resource cluster from VI client
  You will not see the Enhanced VMotion options
  during the creation of the resource cluster.
• Edit Settings to see VMWare EVC option and
  select Enable EVC for Intel Hosts
• Decide baseline feature set/instruction set for
  the cluster Default Merom-based processors
• Advanced Usage Change the baseline instruction
  set

20
Enabling EVC for an existing resource cluster