Virtualization: Towards More Flexible and Efficient Grids

1
Virtualization Towards More Flexible and
Efficient Grids

• Kate Keahey
• keahey_at_mcs.anl.gov
• Argonne National Laboratory

2
The Grid Metaphor
What happens if a power station fails?
How do we store energy?
How do we charge for energy?
What elements make for a safe and efficient
power Grid?
How do we ensure quality of service?
How do we reliably deliver energy?
How do we make sure that supply meets demand?
3
Computational Grids
How can we manage different computing
environments?
What is the unit of resource usage?
We need a computon that will combine
environment and enforcement aspects as well as a
way of managing the multi-dimensional nature of
the Grid
How can we negotiate for computation?
How can we ensure that disk, CPUs, network are
all available?
4
Grids Today

• Grid Middleware Tools
• Security, Data Management, Resource Management
  Scheduling, Monitoring
• Standards GGF, OASIS
• Implementations Globus Toolkit, Condor and
  others
• Many new services are being developed
• Significant deployments and use of Grid
  infrastructure
• TeraGrid, Open Science Grid (OSG), Grid 3, many
  European deployments
• Multiple projects making production use of Grid
  infrastructure.
• Still issues heterogeneity, lack of satisfactory
  control and accounting, no on-demand computing

5
The Virtualization Layer

• Virtual Grids virtualize computers, networks,
  disks, memory
• Overlay networks, virtual storage
• Use middleware to map the virtualized constructs
  onto physical hardware
• Trust middleware to map and remap the virtual
  environment as needed
• Trust market forces to ensure that physical
  resources are plentiful when you need them

6
Virtual Workspace

• For now focus on virtual workspace
• Unit of enforcement, a computon for the Grid
• Representation of a desired environment
• Later put all elements of the system together
  into a virtual Grid
• We need progress in the following areas
• Protocols to dynamically negotiate and describe a
  workspace
• Ongoing work at GGF WS-Agreement, JSDL spec
• A unit of enforcement
• A critical mass implementation
• Recent revival in virtual machine technologies
  provides potential for such an implementation

7
Virtual Machine Basics
App
App
App
App
App
Guest OS (Linux)
Guest OS (NetBSD)
Guest OS (Windows)
Virtual Machine Monitor (VMM) / Hypervisor
Hardware

• A VM can serialize all of its state (including
  RAM)
• A VM image is simply a collection of files
• Disk partitions, RAM, configuration file
• Such image can be easily moved (migrated) between
  hypervisors of the same type
• Such image can also be saved and used for
  rollbacks

8
Different Hypervisor Implementations

• Depending on the layer you virtualize you will
  end up with a different VM
• API language VMs (JVM)
• ISA system VMs (VMware)
• Different types of system virtual machines
• Full virtualization (VMware)
• Run multiple unmodified guest OSs
• Para-virtualization (Xen, UML, Denali)
• Run multiple guest OSs ported to a special
  architecture
• Single OS image (Vserver)
• What is the cost of using VMs?
• Paper From Sandbox to Playground Dynamic
  Virtual Environments in the Grid, Grid 2004

9
The Need for Speed
Paper Xen and the Art of Virtualization, SOSP
2003
10
Licensing and Distribution

• License
• Open source (Xen, UML)
• Visible effects of open source community at work
• Commercial (VMware)
• Also, XenSource
• Distribution/Installation
• Para-virtualization requires kernel modifications
• Yes, but everything else stays the same
• Xen is (or soon to be) part of multiple
  distributions Fedora Core 4, Debian, inofficial
  Gentoo, Mandrake and SUSE distributions
• Work on making Xen part of the Linux kernel
• Privilege
• Xen (root, patch kernel, domain 0 privileges
  setup)
• VMware Workstation (root, installation only)
• UML user-level

11
What Makes VMs Great

• Summary of VM properties
• Good isolation properties
• Generally enhanced security, audit forensics
• Excellent enforcement potential
• Details depend on implementation
• Customizable software configuration
• Library signature, OS, maybe even 64/32-bit
  architectures
• Serialization property
• VM images (include RAM), can be copied
• The ability to pause and resume computations
• Allow migration
• How do we make VMs available over the network and
  manage them so as to leverage this potential?
• Challenges security, enforcement, protocols

12
Grid Services

• Web Service Resource Framework
• An extension of Web Services
• Provides standard mechanisms for
• Creation
• Lifetime Management
• State management, inspection (notification)
• Globus Toolkit 4
• Implementation of the WSRF framework
• Available since April 2005
• Provides secure authentication, authorization as
  well as tools for fast transfer, replica
  management, monitoring, and others.

13
What are Virtual Workspaces?

• Virtual Workspaces environments that can be made
  available dynamically the Grid
• well-defined properties in terms of environment
  definition and resource usage enforcement
• Examples
• A physical cluster booted to a desired
  configuration (e.g. Cluster on Demand)
• A Grid3 node dynamically configured using Pacman
• A cluster partition configured with a hypervisor
• A VM representing an OSG configuration enforcing
  memory and CPU usage
• Workspaces can be implemented using a variety of
  technologies
• VMs are the most promising

14
Virtual Workspace

• Environment Aspect (workspace meta-data)
• Information/state that outlives its deployment
• Generic information (name, time to live)
• Attested software partition information OS, OSG
  configuration, application installation, etc.
• Services ssh, GRAM, pre-configured job
• Resource allocation request (deployment time)
• Flexibly negotiated within desired constraints
• See GGF WS-Agreement standard
• Memory, disk, networking, etc.
• See GGF JSDL standard
• On deployment the actual resource allocation
  information becomes available for inspection
• Atomic workspaces and virtual clusters
• Clusters are simply aggregate workspaces

15
Deploying Workspaces in the Grid

• Define workspace environment
• Manage workspace
• Negotiate workspace deployment characteristic

Workspace Wizard (VW Factory)
manage workspace environment
Workspace Management Service (VW Repository)
workspace metadata
Workspace Service (VW Manager)
terminate workspace deployment
manage activities within the workspace
16
Current Implementation

• Current prototype using Globus Toolkit 4
• Leveraging standard Grid Service features
• Workspace Wizard
• Returns workspace meta-data
• Very rudimentary implementation
• Workspace Service
• Create takes workspace meta-data and a
  deployment descriptor
• Manage
• renegotiate resource allocation
• Also traditional Grid Service management TTL,
  etc.
• Destroy
• Different options pause, shutdown or destroy
• First tech preview release expected later this
  month

17
How dynamic is the deployment?

• Automatic
• Protocol-based
• Moving towards better articulation of migration
• Renegotiation of resource allocation
• How fast is this deployment?
• Deployment of workspace for EMBOSS suite
• Manual 45 minutes
• Based on pre-configured Vmware VMs 6 minutes
• Based on pre-configured Xen VM
• How much overhead does workspace deployment add
  over what we have today?

18
How much deployment overhead are we adding?

• GRAM job execution
• GRAM job execution in a paused Xen VM
• job execution in a booted Xen VM (pre-configured
  job)

• Using a paused VM allows us to save on
  initiation time

19
Workspace Service Virtual Clusters
20
Workspace Deployment Across Technologies

• Basic node configuration (/-boot from image)
• Cluster on Demand, PXE, bcfg
• On the order of many minutes (30 minutes)
• Refining configuration, creating access
• Dynamic account with workspace service (mostly GT4 request processing time)
• Refining Installation 2 hours to configure an
  ATLAS node using Pacman
• Virtual machines
• Deploying images
• Xen 100 ms
• VMware Workstation several seconds

21
Nested Workspaces
It is easier to maintain a few hypervisor
configurations than thousands of user
configurations. Those can be deployed in virtual
machines.
22
Virtual Playgrounds
Application
23
Ongoing Work on Workspaces

• Dynamic resource management with VMs
• Virtual clusters, fine-grained resource
  mangement, migration, moving towards economic
  management
• X. Zhang, T. Freeman
• IP overlay network for virtual machines
• Management infrastructure for VM IP addresses
• T. Freeman L. Chen
• Secure management of VM images
• Image attestation and verification
• Handling image distribution
• Managing workspace identity
• W. Lu, T. Freeman, F. Siebenlist
• Deployment
• Edge Services for OSG with F. Wuertherwein A.
  Rana

24
Related Work

• In-Vigo
• VM-based infrastructure for the Grids
• VM deployment, virtual storage, virtual networks
• Renato Figueiredo, Jose Fortes
• Virtuoso
• VNET virtualizing networks
• Peter Dinda lab
• VIOLIN
• Isolated, virtual networks for VMs
• Dongyan Xu lab
• Cluster on Demand
• Clusters of VMs on demand, also networking,
  resource management
• Jeff Chase and lab

25
The Challenges that Lie Ahead

• Deployment
• How do I prepare a cluster for VM execution?
• Reserve and publish
• Site-specific versus Grid-specific
• What security trade-offs are acceptable?
• How will VM usage change site configuration?
• And many, many others
• Environment configuration management
• How to configure and manage a VM?
• GGF CDDLM working group
• Packaging infrastructures
• Security
• Huge potential how are we going to leverage it?
• Economics, Grid markets, and many others

26
Conclusions

• Virtual is the new real!
• Virtualization is emerging as an important
  abstraction layer in the Grids
• Virtual workspaces are cornerstone of this new
  abstraction layer
• Rapidly developing VM technology has the
  potential to implement a computon for the Grids
• Fast, accessible VMs
• critical mass implementation for virtual
  workspaces
• Two sides to providing computation on tap
• Abstractions and enforcement mechanisms
• Protocols
• There is much ongoing work in VMs but even more
  challenges still like ahead

27
If you like a challenge, give us a call

• keahey_at_mcs.anl.gov