Quick Overview of NPACI Rocks

1
Quick Overview of NPACI Rocks

• Philip M. Papadopoulos
• Associate Director, Distributed Computing
• San Diego Supercomputer Center

2
Seed Questions

• Do you buy-in installation services? From the
  supplier or a third-party vendor?
• We integrate. Easier to have vendor integrate
  larger clusters
• Do you buy pre-configured systems or build your
  own configuation?
• Rocks is adaptable to many configurations
• Do you upgrade the full cluster at one time or in
  rolling mode?
• Suggest all at once (very quick with Rocks) can
  be done as a batch job.
• Can support rolling, if desired.
• Do you perform formal acceptance or burn-in
  tests?
• Unfortunately, no. Need more automated testing.

3
Installation/Management

• Need to have a strategy for managing cluster
  nodes
• Pitfalls
• Installing each node by hand
• Difficult to keep software on nodes up to date
• Disk Imaging techniques (e.g.. VA Disk Imager)
• Difficult to handle heterogeneous nodes
• Treats OS as a single monolithic system
• Specialized installation programs (e.g. IBMs
  LUI, or RWCPs Multicast installer)
• let Linux packaging vendors do their job
• Penultimate
• RedHat Kickstart
• Define packages needed for OS on nodes, kickstart
  gives a reasonable measure of control.
• Need to fully automate to scale out (Rocks gets
  you there)

4
Scaling out

• Evolve to management of two systems
• The front end(s)
• Log in host
• Users home areas, passwords, groups
• Cluster configuration information
• The compute nodes
• Disposable OS image
• Let software manage node heterogeneity
• Parallel (re)installation
• Data partitions on cluster drives untouched
  during re-installs
• Cluster-wide configuration files derived through
  reports from a MySQL database (DHCP, hosts, PBS
  nodes, )

5
NPACI Rocks Toolkit rocks.npaci.edu

• Techniques and software for easy installation,
  management, monitoring and update of clusters
• Installation
• Bootable CD floppy which contains all the
  packages and site configuration info to bring up
  an entire cluster
• Management and update philosophies
• Trivial to completely reinstall any (all) nodes.
• Nodes are 100 automatically configured
• Use of DHCP, NIS for configuration
• Use RedHats Kickstart to define the set of
  software that defines a node.
• All software is delivered in a RedHat Package
  (RPM)
• Encapsulate configuration for a package (e.g..
  Myrinet)
• Manage dependencies
• Never try to figure out if node software is
  consistent
• If you ever ask yourself this question, reinstall
  the node

6
Rocks Current State Ver. 2.1

• Now tracking Redhat 7.1
• 2.4 Kernel
• Standard Tools PBS, MAUI, MPICH, GM, SSH,
  SSL,
• Could support other distros dont have staff
  for this.
• Designed to take bare hardware to cluster in a
  short period of time
• Linux upgrades are often forklift-style. Rocks
  supports this as the default mode of admin
• Bootable CD
• Kickstart file for Frontend created from Rocks
  webpage.
• Use same CD to boot nodes. Automated integration
  Legacy Unix config files derived from mySQL
  database
• Re-installation (we have a single HTTP server,
  100 Mbit)
• One node 10 Minutes
• 32 nodes 13 Minutes
• Use multiple HTTP servers IP-balancing switches
  for scale

7
More Rocksisms

• Leverage widely-used (standard) software wherever
  possible
• Everything is in RedHat Packages (RPM)
• RedHats kickstart installation tool
• SSH, Telnet (only during installation), Existing
  open source tools
• Write only the software that we need to write
• Focus on simplicity
• Commodity components
• For example x86 compute servers, Ethernet,
  Myrinet
• Minimal
• For example no additional diagnostic or
  proprietary networks
• Rocks is a collection point of software for
  people building clusters
• It evolving to include cluster software and
  packaging from more than just SDSC and UCB
• ltyour-software.i386.rpm your-software.src.rpm
  heregt

8
Rocks-dist

• Integrate RedHat Packages from
• Redhat (mirror) base distribution updates
• Contrib directory
• Locally produced packages
• Local contrib (e.g. commerically bought code)
• Packages from rocks.npaci.edu
• Produces a single updated distribution that
  resides on front-end
• Is a RedHat Distribution with patches and updates
  applied
• Kickstart (RedHat) file is a text description of
  whats on a node. Rocks automatically produces
  frontend and node files.
• Different Kickstart files and different
  distribution can co-exist on a front-end to add
  flexibility in configuring nodes.

9
insert-ethers

• Used to populate the nodes MySQL table
• Parses a file (e.g., /var/log/messages) for
  DHCPDISCOVER messages
• Extracts MAC addr and, if not in table, adds MAC
  addr and hostname to table
• For every new entry
• Rebuilds /etc/hosts and /etc/dhcpd.conf
• Reconfigures NIS
• Restarts DHCP and PBS
• Hostname is
• ltbasenamegt-ltcabinetgt-ltchassisgt
• Configurable to change hostname
• E.g., when adding new cabinets

10
Configuration Derived from Database
Automated node discovery
mySQL DB
Node 0
insert-ethers
Node 1
makehosts
makedhcp
pbs-config-sql
Node N
/etc/hosts
/etc/dhcpd.conf
pbs node list
11
Remote re-installationShoot-node and eKV

• Rocks provides a simple method to remotely
  reinstall a node
• CD/Floppy used to install the first time
• By default, hard power cycling will cause a node
  to reinstall itself.
• Addressable PDUs can do this on generic hardware
• With no serial (or KVM) console, we are able to
  watch a node as installs (eKV), but
• Cant see BIOS messages at boot up
• Syslog for all nodes sent to a log host (and to
  local disk)
• Can look at what a node was complaining about
  before it went offline

12
Remote re-installationShoot-node and eKV
192.168.254.254
Remotely starting reinstallation on two nodes
192.168.254.253
13
Monitoring your cluster

• PBS has a GUI called xpsmon. Gives a nice
  graphical view of up/down state of nodes
• SNMP status
• Use the extensive SNMP MIB defined by the Linux
  community to find out many things about a node
• Installed software
• Uptime
• Load
• Slow
• Ganglia (UCB) IP Multicast-based monitoring
  system
• 20 different health measures
• I think were still weak here learning about
  other activities in this area (e.g. ngop, CERN
  activities, City Toolkit)

14
Cern

• Cern.ch/hep-proj-grid-fabric
• Installation tools wwwinfo.cern.ch/pdp