Notes from Installing a Mac G5 Cluster at SLAC

1
Notes from Installing a Mac G5 Cluster at SLAC

• Chuck Boeheim
• SLAC Computing Services

2
Background

• New joint Stanford/SLAC department Kavli
  Institute for Particle Astrophysics
• Choices for compute cluster included joining
  Opteron purchase, or Mac G5 cluster
• Astronomers like Macs
• Compatible with laptops and desktops they were
  buying

3
Configuration

• 2 Mac G5 file servers
• 2 2.5TB Apple XServ Raid units each
• 2 Mac G5 interactive servers
• 10 Mac G5 compute nodes
• No graphics cards
• No CD drives
• Plus two development servers for us

4
(No Transcript)
5
Physical

• Good overall engineering
• Good airflow quiet
• Hot swap disks in server nodes
• Rails fit standard racks well
• Easy to mount

6
Management

• Supports serial console (57,600 baud)
• No graphics card in G5 server
• Supports serial bios
• But must press button to boot in bios
• No power management
• Supports network install
• Initial disk partition and boot a little tricky
• Subsequent installs can be fully automated

7
Installing the first server

• Secret handshake
• Install from CD
• Run Server Assistant on another Mac to start
  install, and again to configure
• Run Apple Remote Desktop to get graphic login
• Command line alternatives are available, though
  obscure

8
The Partitioning Conundrum

• System has initial OS with one 250GB partition
• Cannot re-partition the boot drive
• One solution
• Boot server from CD
• Ssh to server
• Partition the disk
• Diskutil partitionDisk disk0 3 JournaledHFS
  System 10G JournaledHFS Cache 1G
  JournaledHFS Work 200G
• Netboot may also be possibility
• Fastest solution insert disks in 2nd and 3rd
  bays of server and format

9
Reference Server

• Install second server
• Mac OS X Server 10.3.5
• All patches
• Xcode tools
• Local directory configuration
• AFS
• Fink
• Shutdown, move hard disk to bay 2 of netboot
  server

10
Make a Network Install Master

• Netboot serves disk images
• Uses DHCP, BSDP1, TFTP, and NFS to serve disk
  image to target machine
• Use Network Image Utility to make image of
  reference disk
• Turn on Netboot in Server Admin
• Boot client from network (different secret
  handshake)
• 1 Boot Server Discovery Protocol

11
Network Install Sequence
DHCP Server
Configure with DHCP helper address
Router
DHCP broadcast
DHCP response
BSDP broadcast
Target Server
BSDP response
Netboot Server
TFTP bootloader
NFS disk image
12
Netboot Issues

• Our standard DHCP server worked just fine.
• BSDP is an extension of DHCP
• We put on same subnet as cluster for simplicity
• Should be able to use router helper to put on a
  different subnet
• Network Image Utility had some conflicts

13
Users Looks like Unix

• Use NIS, LDAP, Kerberos for accounts
• Most utilities present ssh, bash, perl, emacs,
  X-windows
• Fink supplies most gnu tools

14
Sysadmins Sorta like Unix

• Strong BSD heritage
• Some things in different places than Linux
• GUI tools for configuring
• Most, but not all, can configure a remote server
• Serversetup and networksetup commands can
  configure most settings
• Directory Access (setup NIS or LDAP) seems not to
  be scriptable

15
Startup Befuddlement

• Not inittab or /etc/rc.d based
• SystemStarter starts many daemons
• Ssh starts out of xinetd
• Ypbind started when needed by Directory Services
• Watchdog starts server processes
• /etc/watchdog.conf
• Sorta like inittab
• Mac OS X 10.4 (Tiger) is going to introduce yet
  another way

16
Authentication Compatibility

• Can use NIS, LDAP, Kerberos, Active Directory
• Supports PAM, but not everything uses it
• LoginWindow (and a few other things) are directly
  kerberos-aware and dont use PAM

17
Infrastructure Fit

• Taylor (SLACs config tool) ran with little
  modification
• About half the config modules worked with little
  change
• The remainder took completely Mac OS X specific
  rewrites

18
Nits

• /etc/passwd is there, but not used. Uses netinfo
  instead for local accounts
• Shadow passwords in different place and format
  than linux, solaris
• afs permissions default is to copy owner mode
  bits to group and other
• Copy file from afs to local, ends up
  world-readable
• Change realmodes in /var/db/openafs/etc/config/s
  ettings.plist
• Mac HFS file system is not case-sensitive
  Makefile and makefile are the same!

19
Conclusions

• Good hardware package, but lacking power
  management
• Network install suitable for cluster operation,
  but still a few wrinkles
• Generally good configuration management tools,
  but some divergence from standard Unix tools