Chemical Supercomputing on the Cheap: Cobalt cluster

1
Chemical Supercomputing on the Cheap Cobalt
cluster

• 94GFlops computer system at cdn3680/gigaflop
• Mar98 Feb99

S. Patchkovskii, R. Schmid, and T. Ziegler
Department of Chemistry, University of Calgary,
2500 University Dr. NW, Calgary, Alberta, T2N 1N4
Canada
2
Disclaimer

• We are chemists, not computer scientists we do
  not build computers for a living we build them
  for doing chemistry

3
Outline of the talk

• What do we need, and why build our own?
• What went in Nodes and communication hardware
• What it runs on System and application software
• How does it work Stability, performance, and
  resource utilization
• What can it do Chemical research with Cobalt
• Conclusions and Acknowledgements

4
Criteria for Cobalt design

• Should provide a useful resource for ?5 years
• Minimal hardware purchase and maintenance cost
• No low-volume boutique hardware
• No experimental bleeding edge hardware
• Minimal system maintenance
• Use software which has been around for a while
• If it works, do not tinker with it!

5
Cobalt Nodes
Compaq Personal Workstation 500au.
For a comparison, in May99 a top of the line
550MHz Intel Xeon workstation with 512Kb of L2
cache achieved 24.4 SpecInt 95 and 17.1 SpecFP 95
and cost about cdn4400 from Dell.
() SpecInt and SpecFP values estimated from
published results for a 500au system with 2Mb L3
cache.
6
Cobalt Network
4 x 24-port 3com SuperStack II 3300, with matrix
module. Maximum intra-switch bandwidth
1Gbyte/sec.
Latency and bandwidth measured with Larry McVoys
Lmbench using otherwise idle nodes.
7
Keeping Cool
8
The Cluster
Computers on benches all linked together
9
System software

• Operating system Tru64 (Digital) Unix
• Remote boot/local scratch setup
• Centralized error monitoring facility
• NFS and NIS, user files distributed over nodes
• C, C, and Fortran bundled under CSLG
• Parallel programming
• PVM and MPI
• Batch queuing
• DQS

10
Application software
() Gaussian supports cluster environments with
Network Linda - an extra-cost package is not
available on Cobalt
11
Total construction cost
12
Parallel execution ADF

• Time-limiting step Cartesian space numerical
  integration
• Low demands on bandwidth and latency
• Infrequent synchronization due to replication of
  serial sections
• Parallel scaling is limited by static local
  balancing

13
ADF An example

• Nitridoporphyrinatochromium
• Full geometry optimization
• 38 atoms
• 580 basis functions
• C4v symmetry
• 45Mbytes of memory
• Serial time 683 minutes

14
Parallel execution PAW

• Time-limiting step Fast Fourier transforms
• Extensive memory requirements
• Sensitive to bandwidth and latency gt20 link
  bandwidth utilization on Cobalt
• Frequent synchronization, massive data exchange
• Parallel scaling is limited by the network
  interconnect

15
PAW An example

• CH3I Rh(CO)2I2-, SN2 reaction
• 11Å unit cell
• Serial time per step 83 seconds
• Memory 231Mbytes

16
Stability Node uptime
17
Average CPU utilization
Average CPU utilization over last 9 months 77
18
Job size distribution
19
Chemical research with Cobalt

• Dynamical simulation of the ethylene insertion
  step in single-site polymerization, using a
  realistic model of the counterion (M. Chan and T.
  Ziegler)
• PAW simulation 6 months x 8 Cobalt nodes 25,000
  time steps

20
Chemical research with Cobalt

• Polymerization and co-polymerization catalyzed by
  late transition metal complexes (A. Michalak and
  T. Ziegler)
• ADF study 6 months x 12 Cobalt nodes

21
Chemical research with Cobalt

• Theoretical prediction of EPR g-tensors of
  transition metal complexes (S. Patchkovskii and
  T. Ziegler)
• 4 weeks x 4 Cobalt nodes

22
Conclusions

• Commodity-based computer clusters are uniquely
  suited for quantum chemistry research
• Medium-sized clusters can be built easily, at
  PC-level per-node price points
• Such clusters can be utilized efficiently with
  only minimal system administration and
  maintenance
• Go for it!

23
Credits and Acknowledgements

• Financial support for the construction of Cobalt
  was provided by
• Canada Foundation for Innovation
• Alberta Intellectual Infrastructure Partnership
• Department of Chemistry, UofC
• Scientific Chemistry Simulations, Inc.
• Nova Chemicals
• Mitsui Chemicals
• Many thanks to all members, current and past, of
  Prof. Tom Zieglers research group at UofC