High Performance Scientific Computing

1
High PerformanceScientific Computing
http//www.hpsc.csiro.au
2
CSIRAC

• Commissioned 1949
• 1kHz clock speed
• 20 bit words
• 2K RAM
• 5K disk
• 7 Tonnes
• 30kW

• Sample Applications
• Snowy hydroelectric scheme
• Melbourne electricity grid
• Forestry Commission
• Solar radiation tables
• Structure of metals

3

• CSIRAC is the 4th electronic computer ever built
• Oldest to survive to present day (Melbourne
  Museum)

4
Evolution of Peak Capabilities
5
CSIRO HPSC in 2005
CSIRO
CSIRO-IT
IVEC
eScience
CSIRO HPSC
Local Clusters
Cycle Harvesting
HPCCC
Software Pool
Bureau of Meteorology
Central Systems
APAC
6
HPCCC is a joint venture of CSIRO and the Bureau
of Meteorology (BoM) Located at BoM Head
Office Level 11, 700 Collins St, Docklands,
Victoria http//www.hpccc.gov.au
7
Docklands, Melbourne
8
Facilities
9
NEC SX-6 TX7
NEC SX-6 parallel vector supercomputer 28 nodes
224 CPUs 1.8TB memory 1.8TFlop/s 2 NEC
TX7/i9510 dedicated disk servers 32 1.3GHz ia64
CPUs 22TB disk (130GFlop/s) 5 3 nodes
CSIRO, remainder BoM
10
NEC SX-6
11
NEC TX7
12
SGI Altix 3700

• NUMA shared memory architecture
• 64 1.3GHz ia64 CPUs
• 174GB memory 6TB disk 330GFlop/s (peak)
• Upgrade to 128 CPUs 238GB
• General purpose compute server
• Front end for datastore

13
SGI Altix
14
IBM eServer 1350
General Purpose cluster 28 nodes dual 3.2GHz Xeon
2GB RAM 28 nodes dual 3.2GHz Xeon 4GB RAM 28
nodes dual 3.2GHz EM64T 2GB RAM
Infiniband 1.1TFlop/s (peak) Dedicated
cluster 26 nodes dual 3.2GHz Xeon 2GB RAM Used
by CSIRO Marine Atmospheric Research in Perth
and Aspendale xCAT cluster management tools
15
IBM Cluster
16
Data Migration Facility
StorageTek 9310 Powderhorn robot tape silos 1.3
Petabyte maximum capacity 110TB primary holdings
(doubles every 8 months) Automatic migration of
large/old files between primary and secondary
disk and near-line tape storage
17
Datastore
18
Datastore
19
Batch Queues

• Altix and clusters use Torque
• NECs use NQSII batch system with ERS fair share
  scheduler

Why hasnt my job started?
Why hasnt my job finished?
1995
2005
20
Data Services

• HSM integrated with Altix system
• Hosting a Storage Resource Broker (SRB) zone
• Federated data for ARRC
• ATNF evaluation
• OPeNDAP remote data access to
• authenticated users

21
User Support

• Helpdesk facility
• Problem reporting
• management (wreq)
• Application support
• Web manuals
• User guides
• APAC Software map
• Technical bulletins
• Newsletter

22
Applications Software

• Local network license serving
• MSC suite
• CFD-Ace
• Ensight
• S-Plus
• Gaussian
• NAG
• Mathematica, Matlab

23
HPSC Partnerships
partner or perish
24
HPSC Partnerships

• Bureau of Meteorology HPCCC joint venture
• CSIRO and APAC
• Partner access to National Facility
• APAC Grid Program
• Computational Tools Techniques
• Divisional membership of iVEC and TPAC
• HPSC co-investment in new SGI at iVEC
• Dedicated clusters for internal clients

25
Scientific Computing Desktop
Computational Tools Techniques (CTT)
Program Single sign-on access to multiple
machines/sites
26
CSIRO Bioinformatics Facility

• CBF operating since 2003
• Based at Black Mountain, Canberra
• 5 Dell P4 server nodes
• 2700GB SCSI RAID
• 66 nodes with dual P3 30GB disk
• 327 user accounts from 9 CSIRO Divisions

http//www.cbf.csiro.au
27
CSIRO User Base

• CSIRO has gt6000 staff distributed across gt110
  sites
• Broad range of scientific disciplines and
  capabilities
• Strategic, co-investment and contract research
• In-house and commercial software

28
High Performance Computing in CSIRO
Division HPCCC IVEC APAC NF CBF Local Software
Australia Telescope National Facility ?
ENSIS / Forestry Forest Products ?
Energy Technology ? ?
Entomology ?
Exploration Mining ? ? ? ?
Food Science Australia
Industrial Physics (?)
ICT Centre ?
Land Water (?) ?
Livestock Industries ? ?
Manufacturing Infrastructure Technologies ? ? ? ?
Marine Atmospheric Research ? ? ?
Mathematical Information Sciences ? ? ? ?
Minerals (?) ?
Molecular Health Technologies ? ? ? ?
Plant Industry ?
Petroleum Resources ? ? ? ?
Sustainable Ecosystems
Textile Fibre Technologies ?
29
BLUElink Project

• Operational forecasting of ocean currents
• CSIRO
• Bureau of Meteorology
• Royal Australian Navy

Andreas.Schiller_at_csiro.au
30
BLUElink Ocean Data Assimilation
Correlations in surface height with reference
location (star)
Peter.Oke_at_csiro.au
31
Ocean Forecasting Model
Daily averaged eddy structures along Leeuwin
current
Andreas.Schiller_at_csiro.au
32
Relocatable Ocean-Atmosphere Model
ROAM High resolution regional coupled
ocean-atmosphere model
e.g. localised region in Gulf of Carpentaria
Andreas.Schiller_at_csiro.au
33
Relocatable Ocean-Atmosphere Model
Atmospheric component of coupled regional model
Used to forecast wind conditions during RAN
relief operations in Banda Aceh
Debbie.Abbs_at_csiro.au
34
Modelling of Tropical Cyclones
TC Bobby, 1995 colour wind velocity Grey
isosurface total ice field
Debbie.Abbs_at_csiro.au
35
Discrete Element Modelling
3D impeller mixer
Grinding mill
Plough Share mixer
Paul.Cleary_at_csiro.au
36
Smoothed Particle Hydrodynamics
SPH meshless continuum mechanics method Well
suited to complex geometries, mixtures and free
surfaces e.g. casting of aluminium ingots
Paul.Cleary_at_csiro.au
37
SPH of High Pressure Die Casting
Rocker cover
Steering column
Differential cover
Paul.Cleary_at_csiro.au
38
SPH for Fluid Animation
Paul.Cleary_at_csiro.au
39
SPH for Fluid Animation
Paul.Cleary_at_csiro.au
40
Fire Modelling
Sydney Harbour Tunnel Fire Smoke Ventilation
Yunlong.Liu_at_csiro.au
41
Smoke Generation in Train Fires
Yunlong.Liu_at_csiro.au
42
Heat Release in Train Fires
Temperature at carriage door
Collaboration with Queensland Rail
Yunlong.Liu_at_csiro.au
43
Grain scale
Predictive Geomechanics
Crustal scale
Mantle scale
Alison.Ord_at_csiro.au
44
Groundwater Contamination

• Simulated distribution of tracer adsorbed to
  hydrocarbon ganglia passing through a layered
  aquifer
• Central injection well
• 4 extraction wells
• Contours fluid head

Mike.Trefry_at_csiro.au
45
Mineral Processing
3D fluidised bed model of fluid coker
stripper Used to extract liquid hydrocarbons,
fuel gas and coke from bitumen derived from oil
sands
Syncrude plant
Phil.Schwarz_at_csiro.au
46
Stratigraphic Forward Modelling
Simulate transport and deposition of sediment in
response to climate, current, river inputs
etc Effect of global warming on seabed and
coastal features Location and accessibility of
oil reservoirs
Cedric.Griffiths_at_csiro.au
47
CO2 Geosequestration
Stress changes on fault
2525 m 60 yr
Convection of dissolved gas
Calcite
Free gas
pH
Lincoln.Paterson_at_csiro.au
48
Polymer Simulation
Non-equilibrium molecular dynamics of
concentrated dendrimer solutions in shear flow
BTodd_at_it.swin.edu.au
49
CSIRO HPSC Team
Rhys Francis Director
Erika Stojanovic Executive Coordinator
Alf Uhlherr Science Strategy Manager
Gareth Williams Outreach Manager
Robert Bell Chief Technical Officer
Jeroen van den Muyzenberg Senior System
Administrator
Len Makin User Support Manager
Rowan McKenzie Senior Software Engineer
King Lung Chiu Software Engineer
Bob Smart Systems Integration Networking
Polly Morgan System Administrator
50
http//www.hpsc.csiro.au