NCSC Overview

1
NCSC Overview

• SEACOOS
• 27 January 2003

2
Computational Resources

• SGI Origin 2400
• 48 R12000 processors 400MHz
• 24 Gbyte memory
• IRIX 6.5 (one instance)
• IBM SP
• 720 POWER3 processors 375MHz
• 360 Gbyte memory
• AIX 4.3 (180 instances)

3
Other Resources

• Mass Storage System
• DMF
• Hosted on Origin 2400
• /dmf/edu/login
• Software suite
• /usr/local/apps
• /usr/local/bin
• /usr/local/lib
• Scientific Support
• Porting
• Optimization

• NCREN
• High performance IP network
• Abilene connectivity
• OC-48
• Commodity internet connectivity
• Level 3
• QWEST
• Sprint
• ISP peering
• Video Network

4
Origin 2400

• Architecture
• ccNUMA
• Physically distributed memory
• Single address space
• Hardware cache coherence
• Operating Environment
• Single system image
• Interactive and batch processes share system
  (thus may impact each other)

5
Origin 2400

• MIPS R12000 Processor
• 400 MHz
• 8 MB Cache
• 4 instructions/clock
• Load/store
• ALU1
• ALU2
• FPADD
• FPMUL
• 64-bit addresses
• 64-bit floating-point units

• MIPSpro Compilers
• CC, cc, f77, f90
• ABI (application binary interface)
• 32
• n32 (recommended)
• 64 (data size gt2GB)
• Optimization
• O3 (suggested)
• SWP, IPA, LNO

6
Origin 2400

• sonoma.ncsc.org
• Interactive process limits
• 30 minutes CPU time
• 512 MB
• 4 processors
• Batch System
• NQE
• qsub
• qstat
• qdel

7
IBM SP

• Architecture
• Cluster 180 nodes
• SP switch interconnect
• SMP nodes
• Operating Environment
• Each SMP node runs its own copy of AIX
• POE provides some single system features for
  the cluster
• Various node types
• Interactive
• Batch
• File server

8
IBM SP

• Node Architecture
• 4 POWER3 375 MHz processors
• 2 Gbyte memory
• Switch network
• Ethernet network
• Disk

9
IBM SP

• POWER3
• 375 MHz
• Cache 64KB L1 8MB L2
• 7 functional units
• 64 bit floating-point units

10
IBM SP

• XL Compilers
• xlf, xlf90, xlf95, xlc, xlC
• Optimization
• O3
• qarchpwr3
• Thread safe
• xlf_r, xlf90_r, ..
• MPI
• mpxlf, mpxlf90,

• Parallel job execution controlled by POE
• There is no mpirun command on the SP

11
IBM-SP Filesystems

• File system types
• local
• system directories
• /tmp (1GB)
• NFS
• /home
• /usr/local
• /share (300 GB)
• GPFS
• /gpfs
• 1 TB
• DMF
• Only on interactive nodes

...
SP Node
SP Node
SP Node
SP Node
Switch
12
IBM SP

• Interactive nodes
• sps001.ncsc.org
• sps002.nscc.org
• sps003.ncsc.org
• Batch nodes
• Batch jobs can use up to 512 processors (long
  wait)
• Jobs should not share processor with other user
  processes

• Batch System
• Loadleveler
• Easy/effective MPI scheduling
• OpenMP somewhat more complicated and less well
  behaved
• llsubmit
• llq

13
Roadmap

• IBM p690 (Regatta)
• 32 POWER4 1.3 GHz processors
• 128 GB memory
• AIX 5.1
• LSF
• Expect to move to production during 1Q 2003

14
Roadmap

• Scalable Parallel Applications
• Migrate from IBM SP to commodity linux clusters
• IA32 XEON processor based 2-way nodes
• LSF batch system
• Expect initial small clusters during 1Q 2003.
  Larger cluster 2Q 2003.

• Mass Storage System
• Move off of Origin system during 2003
• Authentication
• Certificate based single sign on for all NCSC
  systems
• Environment
• Common batch system (LSF)
• Common home directories
• Common application directories
• Isolate interactive use from batch systems

15
Projects/Accounts

• Universities begin contributing to funding
  beginning July 2003
• Allocation process to be revised but proposal
  is to maintain existing structure from user
  perspective
• Familiarization Accounts
• Computational Science Faculty Accounts
• Allocated Research Accounts
• Classroom Accounts

• Monitoring use
• usage command on any production system
• Allocations and usage based on CPU-Hrs used