Cornell Theory Center (CTC) is a high-performance computing and interdisciplinary research center at Cornell University. Scientific and engineering research projects supported by CTC represent a variety of disciplines, including biology, behavioral and

1
Cornell Theory Center
Cornell Theory Center (CTC) is a high-performance
computing and interdisciplinary research center
at Cornell University. Scientific and engineering
research projects supported by CTC represent a
variety of disciplines, including biology,
behavioral and social sciences, computer science,
engineering, geosciences, mathematics, physical
sciences, and business. CTC is unique among HPC
centers in that it is a showcase for
Windows-based high-performance computing. In
1999, we successfully moved our research
community from a traditional, proprietary
supercomputer running UNIX to an
industry-standard cluster complex.
Microsoft and Dell are major
supporters of CTC, which is directed
by computer scientist Thomas F. Coleman.
http//www.tc.cornell.edu/
2
Case Studies

• Plant breeders and molecular biologists
  generating detailed information on the genetic
  makeup of crops e.g. rice and wheat
• Management model for manufacturing that helps
  managers find best balance between capacity,
  inventory, and subcontracting
• New stochastic path approach for modeling
  protein dynamics in more meaningful time scales
• New social science research made possible by
  developing new statistical methods for analyzing
  census data
• Simulation of crack growth and strength
  evaluation of airplane components

3
Compute Power
Over 700 Processors Giganet Switch (100 GB/sec)
interconnect 100 Mb switched Ethernet 1-4 GB
Ram/Node, 27-56 GB Disk/Node Serial, Dual, and
Quad processors Dell PowerEdge Servers Pentium
processors, 600 Mhz 2.4Ghz 256 KB - 2 MB
Cache/Processor (SMP) Tivoli Backup
SystemMicrosoft Distributed File
SystemFileservers, 5 Terabytes
Storage http//www.tc.cornell.edu/Services/Docs
/resources.asp
11/ 2002 256-processor cluster in top 100 618.3
gigaflops over 256 2.4Ghz Linpack benchmark
4
Visualization
Consultation on how to visualize data Creation
of animations, videos, publication graphics
Independent study opportunities for
undergraduate students Training in the use of
OpenDX Video production suite Three-wall
Windows CAVE Virtual reality environment (show
VRML files) Write your own with OpenGL or
OpenInventor Free by appointment with
kford_at_tc.cornell.edu Computer Lab of
Visualization Workstations http//www.tc.cornell.
edu/services/vis/index.asphttp//www.tc.cornell.e
du/services/docs/cave/ http//www.tc.cornell.edu/s
ervices/edu/topics/topics.asp?sectionopendxhttp
//www.tc.cornell.edu/services/Allocations/Account/
visform.asp http//www.tc.cornell.edu/Services/Do
cs/Collab/
5
Software
See online list for software available on the
login nodes, various types of compute nodes and
the Visualization Collaboratory. This list is
updated as new packages are added. Software is
added by researcher's request if feasible.
C/C compilers Microsoft Visual C, Intel
C/C, Gnu gcc/g Fortran compilers Intel
Fortran, PGF77, Compaq Visual Fortran Libraries
MPI/PRO parallel programming, Numerical Libraries
Applications software Computational chemistry,
Data analysis, Finite element modeling, Fluid
dynamics, Visualization tools, . .
. http//www.tc.cornell.edu/Services/Software/ht
tp//www.tc.cornell.edu/services/edu/topics/softwa
re/
6
Allocations

• Research allocation
• Submit proposals at any time.
• Undergraduate or graduate students must have
• their requests submitted by a faculty advisor.
  
• Non-renewable six-month Exploratory
• Allocations of 100 hours. Simplified
  application.
• Charging is calculated using wall-clock hours
  i.e. you will be charged for the time nodes are
  reserved for your use, not for the time you are
  actually using them.
• Wall-clock hours on all of the node types are
  charged the same flat wall-clock rate, regardless
  of number of processors. Parallelize!
• Charges for compute nodes only, not for login
  nodes or the collaboratory machines.
• http//www.tc.cornell.edu/services/Allocations/
  http//www.tc.cornell.edu/services/edu/topics/sta
  rt/

7
Access
Connect to a login node (ctclogin1, ctclogin2, )
to compile and submit jobs. The login nodes are
SSH servers, download an SSH client. Login.bat is
run automatically. For full windows desktop
functionality from your machine, use a terminal
server, Microsoft Terminal Server Client for
Windows, Citrix ICA for non-windows. For command
line access, use telnet ssh or telnet
ctclogin1.tc.cornell.edu http//www.tc.cornell.
edu/Services/Docs/Access/
8
Development Environment
Programs can be compiled on your Windows machine,
in the Collaboratory in Rhodes Hall, or from one
of the login nodes reached via Terminal Server or
telnet. Work from the command line or in a GUI
environment. GUI Development environment Start
Programs Microsoft Visual Studio .NET
Microsoft Visual Studio .NET Command
line Start Run cmd (bring up a command
line window) setup_intelc32 (set environment
variables using setup script ) cl hello.c
(compile serial) mpicc hello.c (compile MPI)
To see the full list of setup scripts,
issue call "H\CTC Tools\setup_list.bat
9
Compute Environment
BatchScheduler
Compute Nodes
CCS
Login Nodes
H
ctclogin
Fileserver
10
Serial Batch Job Script

• Prepare your executable
• Write a batch script file
• From one of the login nodes, submit the script to
  the scheduler
• Check the status of the queue, your job, etc The
  remainder of the steps can be done by hand or
  script
• When your job has started copy files to the
  local T drive run the program
• When your job has ended Copy any output files
  back to H clean up the T drive end the job

Compute Nodes
11
Parallel Batch Job Script
The main batch script runs on the Master
node Start the script with the four required
REM CCS keyword commands Create the machines
file on T of the master node Use mpirun to run
setup.bat on all nodes listed in the machines
file Create a working subfolder on the compute
node T drive Copy the executable (and data
files, if necessary) to each node Copy any files
needed only on the Master node Move the machines
file from T to T\your_userid Use mpirun to run
the MPI program executable on all nodes listed in
the machines file Your_program.exe runs on all
the nodes in the machines file Use mpirun to run
cleanup.bat on all nodes listed in the machines
file Copy the output files from each node to
H Cleanup the working subdirectory on each
node Release the nodes
12
Parallel Job on a Single Node (page 1 of 2)

• REM CCS account your_userid
• REM CCS type batch
• REM CCS nodes 1
• REM CCS minutes 10
• REM CCS requirements 1_at_development
• T
• mkdir USERNAME
• cd USERNAME
• del /Q /S T\USERNAME\.
• call machinemaker

Call machinemaker (in CTC Tools, therefore
already in your path) to produce a machines list,
listing the node(s) your job has been allocated.
Note if you have more than one job running, you
can assure that you'll create the machines file
for the right jobid by issuing machinemaker jobid
13
Parallel Job on a Single Node (page 2 of 2)

• REM Copy the executable and data files to the
  node
• copy H\users\USERNAME\lab\wavesend.exe
  T\USERNAME
• copy H\users\USERNAME\lab\wave.in
  T\USERNAME
• mpirun -wd T\USERNAME -np 2 wavesend.exe
  1gtwave.out 2gtwave.err
• REM Copy any output created by the master task
  back to the H drive.
• copy T\USERNAME\wave. H\users\USERNAME\lab
• REM Cleanup the working subdirectory on each node
  
• del /Q /S T\USERNAME\.
• REM Release the nodes

Run the MPI program with mpirun. Set -np to the
number of tasks. Set -wd to the working directory
in the case where your files are not in your
current directory.
14
Parallel Job on Multiple Nodes (page 1 of 4)
REM CCS account your_userid REM CCS type
batch REM CCS nodes 4 REM CCS minutes 10 REM
CCS requirements 4_at_development REM Move to the
T drive T call
machinemaker REM Set the communication mode to
TCP or VIA set MPI_COMMVIA
Run machinemaker to create a file called
"machines", which contains a list of machines
allocated to this job
15
Parallel Job on Multiple Nodes (page 2 of 4)
mpirun -np 4 H\users\USERNAME\lab\ex3setu
p.bat
Use mpirun to call a batch script called
setup.bat. Setup.bat to copy the executable,
data files, etc onto the nodes listed in the
machines file. When calling setup.bat, set -np
to the number of nodes, not the number of tasks.
REM setup.bat REM Create a working subdirectory
on each node T mkdir your_userid cd
your_userid del /Q /S T\your_userid\.
REM Copy the executable (and data files, if
necessary) to each node copy H\users\your_userid\
lab\wavesend.exe T\your_userid
16
Parallel Job on Multiple Nodes (page 3 of 4)
REM The input file is only needed on the master
node copy H\users\USERNAME\lab\wave.in
T\USERNAME cd
USERNAME move T\machines T\USERNAME
mpirun -wd T\USERNAME\ -np 8
wavesend.exe 1gtwave.out 2gtwave.err
copy T\USERNAME\prog. H\users\USERNAME\l
ab
Move the machines file from T to T\your_userid.
It couldn't be put there earlier.
Run the MPI program with mpirun. Set -np to the
number of tasks. Set -wd to the working directory
used by mpirun.
Copy any output created by the master task back
to H.
17
Parallel Job on Multiple Nodes (page 4 of 4)
mpirun -np 4 H\users\USERNAME\la
b\ex3cleanup.bat REM Release the
nodes ccrelease
Next another script (cleanup.bat) is run all
nodes to both copy any other output files back to
the H drive and to clean up the subdirectory.
When calling cleanup.bat, set -np to the number
of nodes,not the number of tasks.
REM In this example, only the master node has
output files. copy T\your_userid\wave.
H\users\your_userid\lab REM Cleanup the working
subdirectory on each node del /Q /S
T\your_userid\.
18
MCP
Compute Node
Compute Node
Fileserver (H)
Compute Node
Compute Node
Standard copy Fileserver is accessed N times
Copy with mcp Fileserver is accessed 1 time
19
More Information
Consulting - Send questions about use of the
Velocity Cluster to consult_at_tc.cornell.edu or
call 254-8686 and ask for a consultant. Can
answer specific questions, help with problems,
discuss strategies and code planning.
Strategic users (faculty with proven
application, suitable for scaling) have access to
more in-depth long-term consulting
help. http//www.tc.cornell.edu/http//www.tc.co
rnell.edu/Services/Docs/UserGuide/
http//www.tc.cornell.edu/services/docs/http//ww
w.tc.cornell.edu/services/edu/topics/http//www.t
c.cornell.edu/services/edu/events/
http//www.tc.cornell.edu/Services/Docs/Tips/cygw
in.asp http//www.tc.cornell.edu/Services/edu/topi
cs/writing2/