Visualization on New York Blue

1
Visualization on New York Blue Roman
Samulyak AMS Department, Stony Brook
University and Computational Science
Center Brookhaven National Laboratory
2
Outline

• Hardware for visualization
• Main visualization software development projects
• Visualization with VisIt
• Ideas of remote visualization
• VisIt capabilities
• Serial and parallel visualization
• Supported data formats

3
New York Blue

• Features of New York Blue
• IBM Blue Gene
• 128 teraflops
• 18-rack configuration
• 36,864 processors
• Typical simulation generates large data sets
• Data rendering / visualization of data is not
  possible in serial
• New York Blue needs a state-of-art parallel
  visualization software
• Negative features
• Trimmed version of operating system
• Lack of graphics libraries etc.
• Sophisticated visualization software can not be
  installed on IBM Blue Gene

4
Visualization Cluster

• A practical solution suggested by visualization
  experts
• perform visualization on a separate
  visualization cluster
• cluster must share home directories with the
  main supercomputer
• fast wide bandwidth connection to the main
  machine
• LLNL visualization solution
• BlueGeneL 478 TFLOPS sustained and 596 TFLOPS
  peak performance
• Blue Gene/P 1-PFLOPS, 294,912-processors,
  72-racks
• Visualization cluster 512 Intel based nodes
• BNL initial visualization solution
• 4 node 32 processor cluster, IBM PowerPC, 64 bit
  architecture
• 8 processors on a node have shared memory (8 GB)
• shared home directories, 1GB connection to the
  main machine
• Name vis1.bluegene.bnl.gov to
  vis4.bluegene.bnl.gov

5
ASCI Visualization Software

• ASCI (Advanced Strategic Computing Initiative)
  was a supercomputing initiative of the Department
  of Energy, created to help the maintenance of the
  United States nuclear arsenal after the 1992
  moratorium on nuclear testing
• ASCI Supercomputers ASCI White, ASCI Red, ASCI
  Blue Pacific, ASCI Blue Mountain, ASCI Thor's
  Hammer, ASCI Purple, Blue Gene
• Large parallel visualization research
• Supported main visualization projects
• SCIRun, Utah University, targeted shared memory
  machines
• ParaView, Los Alamos and collaborators,
  distributed and shared memory supercomputers
• VisIt, Lawrence Livermore National Laboratory,
  distributed and shared memory supercomputers
• ASCI visualization software became publicly
  available

6
SciRun http//software.sci.utah.edu/scirun.html

• SCIRun is a Problem Solving Environment (PSE),
  for modeling, simulation and visualization of
  scientific problems. It is available for free and
  open source.

7
ParaView http//www.paraview.org

• The ParaView project started started in 2000 as
  a collaborative effort between Kitware Inc. and
  Los Alamos National Laboratory within ASCI Views
  program.
• Today, ParaView development continues as a
  collaboration between Kitware, Sandia National
  Labs, CSimSoft, Los Alamos National Lab, Army
  Research Lab and others.
• ParaView is an open-source, multi-platform
  application designed to visualize data sets of
  size varying from small to very large.
• The goals of the ParaView project include the
  following
• Develop an open-source, multi-platform
  visualization application.
• Support distributed computation models to
  process large data sets.
• Create an open, flexible, and intuitive user
  interface.
• Develop an extensible architecture based on open
  standards.

8
VisIt http//www.llnl.gov/VisIt

• VisIt is a free software application developed
  at LLNL for visualizing and analyzing terascale
  simulation datasets
• Successor to MeshTV
• Similar user interface
• Many more features
• Begun Summer 2000
• Initial version Fall 2002

9
VTK The Visualization ToolKit (not explicitly
related to ASCI)

• The Visualization ToolKit (VTK) is an open
  source, freely available software system for 3D
  computer graphics, image processing, and
  visualization
• VTK consists of a C class library, and several
  interpreted interface layers including Tcl/Tk,
  Java, and Python.
• Professional support and products for VTK are
  provided by Kitware, Inc.
• It may be difficult for some users to write
  parallel visualization codes using VTK libraries

10
VisIt on New York Blue
VisIt is installed on New York Blue visualization
cluster Directory rosamu/software/Visit contains
necessary packages Mesa, Silo, VTK, Python, Qt,
. and two versions of Visit Serial version
rosamu/software/Visit/visit/bin/visit Parallel
version rosamu/software/Visit/visit_par/bin/visi
t
11
VisIt is freely available
Download VisIt from the Web at http//www.llnl.gov
/visit Binary distributions Windows
2000/XP Linux MacOS X AIX (IBM) IRIX
(SGI) Tru64 (Compac) Same user interface on
each platform
12
VisIt is well documented and supported

• Available on the web
• Documentation
• VisIt Getting Started Manual (30 pages)
• VisIt Users Manual (356 pages)
• Getting Data into VisIt (220 pages)
• Includes file with example source code
• VisIt Python Interface Manual (278 pages)
• Frequently asked questions

13
VisIt can access your data

• Reads over two dozen different database formats
  including
• Silo
• Exodus
• PDB (Flash, KullLite, PF3D, LEOS)
• Mili
• SAMRAI
• BoxLib
• Ensight
• VTK (FronTier)
• Vista
• HDF5
• Database reader plug-ins can be developed for
  new formats

Variable types Scalar Vector Tensor
Material Species X,Y pairs
14
Supported mesh types

• 1D Curves
• 2D/3D meshes
• Rectilinear
• Curvilinear
• Unstructured
• Points
• AMR

15
VisIt lets you see and examine your data
Plots Manipulate data or create new data
using operators and expressions Develop new
plots and operators as plug-ins VisIt supports
stereo rendering

• Create derived
• variables using data
• from your database
• Pick
• Lineout
• Query

16
VisIt can handle large datasets
You can run VisIt on your desktop computer and
have VisIt process large data in parallel on a
remote supercomputer Client/Server
architecture Uses fast local graphics
hardware MPI parallel compute engine
Scalable rendering in parallel for largest
datasets
17
VisIt architecture
4 main components Graphical User Interface
(GUI) Viewer Database server Compute
engine GUI and Viewer usually meant to run
locally on your desktop computer Database
server and parallel compute engine can run
on remote computers where the data files are
located and talk to the GUI and viewer running on
your desktop computer
18
VisIt has multiple interfaces
Use VisIt as an application or a library
C, Python, Java interfaces allow other
applications to control VisIt Interfaces
Graphical user interface Python programming
interface Java programming interface C
programming interface All interfaces send
commands to the viewer and in turn get the latest
state from the viewer Use GUI when interaction
is required Use Python interface to script
actions or use VisIt as a batch mode movie
generation tool
19
A brief tour
20
What is a plot?
A plot is a viewable object, created from a
database, that can be displayed in a
visualization window Type of plots
Pseudocolor Mesh FilledBoundary Boundary
Contour Volume Vector Surface Subset
Streamline Curve Histogram Tensor Plots
come from plug-ins so you can extend VisIts
plotting capabilities by writing a new plug-in
21
What is an operator?
An operator is a filter that is applied to a
database variable before the compute engine uses
that variable to generate a plot Operators
come from plug-ins so you can extend VisIts data
manipulation capabilities by writing a new plug-in
22
Reflect operator
The Reflect operator reflects database geometry
across one or more axes Use this operator when
your simulation data contains only part of the
geometry and relies on symmetry to recover the
rest of the geometry
23
Threshold operator
Removes cells whose value is not in the
specified range Use this operator when you only
want to look at cells that have values within an
interesting range
24
Inverse ghost zone operator
This operator makes ghost zones visible and
real zones invisible Use this operator when you
want to look at your databases ghost zones
25
Three slice operator
This operator slices plots using three
axisaligned slice planes Use this operator when
you want to see into the interior of 3D plots
26
Quantitative Analysis Expressions
VisIt allows you to create new derived
variables from values in your database using
VisIts expression language Use expressions to
derive quantities that were not stored in your
database Expressions can operate on scalars,
vectors, tensors, or on meshes VisIt provides
built-in math functions Trigonmetric
functions Logarithmic functions Vector
functions
27
What is a valid expression?
VisIts expression syntax roughly resembles the
C language, although there are a few differences
28
Built-in expressions
The table to the right lists some of VisIts
built-in expression functions A more complete
list can be found in the VisIt Users manual or
in the help provided in VisIts Help window
29
Creating expressions
Databases can define expressions You can
create expressions using the Expression window
Click New button to create new blank expression
Fill in name, definition, variable type Click
Apply button
30
Remote Visualization
31
An Overview of Remote Visualization
Simulations are almost always run on a powerful
supercomputer Databases usually reside on the
computer that generated the data Run VisIt on
your local workstation but do the data I/O and
processing on the computer that generated the
data The GUI and viewer run locally while the
database server and parallel compute engine run
on the remote supercomputer Moving data is
not necessary No performance degradation due
to encryption of X11
32
Distributed mode
To run in Distributed mode, begin by opening
the File Selection Window and type the name of
the computer where the files are stored into the
Host text field You can also select the name
of a known host from the Hosts list Once the
database server is launched on the remote
computer, the files for the remote computer are
listed in the window. Add the files to be
visualized to the Selected files list and dismiss
the window Create plots as usual using a
remote file from the selected files list
33
Passwords
Sometimes when you try to access files on a
remote computer, VisIt prompts you for a
password, opening a Password Window When
prompted, enter your password and click the OK
button or press the Enter key to proceed
34
Useful trick SSH Tunneling

• Standard login to NYBlue or visualization
  cluster
• On your desktop ssh ssh.bluegene.bnl.gov
  (enter password from the cryptocard)
• On SSH gateway ssh fen.bluegene.bnl.gov (enter
  passphrase for the gateway)
• How to login to NYBlue multiple times without
  entering any passwords / passphrases and without
  braking security rules?

35
SSH Tunneling I
Create config file in .ssh directory on your
desktop Host sshbgl? HostName ssh.bluegene.bnl.lo
cal? User rosamu? LocalForward 30022
vis1.bluegene.bnl.gov22?? Host
vis1.bluegene.bnl.gov? HostName localhost? User
rosamu? Port 30022? HostKeyAlias localhostvis
36
SSH Tunneling II
Then on the desktop ssh -N -f -q -X sshbgl enter
password from the cryptocard ssh-agent
SHELL ssh-add enter the local machine
passphrase ssh -X vis1.bluegene.bnl.gov Lets
you in without password or passphrase If your
dont have a passphrase for the local
desktop, run ssh-keygen -t dsa and copy your
identity_dsa.pub into authorized_keyes on the
NYBlue
37
Connection progress dialog
The Connection progress window is opened when a
parallel compute engine cannot be launched after
a few seconds The dialog is visible until a
remote compute engine connects back to the viewer
or the connection is cancelled Clicking the
cancel button after VisIt submits a job to the
batch system does not remove the job On heavily
saturated batch systems, it might be prudent for
you to manually remove your compute engine job
from the batch queue
38
Host Profiles
When VisIt launches a compute engine, it looks
for something called a host profile Contains
information VisIt uses to launch a compute engine
on a remote computer remote user name number
of processors parallel launch method (many
other options) You may also have multiple host
profiles for any given computer Common to have
separate host profiles for running VisIt in
serial and parallel
39
Host Profile Window
The Host Profile Window is divided vertically
in two areas Top area contains tabs for host
profiles for a certain computers and controls to
create or delete host profiles The bottom area
of the window displays all attributes of the
selected host file
40
Host Profile Window, Continued
41
Setting Parallel Options
In order to make launching compute engines
easier, VisIt hides the details of the parallel
launch program, which may vary from computer to
computer VisIt allows you to set some common
parallel options and it figures out how to launch
the compute engine on the specified computer
42
Parallel options you can adjust
You may alter various Parallel Option settings
prior to running your job - partition/pool -
number of processors used - number of nodes -
parallel time limit given to run your program
43
Engine Option Window
This window provides a list of host profiles
from which to choose Once a host profile is
chosen, parallel options can be altered to fine
tune how the compute engine is launched Click
the OK button to confirm your choices or changes
44
Compute Engine Window
This windows purpose is to display the
progress of a compute engine as it completes a
task, via status bars The window displays
information about the compute engine such as how
many processors and nodes it uses Interrupt a
compute engine Close a compute engineCompute
Engine Window
45
Thank You