Realistic modelling of complex problems on Grids

1
Realistic modelling of complex problems on Grids

• John Brooke (University of Manchester)
• Peter Coveney PI RealityGrid (University
  College London)
• Stephen Pickles (University of Manchester)
• Thanks also to the other RealityGrid
  co-Investigators
• John Darlington (Imperial College)
• Steve Kenny and Roy Kalawsky (Loughborough
  University)
• John Gurd (University of Manchester)
• Mike Cates (University of Edinburgh)
• Adrian Sutton (University of Oxford)

2
The RealityGrid project

• Mission Using Grid technology to closely couple
  high performance computing, high throughput
  experiment and visualization, RealityGrid will
  move the bottleneck out of the hardware and back
  into the human mind.
• Scientific aims
• to predict the realistic behavior of matter using
  diverse simulation methods (Lattice Boltzmann,
  Molecular Dynamics and Monte Carlo) spanning many
  time and length scales
• to discover new materials through integrated
  experiments.

3
Partners

• Academic
• University College London
• Queen Mary, University of London
• Imperial College
• University of Manchester
• University of Edinburgh
• University of Oxford
• University of Loughborough

• Industrial
• Schlumberger
• Edward Jenner Institute for Vaccine Research
• Silicon Graphics Inc
• Computation for Science Consortium
• Advanced Visual Systems
• Fujitsu

4
RealityGrid
Instruments XMT devices, LUSI,
HPC resources Scalable MD, MC, mesoscale modelling
User with laptop/PDA (web based portal)
Steering ReG steering API
Performance control/monitoring
VR and/or AG nodes
Visualization engines
Moving the bottleneck out of the hardware and
into the human mind
5
RealityGrid Characteristics

• Grid-enabled (Globus, UNICORE)
• Component-based, service-oriented
• Steering is central
• Computational steering
• On-line visualisation of large, complex datasets
• Feedback-based performance control
• Remote control of novel, grid-enabled,
  instruments (LUSI)
• Advanced Human-Computer Interfaces (Loughborough)
• Everything is (or should be) distributed and
  collaborative
• High performance computing, visualization and
  networks
• All in a materials science domain
• multiple length scales, many "legacy" codes
  (Fortran90, C, C, mostly parallel)

6
Three dimensional Lattice-Boltzmann simulations

• Code (LB3D) written in Fortran90 and parallelized
  using MPI.
• Scales linearly on all available resources.
• Fully steerable.
• Future plans include move to parallel data format
  PHDF5.
• Data produced during a single large scale
  simulation can exceed hundreds of gigabytes or
  even terabytes.
• Simulations require supercomputers
• High end visualization hardware and parallel
  rendering software (e.g. VTK) needed for data
  analysis.

3D datasets showing snapshots from a simulation
of spinodal decomposition A binary mixture of
water and oil phase separates. Blue areas
denote high water densities and red visualizes
the interface between both fluids.
7
Exploring parameter spacethrough computational
steering
Cubic micellar phase, high surfactant density
gradient.
Cubic micellar phase, low surfactant density
gradient.
Initial condition Random water/ surfactant
mixture.
Self-assembly starts.
Lamellar phase surfactant bilayers between water
layers.
Rewind and restart from checkpoint.
8
Computational Steering - Why?

• Terascale simulations can generate in days data
  that takes months to understand
• Problem to efficiently explore and understand
  the parameter spaces of materials science
  simulations
• Computational steering aims to short circuit post
  facto analysis
• Brute force parameter sweeps create a huge
  data-mining problem
• Instead, we use computational steering to
  navigate to interesting regions of parameter
  space
• Simultaneous on-line visualization develops and
  engages scientist's intuition
• thus avoiding wasted cycles exploring barren
  regions, or even doing the wrong calculation

9
Computational steering how?

• We instrument (add "knobs" and "dials" to)
  simulation codes through a steering library
• Library provides
• Pause/resume
• Checkpoint and windback
• Set values of steerable parameters
• Report values of monitored (read-only) parameters
• Emit "samples" to remote systems for e.g. on-line
  visualization
• Consume "samples" from remote systems for e.g.
  resetting boundary conditions
• Images can be displayed at sites remote from
  visualization system, using e.g. SGI OpenGL
  VizServer, or Chromium
• Implemented in 5 independent parallel simulation
  codes, F90, C, C

10
Philosophy

• Provide right level of steering functionality to
  application developer
• Instrumentation of existing code for steering
• should be easy
• should not bifurcate development tree
• Hide details of implementation and supporting
  infrastructure
• eg. application should not be aware of whether
  communication with visualisation system is
  through filesystem, sockets or something else
• permits multiple implementations
• application source code is proof against
  evolution of implementation and infrastructure

11
Steering and Visualization
Simulation
Client
data transfer
Visualization
Visualization
12
Architecture

• Communication modes
• Shared file system
• Files moved by UNICORE daemon
• GLOBUS-IO
• SOAP over http/https

Simulation
Client
data transfer
Data mostly flows from simulation to
visualization. Reverse direction is being
exploited to integrate NAMDVMD into RealityGrid
framework.
Visualization
Visualization
13
Steering in the OGSA
Simulation
bind
Steering library
publish
Client
connect
data transfer
Steering client
Registry
find
publish
bind
Visualization
Visualization
14
Steering in OGSA continued

• Each application has an associated OGSI-compliant
  Steering Grid Service (SGS)
• SGS provides public interface to application
• Use standard grid service technology to do
  steering
• Easy to publish our protocol
• Good for interoperability with other steering
  clients/portals
• Future-proofed next step to move away from
  file-based steering or Modular Visualisation
  Environments with steering capabilities
• SGSs used to bootstrap direct inter-component
  connections for large data transfers
• Early working prototype of OGSA Steering Grid
  Service exists
• Based on light-weight Perl hosting environment
  OGSILite
• Lets us use OGSI on a GT2 Grid such as UK
  e-Science Grid today

15
Steering client

• Built using C and Qt library currently have
  execs. for Linux and IRIX
• Attaches to any steerable RealityGrid application
• Discovers what commands are supported
• Discovers steerable monitored parameters
• Constructs appropriate widgets on the fly

• Web client (portal) under development

16
RealityGrid-L2 LB3D on the L2G
Visualization SGI Onyx Vtk VizServer
SGI OpenGL VizServer
Laptop Vizserver Client Steering GUI GLOBUS used
to launch jobs
Simulation Data GLOBUS-IO
Steering (XML) File based communication via
shared filesystem Steering GUI X output is
tunnelled back using ssh.
program lbe use lbe_init_module
use lbe_steer_module use
lbe_invasion_module
Simulation LB3D with RealityGrid Steering API
ReG steering GUI
17
Performance Control
18
Advance Reservation and Co-allocationSummary of
Requirements

• Computational steering remote, on-line
  visualization demand
• co-allocation of HPC (processors) and
  visualization (graphics pipes and processors)
  resources
• at times to suit the humans in the loop
• advanced reservation
• For medium to large datasets, Network QoS is
  important
• between simulation and visualization,
• visualisation and display
• Integration with Access Grid
• want to book rooms and operators too
• Cannot assume that all resources are owned by
  same VO
• Want programmable interfaces that we can rely on
• must be ubiquitous, standard, and robust
• Reservations (agreements) should be re-negotiable
• Hard to change attitudes of sysadmins and (some)
  vendors

19
Steering and workflows

• Steering adds extra channels of information and
  control to Grid services.
• Steering and steered components must be
  state-aware, underlying mechanisms in OS and
  lower-level schedulers, monitors, brokers must be
  continually updated with changing state.
• How do we store and restore the metadata for the
  state of the parameter space search?
• Human factors are built into our architecture,
  humans continually interact with orchestrated
  services. What implications for workflow
  languages?

20
Collaborative aspects

• Multiple groups exploring multiple regions of
  parameter space.
• How to record and restore the state of the
  collaboration?
• How to extend the collaboration over multiple
  sessions?
• What are the services and abstractions necessary
  to bootstrap collaborative sessions?
• How do we reliably recreate the resources
  required by the services, in terms of
  computation, visualization, instrumentation and
  networking.

21
Integration with Access Grid?
Participants location and access rights
Virtual Venues Server Multicast
addressing Bridges
Visualization Workflow Workflows saved from
Previous sessions or Created in this session
Who participates?
Service for Bootstrapping session Contains just
enough Information to start other Services, red
arrows indicate bootstrapping
Process Repository Collaborative
processes Captured using ontology Can be enacted
by Workflow engines
Simulation Workflow Workflows saved from Previous
sessions or Created in this session
What do they use?
Application Repository Uses application specific
ontology to describe what in silico processes
need To be utilised for the session
Data Source Workflow Workflows saved from
Previous sessions or Created in this session
Application data, computation and visualization
requirements
22
How far have we got?

• Linking US Extended Terascale Facilities and UK
  HPC resources via a Trans-Atlantic Grid
• We used these combined resources as the basis
  for an exciting project
• to perform scientific research on a hitherto
  unprecedented scale
• Computational steering, spawning, migrating of
  massive simulations for study of defect dynamics
  in gyroid cubic mesophases
• Visualisation output was streamed to distributed
  collaborating sites via the Access Grid
• Workshop presentation with FZ Juelich and HLRS,
  Stuttgart on the theme of computational steering.
• At Supercomputing, Phoenix, USA, November 2003
  TRICEPS entry won Most Innovative Data-Intensive
  Application

23
Summary

• All our workflow concepts are built around the
  idea of Steerable Grid Services.
• Resources used by services have complex state,
  may migrate, may be reshaped.
• Collaborative aspects of Humans in the loops
  are becoming more and more important.
• The problems of allocating and managing the
  resources necessary for realistic modelling are
  very hard, they require (at present) getting
  below the Grid abstractions.
• Clearly the Grid abstractions are not yet
  sufficiently comprehensive and in particular lack
  support for expression of synchronicity.

24
London University Search Instrument

• LUSI is located at and developed by Queen Mary
  College, University of London
• Aim Find ceramics (e.g. rare earth metal oxides)
  with interesting / valuable properties (e.g. high
  temperature superconductivity)
• Motivation theory cannot indicate how to
  construct a compound with a particular property.
  Established methodology in pharmaceutical
  industry uses automated sample generation and
  testing. Let's apply the same idea in materials
  science, exploring properties that are difficult
  to predict superconductivity, luminescence,
  dielectric response

Furnace
XY Table
Instruments
Printer
25
LUSI - schematic
c
c
Predictions
Neural network
Measured data