Gearing Up for Emergency Management Integration of Large Datasets

1
Gearing Up for Emergency Management--------Inte
gration of Large Datasets

• Kenneth Galluppi
• Renaissance Computing Institute
• galluppi_at_renci.org
• Mats Rynge, Lisa Stillwell, Erik Scott, RENCI
• Hugh Devine, Helena Mitasova, MCSU
• Larry Band, UNC-CH
• Jon Goodall, Duke

2
This talk is for you if..

• If you ever
• Hit barriers with
• processing taking
• too long..
• Hit barriers with not having enough disk
  space..
• Hit barriers due to machine limits..
• Hit barriers not having the data readily
  available..

3
But.

• I dont have a solution

4
But.

• But we are working on it

5
Outline of Talk

• Who is RENCI, What are we doing?
• Disaster Requirements for GIS
• GIS and High Performance Computing
• Looking for Collaborators

6
Who is RENCIRenaissance Computing Institute
http//renci.org

• Statewide resource as a catalyst for innovation
  and collaboration
• Exploratory information technologies
• University expertise
• Catalyst for change
• Areas of focus
• Disaster
• Bioinformatics

7
RENCI Resources

• 6 locations Anchor, UNC-CH, NCSU, Duke, ECU,
  UNCA (plus 2 more this year)

8
RENCI Resources

• Teams of People
• Computer Science
• Visualization
• IT Technology
• Resources
• 4096 processor, IBM Blue Gene/L
• 64 Node Dell Clusters
• 10s of TB spinning disk storage (1 PB)
• Visualization laboratories
• 10 and 100 gigE test networks

9
Workshop Visualization on the Grid

• Collaborative
• Multi-disciplinary

• 4 x 4 projectors
• 5120 x 2880 resolution
• 14 ft x 8 ft

10
Why Disaster??
11
120 Million Expenditure/Year
12
PREPARATION
13
Disaster Management

• Most NC disasters are weather-driven
• Decisions are local
• Management runs in a cycle and never ends
• Disaster is multi-faceted

14
Disaster GIS Requirements

• Timely information
• Complex information relationships
• Multi-disciplinary decisions
• Diverse dissemination

15
Information System

• Data Ingest, Adapt to NC data
• Information Simulation/Analysis
• Hindcast
• Nowcast
• Forecast
• Knowledge Decision Support Command Control
• Dissemination
• Government
• Public

16
HPC Need Integrated Tools

• Social Systems
• Economics
• Public Health
• Infrastructure
• Transportation
• Power
• Communications
• Physical World
• Meteorology
• Hydrology
• Geo-base data

17
HPC Need Scales of Relations

• Spatial range 15km 30 meters or less
• Computations from PC to 1000s of nodes
• Data 100 MB to low Terabytes (10,000x)
• Turn around, hours to minutes

18
HPC Need - Dissemination
19
HydroMet System

• Core system for forecasting floods and other
  disasters
• Flood (flash, inundation, landslide)
• Storm surge

20
Flood Forecasting Goal

• Regional forecasts
• Local forecast
• Only for certain locations
• Anywhere
• Static Forecast
• Dynamic real-time

21
Exploring HPC for GIS

• Data Management
• Analysis Engine
• Dissemination

22
Data Management(changing environment)

• 100MB 10s TB (500/TB)
• Distributed and linked data
• Real-time update
• Compute processing is changing to multiple
  processors
• Networks moving Gbs per second

23
Data Management

• Example
• NC DEM LIDAR (5TB)
• NC orthophotos (20TB)
• What to do with it?
• 1000s of tiles???
• What is its native structure (no limits)
• How do you update
• Define uses and integrate to other data
• GIS sets, video streams, sensors

24
Data Management(current work, exploratory)

• Archive v. persistent (access, retrieval and data
  movement)
• Create persistent data for rapid use (10s TB)
• Ad-hoc additions and deletion (100 GBs)
• Need to integrate with other data
• Modeled results (e.g. weather forecast)
• Infrastructure, e.g. transportation
• Use machine memory (100-1000 GB)
• Use multiple processors and parallel I/O systems

25
Analysis Engine(changing environment)

• Domain spaces are expanding
• Spatial
• Statistical and time series
• Simulation
• Processing is distributed and linked, grid
  computing
• 1000s of processors with large memory (16
  GB/processor)
• Special processors for I/O, like graphics and
  floating point

26
Analysis Engine(current work, exploratory)

• Multiple processors and memory use
• Rapid processing (2 tiles/second)
• Distributed, but linked (grid), open GIS modules
  (e.g. GRASS)
• Key modules on parallel systems
• Minimize back and forth, in and out of a GIS,
  boundary blurring
• Module wrapping (services) and integration
• Workflow engines

27
Dissemination(changing environment)

• Graphic engines are multiple processors and large
  memory
• Range of graphic and visualization packages, lite
  to heavy duty imaging
• Displays of wide range
• Walls 16,000 x 16,000 pixels???
• Multiple screens
• PDAs and cell phones
• Internet (10s Gb/sec)
• Image viewing in G-pixelss

28
Dissemination(current work)

• Collaborative technologies
• More than a picture (people)
• Desktop conferencing
• Domes, display tables, stereoscopic devices,
  motion capture, GPS, sensors
• Knowledge Management
• Domain expertise level
• Capture
• Location, location, location
• Where is the user, not the GIS expert

29
(No Transcript)
30
(No Transcript)
31
Mobile Units
32
NC Flood Forecasting
Courtesy of Jon Goodall, Duke
33
Remove the Limits

• What do you NEED to do?
• High-Performance Computing and Communications
• Speed
• Complexity
• Large, repetitive
• Storage, networks, compute power are all evolving.

34
Keep It Real
35
Summary

• RENCI is a resource to NC collaborators
• Disaster in a broad sense, is our focus
• Geo-referenced, complex information underlays
  disaster studies
• HPCC offers different approaches to tough
  problems
• HPCC are tomorrows resources

36
Collaborators

• We are looking for problems and collaborators
  that can address fundamental issues
• Issues of importance to North Carolina and beyond
• Tap NC resources

37
Thank You

• Ken Galluppi
• galluppi_at_renci.org
• 919-445-9649