USGS ShakeCast

1

• USGS ShakeCast
• Delivering Earthquake Shaking DataTo The People
  Who Need It

Philip A. NaeckerChief ScientistGatekeeper
Systems pan_at_gatekeeper.com
2
ShakeCast Features and Architecture

• Project Origins
• Project Team
• Issues with Current Delivery of ShakeMaps
• Project Goals and ShakeCast System Goals
• ShakeCast Terminology

• ShakeCast Features and Functions
• Prototype
• V1.0
• Futures
• ShakeCast Architecture
• Protocols and Transport
• Database
• Development Environment

3
Project Origins

• Success of ShakeMap Project
• ShakeMaps are data rich
• Recurrent difficulties utilizing ShakeMaps
• Consumers have had difficulty with network
  configuration, firewalls, and data processing
• Each consumer wants a different configuration of
  feed, requiring hand holding and creating a
  maintenance morass
• Missed opportunities for data utilization
• Significant risk that information system
  fragility might impact successful application of
  ShakeMaps in a major event

4
ShakeCast Project Team

• What eventually became the ShakeCast Team has
  been in conversation about these issues since
  1998
• USGS Dave Wald and Bruce Worden
• Peter German
• Consulting geologist, seismic data processing
  expert
• Creator of the CUBE system
• Robert Nigbor
• USC Professor
• Long involvement in practical applications of
  shaking data (e.g., Intel)

5
Software Development Expertise

• Gatekeeper Systems
• In business since 1993 selling software for
  utilities
• Specialize in high performance, high availability
  systems for large database, GIS, and Internet
  applications
• Customers include large utilities and
  municipalities
• City of LA, Santa Barbara, PacifiCorp, Las Vegas
  Valley Water District, Georgia Power, Contra
  Costa County, etc.

6
ShakeCast Software Development Team

• Bruce Worden ShakeMap Software Development
• Phil Naecker
• Civil Engineer, 10 years as engineering
  consultant focused on water and wastewater
  utilities
• Built large high performance data management
  systems for business and engineering applications
  since 1971
• Steve Caine
• 45 years experience building software
• Founded firm that sold critical software systems
  to DOD and related agencies for over thirty years
• Internet and Open Systems Expert
• Dave Burke
• 25 years experience building software
• Exceptional competence at building complex systems

7
ShakeCast Problem and Solution

• Current Problems
• ShakeCast Solution

8
Current Delivery of ShakeMaps

• Custom solution for each organization
• Unknown failure modes
• Not built by experts
• Not built using latest Internet technologies
• Difficult to implement and maintain
• Significant work for USGS and consumer
• Not conducive to adding new data products or
  technologies
• Each organization has home-grown notification
  system (or none at all)

9
EQ Systems 2002
Seismic Processing
Akamai Servers
Archives
Public/PrivateNetworks
Public Networks
Instrument Network
Users
PrivateNetworks
ShakeMap Processing
FTP over Public Networks
Private Networks
Consumer InformationSystem
10
Goals ShakeCast Project

• Provide rapid and reliable delivery of
  information about earthquake shaking to the
  people who need it
• Initial audience focuses on utilities, large
  facility owners, municipalities, and news outlets
• Shaking Information is broadly defined
• Satisfy some of the needs for post-event
  activities such as refinement of response plans
  and after-the-fact response assessment by
  accurately recording who knew what when
• Deliver an open technology platform for
  earthquake information delivery that facility
  managers and the USGS can depend on and feel
  comfortable in building upon

11
EQ Systems 2003
Seismic Processing
Akamai Servers
Archives
Public/PrivateNetworks
Public Networks
Instrument Network
Event Notification System
Users
PrivateNetworks
ShakeMap Processing
Processing and Notification
Distribution System
Private Networks
ShakeCast
Consumer InformationSystem
12
Goals Data Delivery and Notification

• Delivery and Notification Must Be Extremely
  Reliable
• Multiple data sources
• Multiple data paths
• Robust testing protocols
• Set and Forget Design
• Fast Under All Load Conditions
• Essentially no training required for proper
  installation and maintenance

13
Goals Other System Attributes

• Firewall Friendly
• System Must Be Complete
• Functional Out of the Box in a few hours
• Deliver data and most commonly-used tools
• Extensible and Flexible
• Able to integrate with more advanced tools
• Customizable by organizations and individual
  users
• Traceable and Auditable
• System and usage audit logs, consistent
  timestamps
• Objective and reproducible notification actions

14
ShakeCast Features and Functions

• Phased rollout of features
• Functions provide infrastructure, not enforce
  policy on how ShakeMaps are used
• Reliability and robustness are designed in from
  the beginning

15
ShakeCast Technology Plan

• Open source using widely used tools
• Apache, Perl, Oracle, Windows, Internet Explorer
• Heavily documented code and databases
• Based on commodity technologies
• Internet and Web
• Email and SMS pagers
• Relational databases
• Security and reliability designed in from the
  start

16
ShakeCast Project Phases

• Phase 1 - Prototype
• Phase 2 Reference System
• Phase 2 Open Source

17
ShakeCast Phase 1 - Prototype

• Funded by USGS
• Purposes are proof of concept, demonstration
• Implemented by Gatekeeper Systems and Bruce
  Worden
• Due Summer 2003
• CalTrans will be first testbed organization
• SBC/PacBell anxious to be next testbed
• Basic functionality, no easy installation or
  customization, single hardware/software platform

18
ShakeCast Phase 2 Reference System

• Shopping for funding now
• Fully installable (goal is one hour if the server
  already has a web server and database)
• Easily configurable
• Multiple hardware/software platforms
• Looking for early adopters
• Expect network of dozens to hundreds of systems
  and tens of thousands of end users

19
ShakeCast Phase 3 Open Source

• Shared development between user organizations
• Enhancements to core system
• Participation by universities
• Commercialization and Extension
• Integration with internal systems
• Improved data for shaking estimates
• Readily available sources for fragility estimates
• Hoping for thousands of systems and millions of
  end users

20
ShakeCast Terminology

• Server every ShakeCast machine is a server
• Upstream/Downstream direction of data flow,
  although complex network topologies are allowed
• Parameters measures of shaking generated by
  ShakeMap system
• Products data files, in various formats and
  multiple parameters, moved between ShakeCast
  machines
• Grid File ShakeMap grids containing the raw
  parameter data
• Notification Detailed electronic message about a
  specific event, system activity, or shaking level
  to a specific user or group of users

21
ShakeCast Software Features

• Reliably and automatically receive and process
  shaking data from ShakeMap
• Let organizations and users define locations of
  interest (facilities) and set shaking thresholds
  (green, yellow, red) in multiple shaking metrics
  (acceleration, instrumental intensity, etc.)
• Reliably deliver to end users electronic
  notification of facility damage estimates in a
  prioritized, customized, easy-to-use form
• Make maps and reports from local servers
  available via the Web
• Easily integrate with consumers other IT systems
• Provide for end-to-end testing and upgrades

22
Prototype Features

• Receive, store and forward ShakeMaps and
  associated metadata in a reliable manner
• Unpack ShakeMap grids into a relational structure
• Notify users of shaking and ShakeCast activity
• Email and pager
• System administrator, ShakeCast developer, and
  end user events
• Produce detailed log files of ShakeCast and user
  activity

23
Prototype Features - Transport

• Receive from multiple upstream ShakeCast and
  ShakeMap servers
• Versioned products
• HTTP push or HTTP polling/pull
• Can transport not only shaking data but system
  metadata i.e. product types, message types, etc.
• Basic filtering for ShakeMap feed
• Bounding rectangle
• Peak grid values
• Retry and basic error handling
• Basic test suites

24
Prototype Features Notification

• On events map generated, errors, recall/cancel,
  product delivery
• On ShakeMap parameters magnitude, acceleration,
  etc.
• On location-specific shaking for any ShakeMap
  parameter
• On exceedence of facility fragility for any
  ShakeMap parameter (green, yellow, red)
• Message format driven by easily customized
  templates, includes direct Web links

25
Prototype Features - Database

• Most configuration information, all ShakeMap
  data, and all user data is stored in the database
• Support for both Access and Oracle
• Site administrator can access database using
  standard SQL and other standard tools such as MS
  Access, Visual Basic, Perl, etc.

26
Additional V1.0 Features

• Consolidated notification
• Professional documentation
• Complete installation procedure with upstream
  registration
• User configuration web pages
• Full support for both Unix and NT

• Deliver ShakeMap web pages locally
• Enhanced error handling
• Enhanced test procedures
• Call out from ShakeCast to private scripts to
  invoke site-specific functions
• Automated end-to-end testing

27
Features Futures

• Richer notification options
• Support for multiple related events
• More intelligent prioritization of messages
• More complicated notification logic
• Support for positive response (confirmation)
• Richer web links with active web pages to help
  users manage large lists of facilities
• More database platforms certified
• Open source shared development environment
• Upstream reporting of ShakeCast usage, server
  health and status, and test results

28
ShakeCast Implementation

• Implementation Principles
• Use open and familiar tools and protocols
• Follow path of least resistance for network and
  security managers
• Open, readable, understandable source code
• HTTP and HTTPS transport
• Relational database for all data storage
• NT Service/Unix Daemon for event loop processing

29
Software Development Environment

• NT and Unix (but could be anything)
• Perl for CGI (but could be anything)
• Standard SQL (and optional SQL-based tools)
• Source code management in CVS (Web-based source
  management system designed for shared
  development)
• No C, VB, Java or compiled languages required,
  providing short development cycles, transparency,
  portability

30
ShakeCast System Data Flow
ShakeCast Server
ShakeCast Server
ShakeCast Server
http
http
httpftp/NFS
Web Pages
ShakeCastData and XML
smtp
NotificationMessages
ShakeMap Server
http
http
EmailSystem
Data and Invocation
Users
ShakeCast Server
GIS Systems Control Systems Alarm Systems
Paging System
Email/ Pager Messages
http
ShakeMap Server
Private Systems
USGS Systems
31
Protocols and Transport Architecture

• HTTP CGI scripts for response to ShakeCast
  requests
• ShakeCast metadata encoded in simple XML
• HTTP GETs for file delivery
• Can easily use HTTPS if needed
• Authenticated server-server exchange
• MD5 secured passwords for authentication
• Separate passwords for each server pair
• New CGI routines can be easily added
• Use Apache HTTP server, but could use others

32
Database Architecture

• Fully normalized data model
• High cardinality transaction data can be purged
  if necessary
• Shared data elements have shared primary keys
• Locally-generated data elements have
  locally-generated primary keys
• Multiple database platforms possible
• MS Access and Oracle currently supported
• Can easily be extended to use Oracle Spatial or
  other high-end database features

33
Questions and Answers
?
?
?
?