Design and Implementation of EarthScope Data Portal

1
Design and Implementation of EarthScope Data
Portal
Chaitan Baru, Kai Lin San Diego Supercomputer
Center
2
The Goal of EarthScope Data Portal

• Seismic data located in IRIS
• GPS data located in UNAVCO
• Drilling data located in SAFOD
• Goal
• Develop a unified web interface for searching and
  downloading EarthScope Data with the common
  denominators
• Data Classes
• Seismic, GPS, Strain, Drilling logs, Tilt,
  Physical sample, Environmental, MT
• Spatial Coverage
• Temporal Coverage
• Networks and stations

3
Architecture of EarthScope Data Portal
EarthScope Data Portal
database
Authentication
Logging
Caching
Web Service
Web Service
Web Service
IRIS
UNAVCO
SAFOD
4
Web Services for EarthScope Data

• getStations
• Get all stations for specified data classes, time
  range, bounding box and networks.
• listData
• Get all datasets for specified data classes, time
  range, bounding box, networks and stations.
• packageData
• Pack selected datasets into a downloadable
  package and return a package ID.
• checkPackage
• Check the status of a package and return the
  progress or a URL for downloading

5
Map Interface

• Use Google Maps
• Challenge more than 5000 stations
• Our Solutions
• Non-cluster mode for high-end computers
• Cluster mode for low-end computers
• The browser sends the current spatial extent and
  the map size to the server
• The server put adjacent points into a cluster and
  sends back to the browser to show on the map
• Possible Alternative Solutions
• Other web based map servers
• Google Earth Plug-in
• Google Maps WMS

6
Search Interface

• Web Forms for Common Denominators
• Data Classes
• Spatial Coverage
• Temporal Coverage
• Networks and Stations
• Challenge memorize the last search parameters
• Solution
• Search parameters are saved as cookies at the
  browser side

7
Data Search Response

• A Data Class (e.g. Seismic, GPS) is a high level
  category of data.
• In many cases, refined subclasses are needed
  (e.g. Raw GPS data, GPS data product based on an
  algorithm).
• Up to each data center to organize data search
  result into a hierarchy that is best to present
  the datasets.
• The portal combines possible three hierarchies
  from each data center to a single hierarchy for
  end users.

8
Data Cart

• Similar to Shopping Cart
• Persistent, i.e., cross sessions
• Users can put selected datasets into the Data
  Cart.
• A typical use case of the Data Cart
• Search data
• Save some datasets into the Data Cart
• Repeat the steps above
• Checkout the datasets in the Data Cart

9
User Tracking

• Use username to track the user if the user is
  logged in
• Use cookie to track the user if the user is not
  logged in
• For each guest user, save a unique user-id as a
  cookie to identify the user
• Every dataset the user selected in the Data Cart
  is saved in the portal database associated with
  the user-id
• A guest user can view his/her Data Cart and
  requested data packages same as login users if
  he/she uses the same browser
• The logging shows that most users prefer guest
  user mode

10
Packaging

• User can request to download all datasets in Data
  Cart as a package.
• The size limit of a package is 2GB.
• The portal sends a request to each data center to
  pack its own data as a zipped file.
• Each data center sends back a package ID for
  tracking the status of the package.
• Each data center needs time to finish packing
  data.
• The portal periodically queries the status of
  each unfinished package until getting a download
  URL.
• The portal downloads the three packages from
  three data centers and repack them into a single
  package and let the user download.
• The final downloadable package expires in 2
  weeks.

11
Question?