A Distributed Data Architecture

1
A Distributed Data Architecture

• Mark Jessop
• University of York

2
Swans
3
Grid Enabled Swans
London
Tokyo
Cape Town
Mexico City
4
How Big is that Lake?

• Heathrow capped at 36 landings per hour.
• If half have 4 engines and half have 2, average
  aircraft carries 3 engines.
• Each engine generates around 1GB of data per
  flight.
• 36 x 3 x 1 108GB raw engine data per hour.
• Factor in the working day and the rest of the
  world
• Terabytes and up!

5
Managing the Flow of Water
6
Plumbing Toolkit

• Data Repository
• Catalogue
• Pattern Match Engine

7
Pattern Match Engine

• Pattern Match Control
• Data Extractor/Encoder
• AURA Encoder
• AURA-G
• Back Check

8
Data Repository

• SDSC Storage Request Broker.
• Manages distributed storage resources.
• Meta Data Catalogue.
• Many configurations.
• Heterogeneous.
• Efficient data delivery.
• C and Java APIs.

9
A Distributed Architecture

• One node per airport.
• Single global MCAT.
• Stream engine data.
• Global Parallel Search.
• Present Results.
• Scalable.
• Robust.

10
Summary

• Large quantities of data arriving globally.
• Distributed architecture for data management and
  search.
• Scalable and Robust.