The Importance of Metadata Management and Data Sharing

1
The Importance of Metadata Management and Data
Sharing

• 2008.11.27. Thursday
• Mauritius
• AfriNIC Meeting
• Main Project Investigator Sue Moon (Associate
  Professor)
• Presenter Seoyeon Kang

2
Talk Outline

• Introduction to CASFI (Collect, Analysis, and
  Share for Future Internet)
• Motivations behind CASFI
• Team members
• Our goals
• CASFI Data Sharing Platform
• System design
• User perspectives

3
Motivations for CASFI

• Measurement research benefits from
• More data
• Diverse data
• Feedback (as always!)
• Build a community of measurement research
• People and schools with data
• Past track record of research in this field

4
Our Team
5
Equipments/Infrastructures of Our Team
KAIST owns 4 1GE DAGMONs
High-End Packet Processing System at Chungnam
Natl U
NG-MON based on DAG Card at POSTECH
6
Research Topics of 2008-2009

• KAIST
• KAIST backbone traffic analysis
• Data sharing platform development
• User behavior analysis
• POSTECH/KHU
• Review manageability issues in Future Internet
• CNU
• Flow/contents identification at 1GB or higher
  speed
• Deep packet inspection (DPI)
• VoIP identification in 3G/3.5G

7
Expected Deliverables
8
Funding Agency

• Originally,
• MIC (Ministry of Information and Communications)
• MKE (Ministry of Knowledge Economy)
• IITA (Institute for Information Technology
  Advancement)
• Technology Development Track for World-Class
  Wide-Area Integrated Network Infrastructure
• IT Core Technology Development
• 5-year, 2 million, 5 profs and 10 students
• March 2008 to February 2013

9
CASFI Data Sharing Platform

• Need for Data Sharing
• It took 10 years for the community build archives
• Now Need for Data Sharing Platform

10
System Design Overview

• Design Philosophy
• Present a façade
• Make it easy for contributors/consumer/administrat
  or
• Provide processing capability to consumers
• Magnitude of Data
• Packet traces 100 Gbytes
• Routing tables 100 of 10 GBytes
• Miscellaneous formats
• Web crawl data of tags, social networks, profiles

11
Present a Façade

• Provide a consistent interface across multiple
  sites
• Allow local browsing of remote metadata

12
System Design
Web interface
Metadata DB
XML-RPC
Server
Storage
Storage
. . . .
13
System design

• Contributor
• Consumer
• Administrator

14
System design

• Contributor
• Consumer
• Administrator

15
Contributor

• Requirement
• Upload data without cumbersome work
• Should keep metadata generating process simple
• Alternative
• Produce metadata semi-automatically in XML form
• Well-known form data(pcap, erf)
• Free-form data

16
Sample scenario - Upload
Metadata DB
Web interface
Contributor
Data
Server
Generate metadata (XML)
Store data
Storage
Storage
. . . .
17
System design

• Contributor
• Consumer
• Administrator

18
Consumer

• Requirement
• Search and locate data
• Get the result of large or partially restricted
  data
• Alternative
• Manage metadata in a database
• Provide processing capability
• Packet analysis program flow size, flow
  duration, packet inter arrival time, etc.
• Primitives SUM, AVG, MIN, MAX, etc.

19
Metadata DB
Web interface
Consumer
Browse Search data
Return the result
Request processing result
Server
Locate data Process
Storage
Storage
. . . .
20
System design

• Contributor
• Consumer
• Administrator

21
Administrator

• Requirement
• Install and upgrade data sharing platform
• Mount local data and give a view of remote data
• Alternative
• Use platform independent languages, java python
• Make data sharing platform independent of
  back-end file systems

22
Sample scenario - Install
MetaDB _at_CNU
Meta DB _at_POSTECH
Web interface
Insert metadata
Server
Generate metadata (XML)
Storage
Storage
. . . .
23
CASFI DSP Web Page

• http//casfi.kaist.ac.kr/casfi-dsp

24
Current Status

• You can
• Register
• Search and view data
• Contribute data with manual input of metadata

25
Project Management Structure
Data Sharing Platform

• WIDE, CAIDA, CNRS
• International collaboration
• Publish data about domestic network traffic
• AfriNIC??

• Leadership in Korea
• Active participation in FIF, AsiaFI, CFI
• Build with research labs and industry ties
  research through workshops

KAIST
CNU
KHU
POSTECH
26
http//casfi.kaist.ac.kr

• QA