ETL and Data Integration (Extraction Transforming Loading)

1
ETL and Data Integration(Extraction Transforming
Loading)

• Wolfgang Schoepp, Thanh Binh Nguyen
• International Institute for Applied Systems
  Analysis

2
Main Goals

• integrate large? volumes data from multiple data
  sources. Two heterogeneity problems
• Different parameters
• Different meaning of information
• framework for integrating data across disparate
  platforms
• extracts and transforms data from multiple
  systems and loads it into data repository (ETL)
• Metadata and Data Content Management
  (Terminology, mappings, etc)

3
Overview GAINS architecture
4
Questions to be addressed

• Data Transformation at House keeper (IIASA) ?
• provide data in term of common and/or native
  format ?
• Data Push or pull ?
• Simple file upload or Web Service?
• Data repository with analyzing tool ?
• Checking rules ?
• Interactive data browser ?
• Open Source?

5
Technical Issues

• Oracle (Postgres) Database
• Application - Java Servlet
• Multiplatform (MSWIN, LINUX, UNIX, MAC-OS)
• Configure heterogeneous data with metadata
  registration
• Open source solution
• Jasper ETL
• Java Framework for user management (GAINS)
• Content Management System (CMS/GAINS)?
• XML - standards

6
ETL Process
Option 3 Data provider E-T House keeper L
Option 1 Data provider E House keeper T-L
Option 2 Data provider E-T House keeper L