TotalETL:infoServer

1
TotalETLinfoServer

• Chris Fournier
• Nathan Clark
• Scott Longley
• Cyril Shilnikov

MQP Project 2005 Sponsored by TotalETL inc.
2
TotalETL

• Small ETL Company
• ETL (Extract Transform Load)
• Used in large companies
• Multimillion dollar business
• Existing Product is infoSight-- desktop solution

3
infoSight
4
infoSight Current Features

• GUI Project creation
• Library of Transformers
• Works with multiple input types
• Single machine
• Single user
• One project at a time

5
MQP Project goals

• Prototype the client-server version of infoSight
• Distributed
• Multi-user
• Database-centric
• Extensible
• Alpha-level code
• Focus on back-end design

6
Project Methodology

• Met with TotalETL team on-site
• Design requirements
• Refine and discuss requirements as needed
• Build core modules, demo end first term
• Build additional modules, final demo.

7
General design overview
Thin Clients
Distributed Server System
Repository
Thick Clients
8
Actual design overview
Client
Security Manager
Session Manager
Event Log Manager
Project Manager
Job Manager
Version Manager
Scheduling Manager
Repository Manager
DB
9
Repository Manager

• System core
• Store all information about
• System operation
• Security
• Projects
• XML Parser to store Projects
• JDBC to connect to DBs

10
Repository Table Design
11
Project Manager Version Control

• Storage and Retrieval of Projects
• In-memory Object -gt XML File -gt Repository
• Version Control
• Per user locking
• Version tracking

12
Job Manager

• Combine Projects into Jobs
• Set interdependencies
• Running Jobs

13
Schedule Manager

• Schedule Jobs
• On request
• Per schedule
• Multiple scheduling strategies

14
Session Manager

• Establish and maintain client connections
• RMI
• Simple, robust, built-in to Java
• Front end for all functions in server
• Security checking
• Authentication of users
• Authorization of commands

15
Security Manager

• Determine users privileges
• Control access to Projects, Jobs, etc.
• Custom Security Model
• Role-based ACLs
• Read, Write, Execute (Projects and Jobs)
• Read, Create, Modify (System Configuration)

16
Event Manager Logger

• Useful for future expansion
• Complex Hierarchy of Events
• All Events Logged
• Log4J format

17
Event Hierarchy
InfoserverEvent
UserEvent
UserLoginEvent
UserLoginFailedEvent
ProjectEvent
UserLogoutEvent
(Other level-2 events)
(Other level-3 events)
Listeners
18
Saving and Loading Projects
Client
Security Manager
Session Manager
Event Log Manager
Project Manager
Version Manager
Repository Manager
DB
19
Creating Jobs from Projects
Client
Security Manager
Session Manager
Event Log Manager
Project Manager
Job Manager
Repository Manager
DB
20
Scheduling Jobs to Run
Client
Security Manager
Session Manager
Event Log Manager
Job Manager
Scheduling Manager
Repository Manager
DB
21
Project Summary

• Relational Database storage
• Projects
• Operational Information
• Job Scheduling
• Tailored Security Model
• Version control
• Logging

22
Future work

• Distributed servers
• Clients, thick and thin
• Support for more databases
• More advanced scheduling algorithms

23
Thanks

• Professor E. A. Rundensteiner
• Arun Shastry
• Greg Goldberg
• Rest of the TotalETL Team

24
Questions?