Enterprise Information Integration

1
Enterprise Information Integration

• DAMA
• Key Components of EII

2
Disparate data shapes and formats make this a
very hard problem
Data shapes are different
Data formats are different
CONSUMERS
SQL
SOAP
CUSTOMER ADDRESS Address 1 char(30) Address
2 char(30) Address 3 char(30) Address
4 char(30) City char(30) State char(2) Zip cha
r(5)
CUSTOMER ADDRESS Unit char(20) Street
No char(20) Street char(40) City char(40) State
/Province char(3) Zip/Postal Code
char(6) Country char(3)
BILLING SYSTEM
ORACLE
SIEBEL
3
Home grown solutions work to a point.
APPLICATION 1
APPLICATION 2
APPLICATION 3
ODBC
Web Service
DATA SERVICES LAYER
ODBC
JDBC
JDBC
Order System
CRM System
Billings System
ERP
4
Views Across Disparate Data Sources
Enterprise Application
Desktop Application
EII VIEWS Separate Logical From Physical Embed
Back-End Connectivity Elevate to Business
Entities Enforce Entitlements
5
Highest Level EII Components
modeling
runtime
Metadata Repository
6
EII Server Architecture
Client or Enterprise Application
Relational (SQL)
Web Services (SOAP)
EII SERVER
MODELING TOOL
Security
EII VIEWS
Cache
Query Processing Engine
METADATA REPOSITORY
Business Transactions
Security
Relational (SQL)
Web Services (SOAP)
Relational Database
Excel
Web Service
Flat File
7
Security Basics

• User Authentication
• Access to EII server
• Ability to query resources
• Resource Authorization
• Public views, resources, physical data sources
• Physical Data Source Authentication
• Supplied by administrator during introspection
• Mapped to EII resource authorization

8
Security Resources vs. Data

• Resource Access
• Read
• Write
• Execute
• Relational Data Access
• Select
• Insert
• Update
• Delete
• Different Data Types
• Relational, Hierarchical

9
Security Additional Factors

• 3rd Party Credentials
• LDAP integration
• Pass Through Credentials
• Supply credentials at runtime
• Non-Secure Physical Data Sources
• Flat files now have secure access mode

10
Performance

• Data Handling Goals
• Reduce movement of data
• Reduce data processing
• How to do this efficiently?
• Optimize!
• Plan Generation
• Plan Execution
• Cache
• Stream

11
Performance Plan Optimization

• Data source capability framework
• Overall features and syntax
• Function mapping
• Case sensitivity and white-space treatment
• Allows pushing work down
• Data Source Grouping
• Rule Based Optimizations
• Cost Based Optimization
• Statistics collections
• Join Algorithm Selection
• Nested loops, Hash, Sort-merge, Distributed
  Semi-join
• Operation pruning
• Orphaned projections
• Null scans
• Overridden ordering

12
Performance Plan Execution

• Pipelined results
• Source to client
• Demand driven data retrieval
• Partial data sets
• No redundant value cloning or conversion
• Dynamic hybrid memory-disk processing
• Parallel execution and retrieval
• scans
• Query throttling
• Queue queries
• Redundant scan multiplexing

13
Performance Caching

• Result Set Caching
• Reduce data source load
• Cache schedule
• Memory, db, or disk
• Query Plan Caching
• Reduce query set-up
• Lower processing time

14
Performance

• Streaming Optimizations
• Streaming transformation
• Streaming result sets
• Relational
• Hierarchical
• Optimization tuning/management
• Manual overrides

15
Modeling Basic View Support

• Standards Support!!!
• SQL, XML, XSLT
• Relational View Generation
• Standard SQL
• RDBMS, Flat File
• Hierarchical Data
• XML document
• Web Service, File
• Transformations
• Relational lt-gt Hierarchical
• Same lt-gt Same
• Complicated with result set generation

16
Modeling Promotion/Publishing

• Relational
• View
• Stored Procedure
• Web Service
• Messaging
• Other

17
Modeling Advanced Features

• Metadata Compatibility
• Import/Export
• Standards support
• Ontology, Semantics
• Global Data Model
• Resource support
• Depth - Views vs. stored procedures
• Binding
• Rebinding Physical Data Sources
• Scripting
• Applying logic to result sets
• Data cleansing

18
Other Factors

• Data Sources
• Breadth and Depth
• Management
• Insight into your middleware
• Automation
• cache scheduling
• API
• Programmatic access

19
Review

• Key Issues
• Type of Data
• Relational, Hierarchical
• Source of Data
• RDBMS, Web Service
• Security
• Authentication, Authorization
• Server, Resources, and Physical Data Sources
• Performance
• Query optimizations, Cache, Stream
• Modeling
• Basic view creation and publishing

20

• Questions?

21
Introducing Composite Views
Client
COMPOSITE INFORMATION SERVER
COMPOSITE VIEWS
Relational Database
Web Service
ERP Systems
Data Warehouse
22
Composite Implementation
Customer Application
BI Tool
Excel
Information windows
jdbc
odbc
WS
studio
Composite Information Server
administration
Security Authorization
metadata repository

• Support Materials
• Documentation
• Training
• Example Application
• Technical Support
• Patches

CIS Core
Packaged apps
Metadata model
Future Products
3rd Party Apps
Customer Application
3rd Party Integration
23

• EII Use Cases

24
EII extends EAI investments
25
Scenario External data needs to be combined
with DWH data
Less than 10 of Enterprise Data exists in the
Data Warehouse
26
EII Enables High Performance Portals
27
Corporate Dashboard
28
EII Advantages Over Traditional Approach
Traditional Approach
EII Approach
Change
BI Tools
1) More Data Sources
BI Server
2) Wider Audience
3) Speed to Solution
Query the DW
4) Faster Change
Data Warehouse
5) Timely Information
Setup 18 Months
Web Services
Data Warehouse
Chg
Move Everything Overnight
ODS
ODS
Setup 1 Mon