Service Oriented Business Intelligence SoBI

1
Service Oriented Business Intelligence (SoBI)
June 2005

• Michael Horne, Conchango Rob Grigg, Conchango

2
Introductions

• Michael Horne
• Managing Consultant / Business Intelligence
  Architect
• PASS Program Committee Manager (Europe)
• 15 years in industry, History in Retail, CPG,
  Finance
• Particular areas of interest,
• ETL / EAI / Data Warehouse Data Integration
• Microsoft Business Intelligence
• SQL Server 2005 (its been a long wait)
• Rob Grigg
• Managing Consultant / Enterprise Architect
• Microsoft .NET PAC member
• 19 years in industry, History in Financial
  Trading, Energy Trading, Retail
• Particular areas of interest,
• EAI / SOA / Web Services
• Software architecture, Development tools and
  methodologies
• .NET and Agile Development
• Whidbey, Indigo

3
About Conchango

• Business Technology Solutions Provider
• Provide consultancy services and intellectual
  property
• Founded in 1991
• Offices in UK and USA
• Over 200 consultants
• Systems integration heritage
• Blue Chip client base
• 70 business from current customers

• Gold Certified Partner in 5 disciplines
• .NET Accelerator Programme Members
• .NET Partner Advisory Council
• Microsoft UK Architects Council
• Microsoft Global Strategic Architects Forum
• Collaborative Solutions Advisory Council
• Business Intelligence Advisory Council
• SharePoint Joint Development Programme
• SQL Server 2005 Ascend and TAP Program
  Participants
• BizTalk 2004 and 2006 TAP Participants
• VS and VSTS TAP Participants
• Indigo TAP Participants

4
SoBI Virtual Team

• Conchango
• Michael Horne
• Rob Grigg
• Microsoft
• Sean Gordon
• Simon Thurman

5
About SoBI

• SoBI is an attempt to combine two architectural
  paradigms that have developed independently,
  namely Business Intelligence/Data Warehousing and
  Service Orientation.

• We are attempting to define a framework we have
  called Service Oriented Business Intelligence
  (SoBI), in which both architectures can exist in
  unison and leverage the benefits of the other.

• We have identified guiding principles to ensure
  that the fundamental tenets of each of the
  component architectures are not broken.

6
Overview Service Orientation (SO)

• Service Orientation is an approach to building
  distributed applications
• Services expose capabilities through interfaces
• Interfaces exchange messages
• Schemas maintain message data standards
• Encapsulates functionality and provides
  abstraction
• We are talking services, not just web services
  (HTTP)
• Service Orientation is the a good approach to
  building agile and flexible applications and a
  good application level integration strategy

7
OverviewData Warehousing

• Data Gathering - Extract, Transform and Load
  (ETL)
• Data Storage - Data Warehouse (EDW)
• Data Presentation (The BI bit)
• OLAP
• Reporting

8
Why did SoBI happen?

• Have been looking at similarities in EAI / ETL
• Conceptually, Architecturally and in the Vendor
  space
• Real Project
• Looks like a BI project
• Client wants Service Orientation
• Complex App Landscape
• Many application silos
• No integration or integration strategy
• Many unstructured data sources (Mostly Excel)
• Dont Want Another Data Silo
• This is not a green field one schema one database
  project
• Want a better future
• Is essential that we build a better foundation
  for the future

9
Do we have common ground?Query the Warehouse
Business Service
Service Bus
Service Facade
Data Warehouse
10
Do we have common ground?ETL
Event Cache
Business Service
Data Warehouse
DW Event Agent
Service Bus
11
The views of SO and BI

• From an SO perspective, BI is seen a collection
  of services
• Data Publication Services (BI)
• Transformation other BI services
• From a BI perspective SO is seen as a collection
  of data sources

12
Different Challenges

• SO
• Small messages on demand
• Transformation tend to be simple
• BI
• Infrequent exchanges of (large) amounts of data
• Transformations complex
• Increasing drive for Real Time DW
• SoBI
• Leverages the strengths at the extremes
• Attempts to exploit this middle ground

Messages vs. Data
13
Core Strengths
14
SoBI Wins

• Provides a logical model of enterprise data which
  can be exploited by SO
• Ability to exploit best of breed transformation
  functionality
• Ability to reuse transformation logic usually
  hidden in ETL
• Support for real time data warehousing
• Ability to consolidate operational and BI data
• Brings interface abstraction patterns to BI
• Provides road map for integration
• Driving improvements in enterprise data quality

15
SoBI Vision

• Provide best practice implementation framework
• To integrate at the most appropriate
  architectural level
• To provide the data modelling of a BI project
  within the Service Orientation strategy of
  leaving the source systems in place
• Provide a common implementation for data
  transformations and data logic
• Data to Data
• Data to Service
• Service to Data
• Service to Service

16
SoBI Principles

• Accept a SO Strategy
• There is a strategic plan for Service Orientation
• System of Record data to be held in Enterprise
  stores or applications.

• Data Modelling
• Build a reference model of the enterprise data

• Data Ownership
• System of Record owns data
• The system (and service) owns the external
  representation
• DW owns Business Intelligence and reference data

• Governance
• Reuse transformation logic where ever possible
• Identify System of Record

17
GuidanceData Warehouse

• It is
• The single version of the truth for BI data
• It will
• Provide open access to data services
• Support ad hoc analysis
• Support pre-canned management reporting
• Consolidate data from disparate source systems
• Support Reference Data
• It will not
• Becoming a dumping ground for all data
• Become the data owner
• Be the default data source to other applications
• Support operational reporting

18
GuidanceService Orientation

• It is
• The architectural approach for application
  integration
• It will
• Provide Application to Application Integration
• Provide some event feeds to the DW
• Describe the services provided and the messages
  passed.
• Fulfil Operational Requirements
• Provide the infrastructure services for all
  applications
• It will not
• Be used in every circumstance
• Replace data import interfaces

19
SoBI FrameworkHigh level Architecture
20
BI TransformationsOverview
Drilldown Reuse transformation logic where
ever possible Whats the value-add?
21
BI Transformations and SoBI
22
BI Transformations and SoBI
23
BI Transformations and SoBI
24
BI Transformations and SoBI
25
Summary / Conclusion

• SoBI is the mixing of approaches from Service
  Orientation and Business Intelligence
• It attempts to solve real word problems of
  integration in an Enterprise of disparate stove
  piped systems
• It attempts to provide for Operational and MIS
  data
• It attempts to provide a road map for better
  class integration
• It attempts to provide a common data
  transformation mechanism
• It sets out guidance in the form of principles
  and patterns

26
Next Steps

• We have a SoBI project to deliver!
• SQL Server 2005 Integration Services is one
  technology that can enable SoBI
• SSIS is an enterprise application
• Support for ETL within an SOA
• XML Source Adapter
• WMI Task
• Web Services Task
• Event Handling
• Excel is a first class citizen
• RTM within project timeframe
• SoBI prototype - TVP 20-23 June 2005
• SoBI White Paper Conchango / Microsoft
• Phase 1 delivery Q405

27
Further Info

• rob.grigg_at_conchango.com
• mick.horne_at_conchango.com
• simont_at_microsoft.com
• segordon_at_microsoft.com

28
SoBI PatternsOverview

• Have identified a number of patterns
• Patterns will be used to prescribe the solution
  in a given scenario
• Including Real world exceptions to the principles
  
• Grouped by
• System Types
• E.g. Batch processing, Real-Time
• System Constraints
• E.g. Processing windows, High data volumes

29
SoBI Patterns System TypesSoBI Compliant system
Event driven transfer System provides Messages
as changes occur
30
SoBI Patterns System TypesSystems with short
life expectancy
Data is held on multiple systems which are known
to have a short life expectancy, but which will
be replaced after project completes
31
SoBI Patterns System TypesNon or Semi
Structured Systems
Data sources which contain information which must
be consumed by the solution but which are held in
semi-structured or unstructured formats, such as
spreadsheets and document management systems.
32
SoBI Patterns System ConstraintsBatch
Processing Systems
Schedule driven batch transfer Source system
access is constrained by an operational
window Notification from scheduler invokes a
pull from the source system
33
SoBI Patterns System ConstraintsBatch
Processing Systems
Event driven batch transfer Source system access
is constrained due to processing
tasks Notification from completion of Job
invokes a pull from the source system