Title: Service Oriented Business Intelligence SoBI
1Service Oriented Business Intelligence (SoBI)
June 2005
- Michael Horne, Conchango Rob Grigg, Conchango
2Introductions
- Michael Horne
- Managing Consultant / Business Intelligence
Architect - PASS Program Committee Manager (Europe)
- 15 years in industry, History in Retail, CPG,
Finance - Particular areas of interest,
- ETL / EAI / Data Warehouse Data Integration
- Microsoft Business Intelligence
- SQL Server 2005 (its been a long wait)
- Rob Grigg
- Managing Consultant / Enterprise Architect
- Microsoft .NET PAC member
- 19 years in industry, History in Financial
Trading, Energy Trading, Retail - Particular areas of interest,
- EAI / SOA / Web Services
- Software architecture, Development tools and
methodologies - .NET and Agile Development
- Whidbey, Indigo
3About Conchango
- Business Technology Solutions Provider
- Provide consultancy services and intellectual
property - Founded in 1991
- Offices in UK and USA
- Over 200 consultants
- Systems integration heritage
- Blue Chip client base
- 70 business from current customers
- Gold Certified Partner in 5 disciplines
- .NET Accelerator Programme Members
- .NET Partner Advisory Council
- Microsoft UK Architects Council
- Microsoft Global Strategic Architects Forum
- Collaborative Solutions Advisory Council
- Business Intelligence Advisory Council
- SharePoint Joint Development Programme
- SQL Server 2005 Ascend and TAP Program
Participants - BizTalk 2004 and 2006 TAP Participants
- VS and VSTS TAP Participants
- Indigo TAP Participants
4SoBI Virtual Team
- Conchango
- Michael Horne
- Rob Grigg
- Microsoft
- Sean Gordon
- Simon Thurman
5About SoBI
- SoBI is an attempt to combine two architectural
paradigms that have developed independently,
namely Business Intelligence/Data Warehousing and
Service Orientation.
- We are attempting to define a framework we have
called Service Oriented Business Intelligence
(SoBI), in which both architectures can exist in
unison and leverage the benefits of the other.
- We have identified guiding principles to ensure
that the fundamental tenets of each of the
component architectures are not broken.
6Overview Service Orientation (SO)
- Service Orientation is an approach to building
distributed applications - Services expose capabilities through interfaces
- Interfaces exchange messages
- Schemas maintain message data standards
- Encapsulates functionality and provides
abstraction - We are talking services, not just web services
(HTTP) - Service Orientation is the a good approach to
building agile and flexible applications and a
good application level integration strategy
7OverviewData Warehousing
- Data Gathering - Extract, Transform and Load
(ETL) - Data Storage - Data Warehouse (EDW)
- Data Presentation (The BI bit)
- OLAP
- Reporting
8Why did SoBI happen?
- Have been looking at similarities in EAI / ETL
- Conceptually, Architecturally and in the Vendor
space - Real Project
- Looks like a BI project
- Client wants Service Orientation
- Complex App Landscape
- Many application silos
- No integration or integration strategy
- Many unstructured data sources (Mostly Excel)
- Dont Want Another Data Silo
- This is not a green field one schema one database
project - Want a better future
- Is essential that we build a better foundation
for the future
9Do we have common ground?Query the Warehouse
Business Service
Service Bus
Service Facade
Data Warehouse
10Do we have common ground?ETL
Event Cache
Business Service
Data Warehouse
DW Event Agent
Service Bus
11The views of SO and BI
- From an SO perspective, BI is seen a collection
of services - Data Publication Services (BI)
- Transformation other BI services
- From a BI perspective SO is seen as a collection
of data sources
12Different Challenges
- SO
- Small messages on demand
- Transformation tend to be simple
- BI
- Infrequent exchanges of (large) amounts of data
- Transformations complex
- Increasing drive for Real Time DW
- SoBI
- Leverages the strengths at the extremes
- Attempts to exploit this middle ground
Messages vs. Data
13Core Strengths
14SoBI Wins
- Provides a logical model of enterprise data which
can be exploited by SO - Ability to exploit best of breed transformation
functionality - Ability to reuse transformation logic usually
hidden in ETL - Support for real time data warehousing
- Ability to consolidate operational and BI data
- Brings interface abstraction patterns to BI
- Provides road map for integration
- Driving improvements in enterprise data quality
15SoBI Vision
- Provide best practice implementation framework
- To integrate at the most appropriate
architectural level - To provide the data modelling of a BI project
within the Service Orientation strategy of
leaving the source systems in place - Provide a common implementation for data
transformations and data logic - Data to Data
- Data to Service
- Service to Data
- Service to Service
16SoBI Principles
- Accept a SO Strategy
- There is a strategic plan for Service Orientation
- System of Record data to be held in Enterprise
stores or applications.
- Data Modelling
- Build a reference model of the enterprise data
- Data Ownership
- System of Record owns data
- The system (and service) owns the external
representation - DW owns Business Intelligence and reference data
- Governance
- Reuse transformation logic where ever possible
- Identify System of Record
17GuidanceData Warehouse
- It is
- The single version of the truth for BI data
- It will
- Provide open access to data services
- Support ad hoc analysis
- Support pre-canned management reporting
- Consolidate data from disparate source systems
- Support Reference Data
- It will not
- Becoming a dumping ground for all data
- Become the data owner
- Be the default data source to other applications
- Support operational reporting
18GuidanceService Orientation
- It is
- The architectural approach for application
integration - It will
- Provide Application to Application Integration
- Provide some event feeds to the DW
- Describe the services provided and the messages
passed. - Fulfil Operational Requirements
- Provide the infrastructure services for all
applications - It will not
- Be used in every circumstance
- Replace data import interfaces
19SoBI FrameworkHigh level Architecture
20BI TransformationsOverview
Drilldown Reuse transformation logic where
ever possible Whats the value-add?
21BI Transformations and SoBI
22BI Transformations and SoBI
23BI Transformations and SoBI
24BI Transformations and SoBI
25Summary / Conclusion
- SoBI is the mixing of approaches from Service
Orientation and Business Intelligence - It attempts to solve real word problems of
integration in an Enterprise of disparate stove
piped systems - It attempts to provide for Operational and MIS
data - It attempts to provide a road map for better
class integration - It attempts to provide a common data
transformation mechanism - It sets out guidance in the form of principles
and patterns
26Next Steps
- We have a SoBI project to deliver!
- SQL Server 2005 Integration Services is one
technology that can enable SoBI - SSIS is an enterprise application
- Support for ETL within an SOA
- XML Source Adapter
- WMI Task
- Web Services Task
- Event Handling
- Excel is a first class citizen
- RTM within project timeframe
- SoBI prototype - TVP 20-23 June 2005
- SoBI White Paper Conchango / Microsoft
- Phase 1 delivery Q405
27Further Info
- rob.grigg_at_conchango.com
- mick.horne_at_conchango.com
- simont_at_microsoft.com
- segordon_at_microsoft.com
28SoBI PatternsOverview
- Have identified a number of patterns
- Patterns will be used to prescribe the solution
in a given scenario - Including Real world exceptions to the principles
- Grouped by
- System Types
- E.g. Batch processing, Real-Time
- System Constraints
- E.g. Processing windows, High data volumes
29SoBI Patterns System TypesSoBI Compliant system
Event driven transfer System provides Messages
as changes occur
30SoBI Patterns System TypesSystems with short
life expectancy
Data is held on multiple systems which are known
to have a short life expectancy, but which will
be replaced after project completes
31SoBI Patterns System TypesNon or Semi
Structured Systems
Data sources which contain information which must
be consumed by the solution but which are held in
semi-structured or unstructured formats, such as
spreadsheets and document management systems.
32SoBI Patterns System ConstraintsBatch
Processing Systems
Schedule driven batch transfer Source system
access is constrained by an operational
window Notification from scheduler invokes a
pull from the source system
33SoBI Patterns System ConstraintsBatch
Processing Systems
Event driven batch transfer Source system access
is constrained due to processing
tasks Notification from completion of Job
invokes a pull from the source system