MetaMatrix: Metadata Management and Enterprise Information Integration

1
MetaMatrixMetadata Management and Enterprise
Information Integration

• Randall Hauch and Chuck Mosher

presented at
Extended Metadata Registry (XMDR) Summer
Meeting Lawrence Berkeley Laboratories July 12,
2005
2
MetaMatrix at Work Today
Providing access to integrated information for

• Government
• Financial services
• Telecommunications
• Life Sciences
• Manufacturing
• Pharmaceuticals

and many others
3
Enterprise Information Integration (EII)

• Exposing information in the ways that people and
  processes understand
• Turns disparate data into usable and meaningful
  information
• More than just federating access to multiple data
  sources
• The essential aspects of EII are
• Real data in real time
• Enterprise-class high performance, scalable,
  manageable, adaptable,
• Includes queries, updates, inserts, deletes and
  procedures
• Ability to cache and stage pre-integrated data
  when desired
• Semantically-relevant views that abstract the
  user from the where and how
• Rich metadata usable by people and machines
• Metadata to drive mechanics of integration and
  application behavior
• Metadata to describe the semantics of the
  integrated information
• Metadata to discover and analyze (e.g.,
  dependencies, impacts, lineage, )
• Information governance that can enforce
  enterprise policies for security, enablement,
  management, integration, semantics
• Productive, maintainable and flexible through
  model-driven approach

4
MetaMatrix Any View From Any Source
Management
Governance

• Information on demand
• High performance
• Highly scalable
• Manageable
• Adaptable
• Reusable

Information Services
MetaMatrix Information Integration Platform
Persons
ODS
Product Data
Resources
Documents
Images
Facilities
Accounting
CRM
5
MetaMatrix Model-driven information integration
Providing a meaningful enterprise information
landscape
6
MetaMatrix Server
The integration engine

• Access through standard and widely-used
  mechanisms
• SQL, XQuery, SOAP over JDBC, ODBC, HTTP(S), JMS
• Scalable and high-performance
• Large data, large numbers of concurrent clients,
  complex queries
• Clustered, connection pools, batching/cursoring,
  streaming, etc.
• Flexible caching and staging, if desired
• Multiple virtual databases (and versions) can be
  deployed in a cluster
• Executing requests for information
• Models declare what should happen during
  integration
• Optimization is done by the engine using a
  variety of algorithms techniques
• Supports reads and writes
• Safety and control
• Provides security for enterprise information
• Flexible mechanisms to leverage existing access
  authorization infrastructure
• Access control and auditing

7
MetaMatrix Server
Model-driven information services
Reusable Integrated Business Objects
Exposed Information Services
Enterprise Information Sources (EIS)
Information Consumers
8
MetaMatrix Server
Model-driving abstracts intention from execution

• Models declare the what
• The intent of the behavior and what should happen
• Maintenance, versioning, packaging, and
  deployment of models
• Interpretation, optimization and execution (the
  how) is left to the runtime engine
• Access providing multiple APIs (JDBC, ODBC,
  SOAP)
• Orchestration planning coordinating the
  interaction with multiple sources, including
  transactional updates
• Caching enable reuse of already integrated
  information
• Safety guard against unexpected and
  unacceptable uses
• Scalability handling massive volumes of
  information and large numbers of concurrent
  consumer requests
• Performance choosing appropriately and using
  the optimized algorithms for integrating
  information
• Security policies applied at runtime applied
  heterogeneously
• Traceability expose and ensure compliance
• Improvements engines can be improved without
  changes in the models
• The models can be used for sharing and
  (re)discovering the what
• Traceability of information and impact analysis
• Search, discovery, maintenance, and reuse

9
MetaBaseCreate, Maintain, Validate, Manage,
and Package the Models
10
Modeling Disparate Data Systems
Each system has its own semantics and syntax
Relational
Transformations
XML
Datatypes
ltcustomergt ltnamegtJohn Smithlt/namegt ltstreetAddres
sgt12 Main Streetlt/streetAddressgt ltcitygtSpringfiel
dlt/citygt lt/customergt
Processes BPM/BPEL
Domain UML/ER
Generic Typed Relationships
Organizations Business Models
Web Services WSDL, OWL-S
Ontologies OWL/RDF
Taxonomies
11
Model Development Environment
Modeling disparate information

• Multiple, domain-specific modeling languages
• Use the modeling language best suited for the
  system being described
• Leverage native artifact structures (e.g., CSV,
  DDL, WSDL, XSD, OWL, etc.)
• Standards-based architecture
• Integrated but extensible environment
• Built on Eclipse plug-in platform
• EMF, UML2, XSD and other plug-ins

12
MetaBase Modeler
Eclipse-based model development environment
Menus
Toolbars
Navigation
Multi-PageEditors
Properties
Problems, Console, Logs,
13
MetaBase Modeler
Model the information sources

• Model the disparate information sources
• Relational DBs
• Content Management Systems
• Files
• Services
• Applications
• Uses and retains domain-specific modeling
  languages
• Relational models have Tables, Foreign Keys,
  Columns, etc.
• UML models have Packages, Classes,
  Attributes, etc.

14
MetaBase Modeler
Model enterprise datatypes
15
MetaBase Modeler
Model the reusable business objects and views

• Define reusable business objects
• Uses enterprise datatypes
• Map to other business objects or to EISs and
  integrate
• Join
• Union
• Functions
• Procedures
• Criteria
• Perform schema and semantic matching

16
MetaBase Modeler
Obtain XML from non-XML sources
GIVEN Fixed XML Schema WANT Data complying to
schema
GIVEN Data Sources containing Information to
integrate
NEED Mapping from Data to XML
?
17
MetaBase Modeler
Model mapping to XML from non-XML sources

• Model Web Service operations
• Model XML messages (based upon XML Schema)
• Map reusable business objects to any XML schema
• Auto-generate WSDL artifacts, implemented by
  engine

18
MetaBase Modeler
Model mapping to XML from non-XML sources
19
MetaMatrix Data Services
SOAP, WSDL and UDDI provide the building blocks

• Are Web Services that make available enterprise
  information
• Expose existing enterprise information through
  web services
• Do all of the work to transform any data in any
  format to a W3C compliant service
• Implements all of the logic to effect the
  transformation
• Do not implement application logic
• In SOA, decouples the data from the application
  while making the data discoverable and
  accessible

20
MetaBase Modeler
Model web services on top of data sources
21
Model Development Environment
Domain-specific and domain-independent
functionality

• General modeling functionality
• Standard tree, table, and property views
• Create, edit, delete, clone, copy/paste, move,
  undo/redo
• Types of objects in model dictated entirely by
  metamodel
• Standard model persistence (defaults to XMI, but
  is customizable)
• Unified and common places for description and
  custom properties
• Rule-based validation of model content identifies
  areas of models that are incorrect or problematic
• Compare two models (or model versions), view
  differences, and merge
• Find and search for objects meeting various
  criteria
• Extensible importers, exporters and wizards
• Integrated diagramming
• UML class, graph, mapping, and transformation
  diagrams
• Ability to add other diagram types

22
Model Development Environment
Eclipse-based integrated toolset
Menus
Toolbars
Navigation
Multi-PageEditors
Properties
Problems, Console, Logs,
23
MetaBase Modeler
Compute differences between models or model
versions
24
Technology behind MetaBase
25
Eclipse Consortium
Partial list of members
26
Eclipse
What is it?

• Universal platform for integrating tools
• Platform for functionally-rich applications
  (rich client)
• Architecture that is open, extensible, and based
  on plug-ins

Rich Clients
27
Model Development Environment
Eclipse-based integrated toolset
Menus
Toolbars
Navigation
Multi-PageEditors
Properties
Problems, Console, Logs,
28
Model Development Environment
Contributing tools, wizards and functionality
Some of these are provided by Eclipse platform,
other are added in by other plug-ins The
importer functionality shows up in the right
place
29
Model Development Environment
Using metamodels to drive behavior
Workspace Explorer
30
Model Development Environment
Using metamodels to drive behavior
Menus
Wizards
Views
Validation
31
Model Development Environment
Using metamodels to drive behavior
Repository Contents
RepositoryConnections
Userinformation
32
Using a Model-Based Approach
To model disparate information

• Information takes many forms
• Relational tables, views, procedures
• XML elements, attributes, namespaces
• Object-oriented classes, properties, operations,
  associations
• Simple and complex datatypes
• Services, components, messages
• Could choose one modeling language or multiple
  tools
• Now have a metadata/model integration and
  interoperability challenge
• But multiple domain-specific languages are best
• Use the modeling language best suited for the
  system being described
• Leverage native artifact structures (e.g., DDL,
  WSDL, XSD, OWL, etc.)
• Support importing and exporting metadata
• One modeling environment to view, discover,
  relate, map, and report across all of the
  different types of models

33
Modeling Multiple Types of Systems
Using multiple modeling languages

• A metamodel defines a domain-specific modeling
  language
• Accurate and precise
• Structured with semantic concepts of the domain
• Support for multiple metamodels / modeling
  languages
• Domain-specific models are easily understood by
  users
• Makes possible mixing and matching
• Enables treating heterogeneous models in
  homogeneous ways
• Use metamodels to drive behavior of modeling
  environment
• Menus, wizards, views are all driven with the
  metamodels
• Object construction is driven by metamodels
• Validation is driven by metamodels
• Diagramming is driven by metamodels
• Models are serialized to files using metamodels
  and XMI rules

34
Modeling Multiple Types of Systems
Some of the modeling standards

• Meta-Object Facility (MOF)
• Defines an architecture for modeling
• Used to create metamodels or domain-specific
  modeling languages that define syntax and
  semantics of models
• XML Metadata Interchange (XMI)
• Defines rules that dictate how models defined
  with MOF are serialized to and from XML files
• Not a single format, but rather patterns for
  defining formats (XML Schemas) in terms of the
  metamodels
• Common Warehouse Metamodel (CWM)
• Defines metamodels for various types of
  information systems
• Relational, record, hierarchical, OLAP, etc.
• Unified Modeling Language (UML)
• The well-known metamodel for object-oriented
  systems
• OMGs Model Driven Architecture (MDA)
• Defines architecture for using models to drive
  systems

35
OMGs Meta-Object Facility (MOF) Architecture
Domain-specific languages (metamodels)
36
Model Development Environment
Extending existing metamodels

• Add custom fields (properties) to existing
  metaclasses
• Modeling activity
• Users can do this very dynamically
• Extensions can be reused
• Different models can use different extensions
• Easily change extensions even after they are used
• Useful when an existing metamodel is structurally
  correct
• Typically want do add some information
• The types of things being modeled doesnt really
  change
• Treated like any other property
• Persistent, searchable, validated, etc.

37
Model Development Environment
Extending existing metamodels

• Define the extensions
• Define the properties
• Define the targetmetaclasses
• Additional properties appear on appropriate
  instances

38
Model Development Environment
Adding new metamodels

• When the system being modeled cant be
  effectively and easily described using an
  existing modeling language
• Constructs and syntax of system are unique or
  sufficiently different
• People that are modeling the system dont need to
  translate concepts in their heads
• Extends the existing behavior for the new type of
  system
• Same ways to create, edit, change, and refactor
  objects
• Same validation framework
• Same wizards, importers, exporters
• Ability to reference models defined by other
  metamodels (e.g., relational, generalized
  relationships, XSD, ontologies, etc.)
• New components can always be added
• Views and editors (e.g., diagrams, forms, etc.)
• Wizards (e.g., importers, exporters)
• Analysis tools

39
Model Development Environment
Adding new metamodels

• Metamodels are defined via UML
• Generate Java source code
• Compile, package and deployable plugins into
  existing installations

40
Model Development Environment
Adding new metamodels
41
Metadata Management with MetaBase
Manage and make the metadata available and usable

• Packaging and deployment
• Models and artifacts are consistent, valid,
  complete
• Supports multiple design/test/production
  processes
• Enables model-driving applications
• Share, manage and control models and other assets
• Configuration management
• Access control policies
• Distributed and scalable
• Repository is searchable and is another valuable
  enterprise asset
• Object-level details (properties, relationships)
• Through SQL, XQuery and SOAP
• Analyze, report, search and discover metadata
• Metadata Federation and Integration
• Integrate multiple disparate metadata
  repositories to provide a single, enterprise
  virtual repository

42
Metadata Management with MetaBase
Packaging metadata for deployment

• Assemble consistent models into deployable
  archive
• Archives the specific models
• Automatically generate additional assets from the
  models (e.g., RDF, OWL, custom formats)

43
Metadata Management with MetaBase
Accessing the metadata in the repository

• Models and other artifacts are added to
  repository
• Contents are available through SQL, XQuery and
  SOAP
• Model contents exposed at the object level
• Non-models available as streams

Access viaSQL, XQuery SOAP
MetaBase Modeler
share, version, manage, discover
44
Metadata Management with MetaBase
Accessing the metadata in the repository

• Repository exposes a virtual database of its
  contents

metamodel-specific virtual layers
Reports (Virtual XML)
Relational (Virtual Relational)
UML2 (Virtual Relational)
Datatypes (Virtual Relational)

metamodel-independent virtual layers
Models (Virtual Relational)
Utilities (Virtual Relational)
History (Virtual Relational)
Metamodels (Virtual Relational)
Store (Physical)
metamodel-independent physical layer
MetaBase Repository
45
Metadata Management with MetaBase
Metadata Reporting
46
MetaMatrix
Future Directions

• Productization of embeddable technologies
• Enhanced support for ontologies
• Automatic generation of OWL/RDF from models
• Ability to import, relate and use ontologies
• Provide semi-automated semantic matching
  capabilities
• Provide data access functionality through
  ontologies
• Additional focus on data services
• Enhanced automation of modeling activities
• Includes generation of data services from
  ontologies
• Enhanced and improved exposure of repository
  contents
• Improved performance and extensibility
• Semantic searching using user-defined lexicons
• Additional analyses and services
• Embed in Modeler and in each virtual databases

47
Summary
Enterprise Information Integration (EII)

• Provides real data in real time in ways that are
  meaningful to consumers
• Hides the syntax and mechanics of disparate data
  sources
• Provides information using semantics of those who
  need it
• Requires modeling environment
• Model disparate data sources with domain-specific
  languages
• Share, collaborate, search, validate, view, map,
  relate, and reuse models
• Requires metadata management
• Control, manage, discover, analyze report
  across all enterprise metadata
• Use the metadata to drive systems
• Requires scalable and high-performance query
  engine
• SQL, XQuery, and SOAP over JDBC, ODBC, HTTP(S),
  JMS,
• Planning, optimization, access control, auditing
• Pre-packaged connectors with extensible connector
  framework
• Reads and writes

48
Extended Metadata Registry (XMDR) Project
Differences, Similarities and Possible Synergies
with MetaMatrix

• Repository vs. Registry?
• MetaMatrix MetaBase is a repository and model
  development suite, focused on creating,
  maintaining, relating, managing, and packaging
  metadata
• XMDR is a registry of metadata analogous to a
  metadata warehouse or mart (is this true?)
• Standards based
• MetaMatrix MetaBase is architected on OMG
  standards, but supports other standards for
  interoperability (W3C XML, XSD, and soon RDF/OWL)
• XMDR is architected largely on the lower level
  W3C standards (XML, RDF, SOAP)
• Both expose mechanisms to query and search
• MetaBase exposes repository contents through
  SQL, XQuery and SOAP
• Exposed schema can be customized or
  federated/integrated with other
  repositories/registries using MetaMatrixs
  model-driven virtual database technology
• Same schema will be available in each virtual
  database instance
• XMDR exposes metadata through query language and
  SOAP
• Both support navigation, discovery, version
  management, rationalization, harmonization, and
  validation (local and global?)
• Both are designed to be highly extensible
• MetaMatrix could provide XMDR services on top of
  MetaBase