Metadata: Challenges

1
Metadata Challenges Opportunities

• Presented to DAMA Central Virginia Group
• 04/22/2004

2
Metadata Challenges Opportunities

• What is metadata?
• Why do we need metadata?
• How is metadata used in the organization?
• What are the challenges with metadata?
• What are some metadata opportunities at
  Dominion?
• Group discussion (Time limitation)
• Are you happy with your metadata
  implementations?
• Do you know who is doing well?

3
What is metadata?

• Data about data.
• Describes the who, what, when, and where of
  data.
• Everything we know about a piece of data
  starting at the point of entry into the source
  system and flowing all the way to the end report
  or cube of a business user. Its the pedigree of
  the data.
• Metadata is not something done after the fact.
  It is the fact.
• Only 10 of business is managing metadata to
  some degree.
• David Marco - Metadata is Knowledge!

4
Why do we need metadata?

• Some business drivers
• Data access
• Information Black Box
• Drive down business costs
• Six Sigma
• Sarbanes-Oxley
• Regulation

5
How is metadata used?

• Technical metadata
• Application metadata
• Process metadata
• Business metadata
• Quality metadata

6
How is metadata used?

• Technical metadata
• Describes
• The location of the data
• The format of the data
• Other Physical Characteristics of the data

7
How is metadata used?

• Application metadata
• Describes
• Which data is accessed
• How the data is used
• How frequent access occurs
• Who is using the data

8
How is metadata used?

• Process metadata
• Describes
• Where the data is captured
• How the data is captured
• What transformation has occurred to the data

9
How is metadata used?

• Business metadata
• Describes
• The meaning of the data in non-technical terms
• Descriptions
• Examples
• Business Rules

10
How is metadata used?

• Quality metadata
• Captures measures of
• Data correctness
• Data integrity
• Data reliability

11
Who uses metadata?

• Data Architects
• Data Administrators
• Application Developers
• Business Analysts
• Systems Analysts
• Application Programmers (e.,g ETL developers)
• Business End-Users (OLAP Ad-Hoc Producers and
  Consumers)
• Security Analysts and Auditors

12
Challenges with metadata

• Why are we struggling?
• Unclear requirements
• Implementation strategies are fuzzy
• Roles and responsibilities are vague
• Limitations of project management
• Scope
• Runaway Train
• Its expensive
• Technology
• Organization
• ROI is vague (limited measurements)

13
Challenges with metadata

• Unclear requirements
• Root cause (difficult to define strategies or
  RRs)
• Requirements (at a minimum)
• Why do we need the metadata?
• What metadata is needed to be captured, where
  and how?
• How much integration is needed?
• How will the metadata be used, by who and
  where?
• What are the security access requirements?
• Are there code of conduct limitations?

14
Challenges with metadata

• Fuzzy Implementation Strategies
• Five Strategies we are looking at
• A centralized integrated tool suite
• Build own process / tools
• Maintain disparate metadata (tools and
  processes)
• Build a common metadata portal
• Do nothing

15
Challenges with metadata

• Vague Roles and Responsibilities
• Sponsor to understand importance and allocate
  resources
• Program Manager to define and communicate the
  strategy
• Source Data Architects to maintain application
  metadata
• Data Stewards to be the authority of business
  metadata
• Metadata Architect to design/maintain the
  technical metadata
• Metadata Administrator to integrate and provide
  access via procedures and tools
• ETL Developers to capture process metadata
• Business users to provide feedback provide
  quality metadata

16
Metadata Opportunities at Dominion

• CIO (Lyn McDermid) Sponsorship
• Data Domain (Data access architecture and
  analysis)
• New role in IT - Data Architect
• Seeded Data Architects within each Business Unit
• Centralized Data Architect Process Group
• Centralized Business Intelligence Organization
  within IT
• BI Mission - the process of making data
  actionable within the organization
• BI needs metadata to be successful

17
Metadata Strategy

• Link Metadata Strategy to BI Strategies
• Approved 2004 Funding for Metadata Repository
  Project
• Requirements Definition
• Define approach for implementing metadata
  within BI
• Determine level of metadata process integration
  
• Resource plan (implementation and operational
  support)
• Build/Buy analysis for support toolltsgt
• Which Organization will maintain the Metadata?
• Pilot Implementation of a in-house Metadata
  Repository Tool

18
End of Dominion Part or Presentation

19
Data Architect

• Designs data associated with a major business
  function or process.
• Maintains current information on data
  definitions, location(s) and uses.
• Identifies proper data sources to meet new
  requests for information.
• Ensures consistency and integrity of data in
  assigned subject area(s).
• Designs new data values or structures to support
  applications development or data access
  requirements
• Designs security and data management processes to
  meet access and availability requirements.

20
Data Resource Repository Pilot Project

• Developed by the IT-Delivery Business
  Intelligence Team and Data Architect Process
  Group
• Requirements
• Inventory resources available for end-use data
  access
• ID data Pedigree - Map data to ID existence,
  intent, source, transformation rules,current
  format and storage locationltsgt
• Business and technical use
• Keep it simple (KISS) - Iterative Development
• Pilot use, ensure benefit and ID future direction
• Reduce manually intensive processes

21
Technical Architecture Model for End-Use Data
Access
OPERATIONAL Source Systems
Transactional Data
EXTRACTION
STAGING
STORAGE / Presentation LAYER
DATA ACCESS
LAYER
LAYER
LAYER
Oracle
DB2
MIDDLEWARE
TEMPORARY
SUBJECT
Browser
DATA
DATA
Delivery
STORES
MARTS
Software
SQL Server
Lotus Notes
Spreadsheets




Ad-Hoc Query



Load
Selection
Cleanse


Requirements

Analytic Applications

Index

Transform

Transport

• Modeling
• Cleanse

Report Writers
Inventory(Re-Use)


Standardize

Refresh
Metadata

(Standardize, Catalog, Monitor)
Security Access within the Data Storage Level and
not via Application (Code of Conduct between
Regulated and Non-Regulated Businesses)
22
Data Resource RepositoryData Model
System
Business Term
Container
Container Elements
Decode Value
23
Data Model Objects

• Container - holds pieces of data
  (file,report,cube,ETL Build)
• Data Element - physical data stored within a
  container
• Business Term - The name for a piece of data as
  expressed in business terms (what it is commonly
  known to the business users). This is the
  building block that will tie all things together.
  
• System - logical or physical grouping of
  containers

24
Demo

• Demo URL ----gt

25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
Example - How does it work?

• A search on Business Term will return all
  specific Container data elements (CDEs) that are
  mapped to it and any transformation rules.
• IF NEW TERM OR CONTAINER
• Define the source data element
• Define the Target (datamart) data element
• Define the Business Term and map both to the same
  Business Term
• We have a billing account identifier(ID_BA) from
  the CBMS system in table CU02TB01 which the users
  want as part of several reports.

33
Define Source Data Element

• Check to see if the Business Term (Billing
  Account Identifier) is currently defined.
• Add if missing
• Find the data element in the source system, what
  table is the master
• Create a System (CBMS), if necessary
• Put a container in the system (CU02TB01), if
  necessary
• Add your data elements (ID_BA) to the container
  (CU02TB01) and map it to the Business Term
  (Business Account Identifier)

34
Define Target (Datamart) Data Element

• Datamart System (Delivery-BI) already exists
• Put a container(D_Customer) in the system
  (ITDBI), if necessary
• Add your data elements (Bill_Account_ID) to the
  container (D_Customer) and map it to the Business
  Term (Business Account Identifier)