Data Modeling A Retrospective

1
Data Modeling A Retrospective

• DAMA COC
• April 2002

2
The Early Years - Practice

• Data as a by-product of function (W. Inmon)
• No data modeling as we know it
• Data requirements defined by interfaces (user and
  system)
• Data storage driven by screens, interfaces and
  physical DB (hierarchical DB or flat files).
• Massive, paper-based methodologies
• Text-based requirement definition
• Monolithic, repetitive, ambiguous, fixed

3
The Early Years Leading Edge

• Object Techniques
• Simula I (Nygaard Dahl, 1965)
• Used initially for simulation
• No general acceptance in industry
• Data Theory
• Normalization (E. F. Codd, 1970)
• Relational data base concepts developed
• Oracle (1977), IBM System/R (1978), SQL (1982)
• Bottom-up approach
• Entity Relationship Diagrams
• Finkelstein, Martin
• Top-down approach

4
The Early Years Leading Edge

• Classical Structured Analysis
• Functional models are primary focus
• Data Flow Diagrams
• Data still in context of function
• Emphasis on current state models
• Little distinction between logical and
  physical
• Data Structure Diagrams (DeMarco, Gane and
  Sarson)
• No capability to model temporal dimension
• Emphasis on conversion of an arbitrary set of
  data structures into third normal form (Yourdon)

5
The Early Years Leading Edge

• Modern Structured Analysis adds
• Entity Relationship Diagrams (ERD) for static
  data modeling
• State transition diagrams for time dependences
• Event partitioning for middle-out instead of
  strictly top-down function modeling (McMenamin
  Palmer, 1984)
• Distinction between logical (or conceptual or
  essential) models and physical models

6
The Early Years Leading Edge

• Structured Programming and Design
• Started at code level (programming)
• Edsgar Dijkstra (1968)
• Eventually expanded to structured design
• Sought to make code more maintainable and promote
  code reuse (encapsulation)
• Parallel initiative to Structured Analysis, but
  independent
• Did not address data modeling

7
The 1980s - Practice

• Cut and Paste data design using 1970s data
  structure common
• Little change in many organizations
• Focus on large systems, complicated interfaces,
  and reporting (not data)
• Static data models using ERDs and Codds
  principles begin to be used
• Structured Analysis more widely accepted, merged
  with Structured Design

8
The 1980s Leading Edge

• Methodologies
• Comprehensive, deliverables-based
• Move from paper forms to CASE tools
• Structured Analysis and Design is basis
• Integrates graphical models into deliverables
• Many similar methodologies
• Information Engineering (Finkelstein Martin)
• Dependent on CASE tools

9
The 1980s Leading Edge

• CASE Tools
• Closely allied to a methodology
• Mechanized creation of graphical models
• Automated some model quality checks
• Segmented the development cycle
• Upper Lower and Full CASE tools
• COBOL code generation
• Major tools implement Information Engineering
• ADW (KnowledgeWare), IEF (Texas Instruments)
• James Martin a major influence

10
The 1980s Leading Edge

• Prototyping
• Mock up of physical implementation
• Usually throw-away since tools differed from
  production system tools
• Extension of earlier practice of designing system
  from user interface
• Often used as short-cut from requirements to
  design and code
• Object-Orientation
• C (1984)

11
The 1990s - Practice

• Information Engineering accepted
• Deliverables-based methodology
• Relational data modeling
• CASE tools used with varying degrees of success
  Shelfware
• CASE tools become more specialized
• Microsoft Office as a CASE tool
• Throwback to 1970s techniques
• Y2K reinforces status quo

12
The 1990s Leading Edge

• RAD (Rapid Application Development)
• Originally applied to development once
  requirements were known
• Timebox development
• Requirements from workshops or focus groups (JAD)
• Abbreviated project lifecycle
• Prototyping to identify features requirements,
  demonstrate something tangible
• Prototype becomes the system
• Little or no design
• Reuse of software components

13
The 1990s Leading Edge

• Object Orientation
• CASE tools provided major impetus
• OOD, OOP and OO DBMS
• OOA
• Unified data and function
• Extended OOD into analysis space
• Class models, state transition models, sequence
  diagrams
• UML (Gooch, Rumbaugh, Jacobson)
• Unification of multiple OO approaches
• Major step in gaining acceptance of OO
• Originated in design and coding, weak in business
  modeling
• Java (1995)

14
The 1990s Leading Edge

• Data Warehouse
• "a copy of transaction data specifically
  structured for query and analysis (Kimball)
• First major data-centered initiative
• New data modeling techniques
• Dimensional models, star joins (Kimball)
• Arose from prior efforts for reporting and
  Management Information
• Emphasis on data transformation
• Data quality becomes important
• Metadata important for information retrieval

15
The 1990s Leading Edge

• Frameworks and Patterns
• Applicable to different parts of lifecycle
• Generic industry models (data and function)
• Models for specific subject areas, like Party
• Models for a particular use of data, e.g. Data
  Warehousing
• Design patterns in O-O, followed by analysis and
  organizational patterns and frameworks
• Framework is a set of cooperating classes that
  make up a reusable design for a specific class of
  software
• Zachmann Framework for entire development process

16
The 1990s Leading Edge

• Business Rules
• Attempt to isolate most frequently-changing
  aspect of a business
• Isolation of rules (as data) from code
• Requires definition of business terms (data)
• Code becomes a rules engine
• Business rules-based methodology (Ross)
• Close integration of pioneers to Zachmann
• Emphasis on business data
• Extends concept of what data is

17
The 1990s Leading Edge

• Middleware
• Alternative to application rewrite
• Emphasis on data transformation
• Simplify complicated interfaces by
• Collecting transformation rules into one place
• Standardize data interfaces
• Use of industry standard data interchange
  specification (EDI)
• Drives need for shared models

18
The 1990s Leading Edge

• Web
• Emphasis on data access and movement
• Separation of data from presentation and logic
• HTTP and XML for data transmission and storage
• Data modeling extended to include storage of
  transitory data (click-stream analysis)
• Portals for access to information (data in
  context)
• Metadata is important for finding information

19
The 1990s Leading Edge

• Metadata
• Any data used to aid the identification,
  description and location of networked electronic
  resources
• Models of models (of models)
• Provides context for data
• Defines data (business terms)
• Emphasis on business meaning of data
• Repositories to store metadata

20
Where are we?

• Application Development
• Traditional Mainframe, client-server,
  relational
• O-O, RAD, O-O Patterns and Frameworks
• Web (HTML, XML, components)
• Middleware to access legacy data
• Data Warehouse/Business Intelligence
• Methodology and Techniques
• UML and Unified Method
• Business Rules Methodology
• Generic Models
• Zachmann Framework

21
Where are we in Data Modeling?

• Relational data modeling
• Object modeling
• Dimensional modeling
• XML modeling
• Metadata
• Business rules
• Generic models (industry subject area) and
  standards
• Data transformation and transmission

22
Where do we need to be?

• Cheaper, faster, better
• Dont start with a blank page
• Reference models
• Metadata
• Consistent view of data across different modeling
  techniques and applications
• Understanding of relationship of data and
  function
• Flexible, integrated approach to data

23
References

• Yourdon, Edward, Modern Structured Analysis,
  Yourdon Press, 1989
• Yourdon, Edward, Just Enough Structured
  Analysis, www.yourdon.com/books, 2001
• Cockburn, Alistair, Software Development as
  Community Poetry Writing, talk given to Central
  Ohio Chapter of the ACM, 1997 from
  www.members.aol.com/acockburn/papers/swaspoem/swpo
  .htm

24
References

• Powell, Ron, A 10 Year Journey, DM Review, Feb.
  2001
• Inmon, Bill, Information Management Charting
  the Course A Brief History of Database Design,
  DM Review, April 2001
• Appleton, Brad, Patterns and Software Essential
  Concepts and Terminology, http//www.enteract.com
  /bradapp/

25
References

• http//ifla.inist.fr/II/metadata.htm
  (International Affiliation of Library
  Associations and Institutions IFLANET Digital
  Libraries Metadata Resources)

26
Presenter Information

• Background
• Application development
• Process and data modeling
• Use of generic industry business models
• Data transformation
• Contact Information
• Phone [phone]
• eMail jonesdf_at_mindspring.com