Empirical Evaluation and Comparison of Enterprise Models: A Framework and Its Application

1
Empirical Evaluation and Comparison of Enterprise
Models A Framework and Its Application

• Jean-Paul Van Belle
• Information System Dept
• jvbelle_at_commerce.uct.ac.za

2
Why research "models"?
You get interesting results!
3
Research Objective

• The development and validation of a
  comprehensive framework for the (semi-automatic)
  analysis and evaluation of (enterprise) models
• Why?
• Model-driven development paradigm
• Evaluation of method(ologie)s but not of output
  (actual model quality)
• A good Ph D topic ?

4
Comparing Enterprise Models
5
A Framework Was Needed
6
Framework 1st dimension
Classification concept Related terms and mappings Reference disciplines
Syntax Symbols, form, shape, structure Computer science, software engineering
Semantics Meaning, denotation, sense Linguistics, lexicography, information science
Pragmatics Background, situation, context Business science, management
7
Framework- 2nd dimension
Classification concept Related terms and mappings
Absolute measures Theoretical das Model an Sich the model as object objective standards intrinsic qualities technical factors Conforms to specification computer science academic.
Relative measures Applied das Model für Uns the model as subject subjective standards extrinsic qualities business factors Fit for purpose information systems practitioner.
8
The Populated Framework
Syntactic Semantic Pragmatic
Absolute Size / Specificity Correctness / error-free Integrity / Consistency Modularity / structuredness / hierarchy Genericity (universality, techn independence) Completeness (domain coverage) Expressiveness Similarity with other models Validity (authority user acceptance) Flexibility / expandibility / portability / adaptability
Relative Complexity Architectural style Perspicuity / Comprehensibility / Understandability / Self descriptiveness Documentation Purpose / goal / appropriateness / relevance Price Availability / Support
9
Theoretical Validation

• Construct efficiency (Occams razor) and
  simplicity
• Perspicuity
• Coverage and completeness
• Orthogonality
• Extensibility, customisability, robustness and
  flexibility.
• Genericity, universal applicability, portability
  and reusability
• Formality, objectivity, absoluteness.
• Theoretical foundation

10
Empirical ValidationThe Model Database

• Purpose to serve as a validating test bank for
  the analysis framework.
• Model Selection Criteria
• Domain The Generic Enterprise
• Sufficiently large size
• 200 entities 300 relationships
• Publicly available
• From very different reference disciplines

11
The Models Systems Engineering

• Reference Frameworks
• Purdue
• Nippon
• ARRI
• OO
• BOMA
• Fowler (patterns!)
• San Francisco
• ERD Libraries
• Silverston
• Hay

12
The Models Practitioners

• ERPs
• BAAN
• SAP R/3
• Real organisations
• AKMA
• NHS
• Data warehousing
• Inmon

13
The Models Various

• Ontologies
• TOVE
• CYC (subset)
• AIAI
• Miscellaneous
• Miller (systems theory)
• Ottawa (linguistic)
• Random
• Finance
• Belgian Accounting
• USB Growth Model

14
Validated SyntacticMetrics and Measures
Criterion Suggested (Validated) Metric / measures
Size CASE (concept) count and adjusted CASE count
Correctness error-free integrity consistency Syntax error, consistency and standards level score
Modularity Nr of groupers, group levels and diagrams
Structure hierarchy Multiple inheritance inheritance depth, reuse ratio
Complexity density Relative connectivity average fan-out plot of Fruchterman-Reingold (for similar-sized models) harmonic mean of fan-out fan-out distribution (chart) fan-out model signature.
Architectural style Layout aesthetics
15
Clustering (Fruchterman-Reingold)
16
Plotting the Fan-out Frequency Distribution
ACIS 2003
17
Validated Semantic Metrics and Measures
Genericity mapping to domain
Coverage Domain coverage score core concept coverage
Completeness Ranking of absolute lexicon coverage
Efficiency conciseness Relative lexicon coverage
Expressiveness Average expressiveness score
Similarity overlap with other models Plot of similarity coefficients most similar neighbours similarity dendogram most important concepts.
Perspicuity comprehensibility readability Normalized rank-adjusted weighted perspicuity count based on user lexica
Documentation Completeness, extensiveness, readability (Flesh Reading Ease)
18
Simple Semantic Analysis Common core concepts
account document organization unit
activity/process/action employee/human resource part
address environment party
agent/actor income/revenue/sales product
asset income statement sale
contract/agreement inventory/stock supplier
corporation/business item time/period
cost/expense money/cash transaction
customer/client order unit
19
Semantic analysis metrics automated procedures

• Flesch Reading Ease Score 206.835 (1.015 x
  ASL) (84.6 x ASW)
• ASL average sentence length (the number of
  words divided by the number of sentences)
• ASW average number of syllables per word (the
  number of syllables divided by the number of
  words)
• Noun synonyms generated by WordNet for the AIAI
  model word "action".Synonyms/Hypernyms (Ordered
  by frequency) of noun action 9 senses
• Sense 1 action gt act, human action,
  human activity
• Sense 2 action, activity, activeness gt
  state
• Rank Adjusted Weighted Perspicuity Count

20
Semantic Overlap
21
Semantic DistanceDendogram
22
Validated PragmaticMetrics and Measures
Validity authority user acceptance Academic author citations
Flexibility expandability adaptability Composite flexibility score
Currency maturity Descriptive table taxonomy
Purpose goal relevance appropriateness Descriptive table
Availability Medium status
Cost Purchase cost
Support Tool vendor support, user base
23
Model Quality Rankings Size Correctness Complexity Aesthetics Expressiveness Perspicuity Documentation Completeness Authority Fflexibility score Overall Ranking
AIAI 18 2 11 11 1 15 13 17 3 12 11
AKMA 15 10 16 7 9 12 9 18 13 15 17
ARRI 12 10 11 10 12 7 15 13 13 18 16
BAAN 6 7 8 11 15 2 8 3 3 3 5
BelgAcc 9 2 19 11 19 20 18 20 19 8 19
BOMA 13 2 11 3 3 7 5 10 18 3 7
CYC 1 7 10 11 3 13 12 1 3 3 4
Fowler 16 7 15 4 12 15 6 14 3 8 10
Hay 2 2 4 1 3 9 2 5 13 8 1
Inmon 3 16 20 8 16 9 18 2 7 15 15
Miller 20 16 2 11 18 19 14 19 19 20 20
NHS 8 10 17 11 11 14 11 11 7 12 13
Nippon 19 10 18 11 12 5 18 8 7 18 18
Ottawa 10 18 3 11 20 17 10 6 7 3 12
Purdue 11 19 1 9 16 4 17 12 7 17 14
SAP 7 1 6 5 7 1 16 4 2 3 2
SanFran 17 19 9 6 10 9 7 16 1 1 8
Silverston 4 2 14 2 6 2 4 9 13 1 3
TOVE 5 10 5 11 2 18 1 7 7 8 6
USB 13 10 7 11 7 5 3 15 13 14 9
24
Extending the Framework to Other Areas

• models of other domains
• web site analysis
• user interface
• algorithms
• documentation training materials
• frameworks methodologies
• software applications
• programming languages
• software architectures
• "any" intellectual work of a conceptual nature

25
Conclusion

• Framework was necessary
• Validation
• theoretically
• empirically
• independent parallel research (Chris Taylor _at_
  QUT)
• Some novel metrics esp. semantic
• Use in other domains
• Useful for composite metrics e.g. quality

26

• Selected References
• Böhm, B. et al. (1978). Characteristics of
  Software Quality. Elsevier North-Holland New
  York.
• Courtot, T. (2000). What to Look for in Packaged
  Data Models. Proceedings of the Meta Data
  Conference, Arlington.
• Edmonds, B. (1999) Syntactic Measures of
  Complexity. Doctoral Thesis, University of
  Manchester, 1999.
• Fowler, M. (1997) Analysis Patterns.
  Addison-Wesley Reading (MA).
• Claxton, J.C. and McDougall, P.A. (2000).
  Measuring the Quality of Models. The Data
  Administration Newsletter, 200014.
• Gillies, A. (1997). Software Quality Theory and
  Management. Thomson London.
• Hay, D.C. (1996) Data Model Patterns. Dorset
  House New York.
• McGabe, T.J. (1976) A Software Complexity
  Measure IEEE Trans. Software Engineering, 2 (Dec
  1976), pp. 308-320.
• Miller, J.G. (1978) Living Systems. McGraw-Hill
  New York.
• Ngo, D. Chek L., Teo, L. et al.(2000) A
  Mathematical Theory of Interface Aesthetics.
  Unpublished working paper.
• Perreault, Y. Vlasic, T. (1998) Implementing
  Baan IV. Que Indianapolis, Indiana.
• Reyns, C., Jorissen, A., Vanneste, J. (1994)
  Inleiding tot Accountancy. UFSIA Antwerp.
• Scheer, A.-W. (1998) Business Process
  Engineering. Reference Models for Industrial
  Enterprises. Springer-Verlag Berlin
• Shepperd, M. (1995) Foundations of Software
  Measurement. Prentice-Hall, London.
• Silverston, L., et al. (2001) The Data Model
  Resource Book Library of Universal Data Models
  For Enterprises. Wiley NY.
• Someya, Y. (1999). A Corpus-based Study of
  Lexical and Grammatical Features of Written
  Business English. Masters Dissertation, Dept of
  Language and Information Sciences, University of
  Tokyo.
• Uschold, M. et al..(1998) The Enterprise
  Ontology. The Knowledge Engineering Review, Vol.
  13.
• Valente, A. (1996). Towards Principled Core
  Ontologies. In Proceedings of the 10th WKNAKBS,
  Banff, Canada, 1996.

27
Questions?
28
Model Size Size Size Correctness Correctness Correctness Correctness Complexity Complexity Complexity Complexity Complexity Complexity Aesthetics Aesthetics
Model Nr of Entities CASE Size Expanded CASE Size Accuracy Consistency Standards Combined Score Cyclomatic Complexity Relative Connectivity Average Fan-Out De Marcos Data Bang AverageData Bang (Harmonic) Mean Fan-out Raw Average Normalised Score
AIAI 94 270 510 2 3 2 7 30 1.82 3.32 220 4.99 1.81 - -
AKMA 82 565 769 1 2 2 5 6 1.15 2.18 160 2.85 1.54 61 54
ARRI 128 430 790 2 2 1 5 79 2.09 3.31 592 4.97 1.81 74 75
BAAN 328 1086 1927 2 2 2 6 377 2.29 5.24 2018 8.7 2.23 - -
BelgAcc 470 1158 1158 3 3 1 7 36 1.44 2.39 599 3.37 1.41 - -
BOMA 183 552 770 3 2 2 7 65 1.68 3.00 557 4.35 1.81 61 45
CYC 777 2623 4537 2 2 2 6 511 2.32 3.60 3507 5.49 1.94 - -
Fowler 120 375 579 2 2 2 6 37 1.67 2.76 372 3.92 1.71 61 46
Hay 291 1292 3465 2 3 2 7 491 3.13 6.17 2470 10.5 2.42 53 32
Inmon 427 2429 2670 1 1 1 3 17 1.08 2.14 682 3.03 1.22 61 63
Miller 48 173 276 1 1 1 3 56 2.6 6.23 272 10.5 3.28 - -
NHS 269 751 1460 0 3 2 5 48 1.7 2.54 622 3.6 1.52 - -
Nippon 147 367 483 1 3 1 5 1 1.39 2.00 149 2.56 1.42 - -
Ottawa 248 703 945 1 1 0 2 208 1.83 3.67 1359 5.48 2.45 - -
Purdue 106 343 866 0 0 1 1 136 2.11 5.03 711 7.99 3.82 66 68
SAP 396 1218 1917 3 3 2 8 285 1.97 3.73 1851 5.64 2.30 58 49
SanFran 109 332 532 1 0 0 1 74 1.68 3.47 520 5.25 1.95 64 51
Silverston 267 1269 2235 2 3 2 7 114 1.51 3.08 950 4.55 1.76 57 41
TOVE 564 1937 2042 2 1 2 5 678 2.28 4.51 3876 7.19 2.33 - -
USB 144 531 770 2 2 1 5 121 2.56 4.02 740 6.22 2.29 - -
29
Expressiveness Expressiveness Perspicuity Perspicuity Documentation Documentation Completeness Completeness Authority Authority Flexibility Flexibility Flexibility Flexibility
Model Raw Expressiveness Weighted Expressiveness GPC NRAWPC FRE FKGL Completeness1 Completeness2 Google PRank Mod P Rank org URL Digitally available? Customizable/ reusable Implementation independence Overall flexibility score
AIAI 10.5 13.0 85 68 23.8 12.0 80 272 7 7 Yes Some Low 1.50
AKMA 8.8 10.0 94 75 34.6 12.0 74 233 5 5 No No High 1.00
ARRI 7.3 8.5 86 77 17.3 12.0 121 346 3 5 No No Med. 0.50
Baan 6.3 8.0 95 81 34.9 11.8 235 636 0 7 Yes Some Med. 2.25
BelgAcc 4.0 5.0 7 4     18 64 0 0 No Yes High 2.00
BOMA 9.7 12.0 91 77 42.3 11.7 156 452 0 4 Yes Some High 2.25
CYC 9.5 12.0 89 74 31.8 12.0 590 1143 6 7 Yes Some Med. 2.25
Fowler 6.8 8.5 88 68 41.2 11.4 100 336 0 7 No Yes High 2.00
Hay 9.3 12.0 93 76 48.0 11.1 201 574 5 5 No Yes High 2.00
Inmon 6.5 7.5 91 76     356 840 4 6 No Some Med. 1.00
Miller 6.5 6.5 68 47 23.7 12.0 44 168 0 0 No Limited Low 0.25
NHS 8.3 9.0 86 70 33.5 12.0 144 398 4 6 Yes No Med. 1.50
Nippon 7.5 8.5 90 78     171 507 0 6 No No Med. 0.50
Ottawa 3.0 2.0 85 63 33.6 12.0 221 573 6 6 Yes Limited High 2.25
Purdue 6.5 7.5 93 79 1.3 12.0 116 383 4 6 No Limited Med. 0.75
SanFran 7.7 9.5 90 76 37.7 12.0 99 310 5 9 Yes Yes High 3.00
SAP 8.8 10.5 94 82 16.9 12.0 236 632 0 8 Yes Some Med. 2.25
Silverston 8.8 11.5 95 81 45.0 11.8 141 461 0 5 Yes Yes High 3.00
TOVE 9.5 12.5 77 60 53.1 11.9 226 571 4 6 Yes Some Med. 2.00
USB 7.5 10.5 94 78 45.4 10.6 113 318 0 5 Yes Limited Low 1.25