Beyond The CMM: WHY IMPLEMENTING THE SEI'S CAPABILITY MATURITY MODEL IS INSUFFICIENT TO DELIVER QUAL

1
Beyond The CMMWHY IMPLEMENTING THE SEI'S
CAPABILITYMATURITY MODEL IS INSUFFICIENT TO
DELIVER QUALITY INFORMATION SYSTEMS IN REAL-WORLD
CORPORATE IT ORGANIZATIONS

• Brett Champlin, MBA, CSP/CCP
• Senior Lecturer, Adjunct Faculty, MSIS MBA
  Programs, Roosevelt University
• President, Assoc. of Business Process Management
  Professionals
• Board of Directors, DAMA International
• Board of Directors, ICCP
• Worker Bee, Corporate America

Information Systems Audit and Control Association
Chicago Chapter January 8, 2004
2
Buggy Software59.5 Billion Annual Cost
3
Suppose You Were

• The Director of Application Development
• Spent 1.5 Million over 2 ½ years
• To get from Level 1 to Level 2 (or 2 to 3)
• 63 increase productivity
• But COO says 60 of time spent fixing data
• A New CIO
• Assessed at Level 3
• 3 years ago
• Current assessment suggests back to Level 1

4
The Point Is

• The CMM is a good model
• but, it isnt a silver bullet
• The CMM is not enough
• to deliver quality information services
• There are some problems with the CMM
• Is there a solution?

5
CMM History

• The Software Engineering Institute established
  1984 at Carnegie Mellon University, Pittsburgh
• Principle customer is the Department of Defense
• U.S. Air Force Project 1987
• Method to select software contractors
• Over 5,000 Assessments performed since 1987

6
The Capability Maturity Model
7
The Capability Maturity ModelProductivity Risk
Optimizing
5
Productivity
Managed
4
3
Defined
2
Risk
Repeatable
1
Initial
8
SEI CMM Assessments
SEI Assessments
9
SEI Assessments
10
Organizational Maturity Profile
August 2001 Assessment of 1018 Organizations
11
Federal Agencies
August 2001 Assessment of 14 Organizations
12
In House Development Organizationscreating
software for internal use
August 2001 Assessment of 158 Organizations
13
Military Services
August 2001 Assessment of 55 Organizations
14
Commercial Organizationscreating software for
profit
August 2001 Assessment of 531 Organizations
15
DoD Contractors
August 2001 Assessment of 212 Organizations
16
Is it worth it?
The Evidence for CMM-based Software Process
Improvement SEMA-3.01, CMU, 2001
17
Degree of SPI Success
the organization
The Evidence for CMM-based Software Process
Improvement SEMA-3.01, CMU, 2001
18
Caution

• Men sic love abstract reasoning and neat
  systematization so much that they think nothing
  of distorting the truth, closing their eyes and
  ears to contrary evidence to preserve their
  logical constructions
• - Fyodor Dostoevsky

19
What works?

• Highly Successful Efforts
• Senior Mgt actively monitors SPI progress
• Clearly stated, well understood SPI goals
• Clear, compensated assignment of responsibility
• SEPG staffed by highly respected people
• Technical staff involved in improvement

• Less Successful Efforts
• Organizational politics
• Turf guarding
• Cynicism from previous unsuccessful improvement
  efforts
• Belief that SPI gets in the way of real work
• Need more guidance on how to improve, not just
  what

The Evidence for CMM-based Software Process
Improvement SEMA-3.01, CMU, 2001
20
What works?

• Highly Successful Efforts
• Senior Mgt actively monitors SPI progress
• Clearly stated, well understood SPI goals
• Clear, compensated assignment of responsibility
• SEPG staffed by highly respected people
• Technical staff involved in improvement

• Less Successful Efforts
• Organizational politics
• Turf guarding
• Cynicism from previous unsuccessful improvement
  efforts
• Belief that SPI gets in the way of real work
• Need more guidance on how to improve, not just
  what

The Evidence for CMM-based Software Process
Improvement SEMA-3.01, CMU, 2001
21
Other Maturity Models

• 120 and counting
• Data MM
• Training MM
• Release MM
• Configuration MM
• Documentation MM
• Business Rules MM
• Etc

OtherModels
22
Other Maturity Models
By Category
23
Other Maturity Models
Percentage by Phase
24
Something Totally Different
25
Missing Pieces?

• Are all of these other models just filling in
  missing pieces?
• Isnt the core of CMM just Project Mgt?
• What about all of these other non-systems related
  Maturity Models?

26
Whats the Problem?

• Management
• Business demands
• Turnover
• Worldwide 10-15
• Slow
• 1.5 2.5 years to improve 1 Level
• New Technology
• Rate of Change is increasing
• Application Development Environments
• Portals HTML Java .Net XML-SOAP-UDDISOA
• Application Development Methodologies
• OO/UML-RUP XML-Web XP/Agile Methods

27
Managements Role
Executive Maturity Must Precede Process Maturity
From "Data Management Maturity Model" Burton G.
Parker, et. al., MITRE Software Engineering
Center, McLean, Virginia July 1995 Parker, B.,
Enterprise-wide Data Management Process Maturity
Framework, Handbook of Database Management,
Auerbach, 1999
28
Speed

• Level One to Level Two x years?
• Each Level after that 1.5 3 years?
• From One to Five, takes 6 12 years?
• Is it really that hard?
• What is the problem with taking on two or three
  level improvements at a time?
• It doesnt take that long to revamp an entire IT
  operation

29
Improvement or Re-Design?
Process Re-Design
Process Improvement
Process Re-Engineering
Time
More
Less
Change
More
Less
?
?
Value

• Process Improvement is incremental
• Process Re-Design is end-to-end re-thinking of
  what we are doing
• Process Re-Engineering is a blank slate approach

30
What about Quality?

• Hmmminformation quality problems arent always
  related to software defects
• What about the data?
• What about lousy requirements?
• What about poorly designed business processes?
• Is software development where the real problem
  lies?

31
Software or Information Quality?

• The Australian Government wanted to know how many
  illegal immigrants there were in the country.
• Matched those arriving (tourists, etc.) with
  those leaving.
• From a population of 12 million they arrived at
  400,000 estimated illegals.
• The actual number was about 100,000

32
Some More Cases to Consider

• 75 of companies report significant problems due
  to defective data
• 92 of claims Medicare paid to community health
  centers over one years time were improper or
  highly questionable
• Wrong price data in retail databases costs
  consumers as much as 2.5 billion in overcharges
  annually
• 96,000 IRS tax refund checks were returned as
  undeliverable one year

33
Cost of Poor Data Quality 600 billion!
34
Systems Dynamics

• Focusing on software development is
  sub-optimizing information and technology
  delivery
• Process integration is given lip-service at Level
  3, but
• Without integrating software development with
  data and operations management, we will continue
  to have quality problems

35
Integrated Processes
High QualityInformation Services
36
ITIL IT Infrastructure Library

• 1. Service Support (delivering and supporting IT
  services that are appropriate to the business
  requirements of the organization.)
• 2. Service Delivery (Service Level Management,
  Financial Management for IT Services, IT Service
  Continuity Management, Availability Management
  Contingency Planning and Capacity Management.)
• 3. Planning to Implement Service Management
  (the steps required to implement or improve IT
  service provision guidance on alignment of the
  business needs to IT.)
• 4. Applications Management(the software
  development life cycle, details on business
  change with the emphasis on clear requirement
  definitions and implementation to meet business
  users' needs.)
• 5. ICT Infrastructure Management (network
  service management, operations management ,
  management of local processors, computer
  installation and acceptance and systems
  management.)
• 6. Security Management (how to organize and
  maintain the management of security of the IT
  Infrastructure, from the IT manager's point of
  view.)
• 7. The Business Perspective(business
  relationship management, partnerships and
  outsourcing, continuous improvement, exploitation
  of information, communication, and technology
  (ICT) for business advantage.)

37
Why Isnt CMM Enough?

• What is the real problem/product?
• Sub-optimizes Information Services Quality
• Must be coordinated with Data Mgt, Operations
  Mgt, etc.
• Too slow
• Redesign or Reengineer the process
• Needs Management Maturity to sustain
• IT Quality programs must be aligned to Business
  Needs

38
What Should We Do?

• CMM is a really good model for Software Quality
  Improvement
• Apply a generic process maturity model to all IS
  processes
• Take a redesign approach to all IS processes
  integrated, from the top - down
• Train all IT managers in quality management
  principles
• Make sure you identify where your real problems
  are before you start fixing things
• Align your improvement efforts to your business
  needs

39
Thank You

• I appreciate your comments and will be pleased to
  answer any questions that you may have.You may
  contact me for further discussion at

Brett Champlin Email brett_at_thechamplins.comPH
847/667-1747Visit my faculty websitehttp//fac
ulty.roosevelt.edu/Champlin/
40
Select References/Sources

• The Electronic College of Process Innovation,
  DoD, http//www.c3i.osd.mil/org/bpr.html
• The Evidence for CMM-based Software Process
  Improvement SEMA-3.01, CMU, 2001
• The IT Infrastructure Library, http//www.itil.co.
  uk/
• Manager Heal Thyself Improving Software
  Processes means Changing Management Processes,
  Derby Rothman, Cutter IT Journal, Oct 2001,
  Vol. 14, No. 10
• Managing the Software Process, Watts Humphrey,
  1989, Addison-Wesley
• Maturity Alone Is Meaningless SPI Reality from
  Industry, Rodenback, Lautum, Solingen, Cutter IT
  Journal, Oct 2001, Vol. 14, No. 10
• Out of Alignment, Paul Strassman,
  Computerworld, March 4, 2001
• Quality Is Free The Art of Making Quality
  Certain, Philip Crosby, 1979, McGraw-Hill
• Software Engineering Information Repository,
  http//seir.sei.cmu.edu/
• Software Process Improvement Is That What We All
  Should Be Doing Again?, Kendall, Cutter IT
  Journal, Oct 2001, Vol. 14, No. 10
• The Value of Software Process Improvement,
  James Douglass, SEI-CMU, Oct 2001
• Worldwide IT Trends Benchmark Report 2002, Meta
  Group

41
Not the CMM

• Level 1 InitialThe software process is ad hoc,
  and occasionally even chaotic. Few processes are
  defined, and success depends on individual effort
  and heroics. Frequent late nights and hollow,
  sunken eyes are common. Programmers at each
  others' throats. Managers mostly very angry.
• Level 2 RepeatableShould a similar project be
  run, it would probably be just as chaotic. Team
  leaders have slim control over the programmers.
  Project manager has installed MS Project, and
  printed out reams upon reams of Gantt charts
  which have already started to block doorways and
  commonly used pathways, e.g. from "zoo" area to
  kitchen. More late nights than not. Programmers
  generally civil to each other, but often murmur
  discontentedly behind each others' backs.
  Managers have bouts of rage during progress
  update meetings.
• Level 3 DefinedThe software process is
  documented, standardized, and integrated into a
  standard process for the organization. Managers
  express keenness to use the process for any new
  projects, as soon as all the current projects are
  finished. Programmers spend more time filling out
  forms than writing software.
• Level 4 ManagedAccurate metrics are collected
  for each project, as and when it fails. Detailed
  "post-mortems" explain why the organization's
  standard process was not used yet again.
  Programmers spend large proportion of day
  updating CV and hanging around the popular job
  sites.
• Level 5 OptimizingA miracle worker was passing
  by one night. Programmers angry because most of
  the jobs out there are made-up.

Source http//www.bad-managers.com/rumours/cmm_le
vel_one.shtml
42
Other Maturity Models

• Automated Software Testing MM Krause
• Business Processes Dr. Jaques Hale', 1994
• Business Rules Barbara Von Halle, Apr '96
• CASE Tool Selection S.L. Pfleeger, November
  1991
• Change Proficiency MM Paradigm Shift
  International
• Competence (Data Warehouse analyst competence) MM
  Brohman Parent, 2001
• Compliance MM Rushmere Consulting, Inc.
• Content Management MM Steelpoint Technologies,
  2002
• Course Website Development MM Fred Beshears,
  Univ of California, Berkeley
• Corrective Maintenance MM Kajko-Mattson
• Customer Intimacy MM, Ken Rudin, Keane Inc.
• Customer Relationship Management (CRM) MM EDS
• Data Management MM MITRE, 1994-96

Data Resource Management MM Champlin, 1996 Data
Warehousing MM Marco, 2002 Documentation
Management Dr. Marcello Visconte
(Chile) e-Business MM - Berkeley Enterprise
Partners EBusiness MM Gardler, 2000 E-Commerce
MM Ludescher, Vienna University of Technology
and Usrey, University of Colorado Earned Value
Management MM Ray Stratton, Management
Technologies, Enterprise Application Integration
MM Schmidt Seidel, AMS, 2001 Enterprise-wide
Data Management Burton Parker ERP Systems MM
Holland FAA-iCMM, FAAs Integrated CMM Federal
Aviation Administration Facilities Renovation MM
APPA
43
Other Maturity Models

• Knowledge MMM Informatie Management Nederland
  (IMN)
• Knowledge Management MM Gallagher, Queens Univ
  of Belfast,
• Knowledge Management MM - Langen, 2000
• Learning Organization MM Lockheed Austin
• Maintenance MM AMS
• Management Information Systems MM Ernst Young
• Measurement Program MM Donnellan Peterson
• Operations-Based for Problem Management MM - Meta
  Group, 2001
• Organizational MM Rosenberg
• Organizational Interoperability MM Clark
  Jones, 1998
• Organizational Project Management MM - PMI
• People Management MM SEI
• Performance Engineering MM Scholz and
  Schmietendorf, 2000
• Portal MM Tanning Technology Corporation
• Process Improvement for Construction Enterprises
  MM - (SPICE) 1998-2000

• Function Point MM Emmons, 2000
• General Practice MM - Gillies
• Government Technology Maturity Model for IT
  Investment and Project Management GAO, 2002
• Human Factored Interface MM J. Earthy
• Information Evolution Model Hatcher, 2002
• Information Process MM Hackos, 1994
• Inspections CM Bob Cerady, HP
• Integrated Product Development MM EPIC/SECAT,
  1997
• Integration for the Digital Enterprise MM
  Paydarfar, 2001
• Internet Agility MM Brian Neymeh, INSTEP Inc.
  1995
• Internet Development (Network Maturity) MM IEEE
• Internet Management MM CMU
• IT Architecture MM US Department of Commerce
• IT/Business Alignment Assessment MM Luftman
  (Harvard)
• IT Governance MM CobiT
• IT Service Capability MM Frank Niessink

44
Other Maturity Models

• Product Development MM Crow, DRM Associates,
  2000
• Programme Management MM Rayner Reiss, The
  Program Management Group, 2000
• Programme Management and System Engineering MM
  European Software Institute
• Project Management MM Cadence Management
  Corporation, 2001
• Project Management MM Enterprise Planning
  Associates, 1998
• Project Management MM IBBS Consulting
• Project Management MM Kerzner, 2001
• Project Management MM Levin, Hill, DeFillipis,
  Ward, Shaltry, Richards/ESI
• Project Management MM PM Solutions, 2001
• Project Management MM Robertson, KLR
  Consulting, 2001
• Project Management MM Wisdm Corp., 2001
• Public Works MM UK
• Security Management MM NSA sponsored
• Service MM Quickarrow.com
• Software Change Management MM - Bendix, 1998

• Software Contracting MM August Automation, Inc.
• Software Maintenance MM MITRE
• Software Reuse MM Davis (SPC)
• Software Systems Documentation Process Oregon
  State University, 1992
• Strategic Management MM McWeeny Ellinger
• System MM - (schools use of IT, based on Nolans
  Stages Model)
• System Acquisition MM Ferguson, MITRE SEI
• System Administration MM Kubicki, 1993
• System Engineering MM Malpass, SEI
• Systems Requirements MM REAIMS, 1994-97
• System Security MM Hefner
• Systems Security Engineering MM Electronic
  Warfare Associates (EWA) Information and
  Infrastructure Group
• Technology MM Edmin.com, 1998
• Testability MM Gelperin, 199
• Testing Bernstein, Suwannasart, and Carlson,
  IIT, 1996

45
Other Maturity Models

• Testing MM Burgess Drabick, 1996
• Total data Quality Management MM - MIT
• Training MM MITRE
• Trusted Software MM Kitson
• Usability/Human Centredness MM Earthy, 1998
• Vulnerability Management security Online, 2001
• Risk Maturity Model, HVR Consulting Services,
  Ltd.
• Risk Maturity Model, Basil Orsini , Human
  Resources Development Canada
• eServices Capability Model (eSCM), CMU
• CMM for Inventory Effectiveness , Jeff Kavanaugh,
  Inforte Corp.
• Data MM, Jeremey Janzen, British Columbia
  Ministry of Forests
• Data Warehouse Information Management MM, John
  Ladley

Data Quality MMM, Meta Group Enterprise
Architecture MM, Meta Group Enterprise Program
Project Management MM, Meta Group Information
Management MM, Meta Group Infrastructure MM, Meta
Group Internal Consulting MM (IT Operations
Services) , Meta Group IT Operational Process MM,
Meta Group IT Performance and Measurement
(Organizational MM), Meta Group IT Procurement
MMM, Meta Group Measurement MM, Meta
Group Outsourcing MMM, Meta Group Process MMM,
Meta Group Telecom MMM, Meta Group
Back