Software Quality and Management: How the World

1
Software Quality and Management How the
Worlds Most Powerful Software Makers Do It

• Dien D. Phan
• Professor
• BCIS Department
• St. Cloud State University
• St. Cloud, MN 56301.

Information Systems Management, Vol. 18, No.1,
2001.
2
Software Quality Concepts

• Weinberg Quality is relative. What is quality to
  one person may be lack of quality to another
• IEEE Quality is the probability of failure-free
  operation of software.
• Fault is the cause of failure in software
• A failure is the departure of the external
  results of a program operation from requirements
• A fault is the defect in the program that when
  executed under particular conditions, causes one
  or more failures.

3
ISO 9001 Standards

• Applied to softw. engineering with 20
  requirements
• Management responsibility, Quality system,
  Contract review, Design control, Document and
  data control, Purchasing control, Control of
  Customers supplied products, Product
  identification and traceability, Process control,
  Inspection and Testing, Control of inspection,
  Inspection of test status, Control of non
  conforming products, Corrective and preventive
  action, Handling, storage, and delivery, Control
  of records, Internal audits, Training, Sevicing,
  and Statistical techniques.
• Organizations must establish policies and
  procedures to address each of the above
  requirements to be ISO 9001 certified.

4
Capability Maturity Integration Model (CMMI)

• Level 1 Initial
• Level 2 Repeatable. Config mgmt, Qual
  Assurance, Subcontract mgmt, Proj tracking, Proj
  planning, requirements mgt.
• Level 3 Defined. Peer rev., Coordination, Prod.
  Engineering, Integr. Mgmt, Training, Org.
  process def., Org. process focus.
• Level 4 Quantitatively Managed. Softw. Quality
  Mgmt., Quant. process mgmt.
• Level 5 Optimized. Process change mgmt,
  Technology change mgmt, Defect prevention.

5
Impacts of CMM on Software Quality

• Putnam (1994) For organizations that moved from
  level 1 to 3
• Time reduced by a factor of 1.7
• Peak staff reduced by a factor of 3.2
• Efforts reduced by a factor of 5.7
• Herbslef et al (1997)For organizations that
  moved from level 1 to 3
• 5 to 1 ROI at Hughes Aircraft
• 7.7 to 1 ROI, 75 decrease on rework, 190
  increase in productivity, and decrease in error
  rates per KLOC from 17.2 to 4.0 at Raytheon

6
Common Quality Measurements

• Mean time between failure
• Mean time to repair
• Defect rate by time
• Defect rate by size
• Defect backlog size
• Number of clean LOC passed first QA attempt
• Cumulative defects per version
• Timeliness to fix defect
• Customer satisfaction level
• Rate of defects per unit of software size
  discovered after first year of delivery

7
Quality Control Processes at IBM -OS/400
Development

• During development
• Scope control Requirements and resources control
• Reviews HiLvl Design, LoLvl Design, and Code
  reviews
• Customer involvement
• Tests and defect removal activities
• Development and quality control tools IDSS, DCR,
  PTR, and quality models
• After delivery
• Quality control for services, administration,
  and documentation.

8
Quantitative Model
9
Causes of bad fixes

• Not thorough enough testing
• Incorrect instructions to solve problems
• Not all steps tested
• Incomplete test bucket (data, programs,
  scenarios)
• Inadequate reviews
• Inability to test user hardware environment
• Insufficient testing of special installations

10
Results

• Defect rate
• Level of user satisfaction and complaints
• Numbers of service calls decreased by 50
• Cost of service calls decreased by 20-40
• System installation time reduced to 2 weeks
• Documentation errors reduced by 50

11
More Recent Studies at IBM

• Kaplan et al. (1995) Secret of Software Quality
  40 Innovations from IBM
• IBM Santa Theresa lab defects decreased by 46,
  service costs decreased by 20, revenues per
  employees increased by 58, and customer
  satisfaction increased by 14
• Mixed results on the cleanroom process

12
The Cleanroom Process

• Rather than developing a software and then
  working to remove defects, the cleanroom approach
  demands the discipline required to eliminate
  defects in specification and design then
  implement in a clean maner. The reasons that it
  is not widely used today are
• Its too theoretical, mathematical, and radical
• It abandons unit testing
• It uses statistical (randomized testing) and thus
  cannot effectively test weak spots in software
• It requires a high level of maturity in software
  development process

13
Quality Management at Microsoft

• Agile process using small teams team decides
  how the work get done
• Process Model

14
Daily (nightly) build/synchcronization and
stabilization process

• Check out source code to work -gtprivate copy
• Tool Diff to find the difference between master
  source since the time code is checked out
• Submit private copy (merging) to quick test
  before daily deadline (daily/nightly-build, daily
  synchronization and stabilization)
• Problem Breaking the build
• Solution Use Build-Master in daily-build

15
Microsofts Quality Strategies

• Grow rather than design and building software,
• Change or replace 50 of code after each release,
• Change in design must be justified by features
  that are at least twice as good,
• Build multiple version simultaneously,
• Develop software at single site,
• Continuously test the products as software grows,
  and
• Use Adaptive Software Development (ASD) process
  to build web-based software for clients

16
IBM, Microsoft Quality Capabilities measured by
SEIs CMM
Legend Y Yes - Not defined
17
Lessons learned

• Development Processes
• IBM focuses on requirements, Microsoft focuses on
  vision
• IBM processes are development oriented, less
  trade off resources, features, and schedule.
  Microsofts are customers oriented, more
  trade-offs allowed at each milestones.
• Microsoft half-life (50 replacement) reduced
  software reuse
• Microsoft teams enjoy greater freedom in the
  planning and development phases

18
Lessons learned

• CMMI
• IBM is at level 4 or 5
• Microsoft is at level 3
• Are there any Silver Bullets?

19
New Software Technology
Technology Payoff Range Comments
4GL -90 to 500 Enormous variability
CASE 9 to 128 Probably 10 on average
Formal Methods 9 Very few studies
Cleanroom 0 to 70 Results depend on interpretation
CMM 6 to 570 Wide Variability- more studies needed
Object Oriented Technology Wide range Promising technology
Agile development Unknown New development process
20
Key success factors

• Feedback, On-line monitoring,
• Project management techniques, quality control
  standards, quality processes,
• Team members are responsible of quality of
  components being built,
• Test early, and test frequently frequent
  synchronization,
• Software reuse,
• Quantitative Defect Removal Models,
• Test cases should be ready before development.
• Version Control, Configuration Management.

21
Conclusion

• IBM relies on both process and quantitative
  defect models traditional software development
  oriented
• Microsoft relies on its process model agile
  development oriented.
• Agile methodology is popular in Web projects