Title: Final Findings Presentation
1 NASA Langley Research Center Software Process
Improvement Initiative (SPII) Findings
Presentation October 27, 1997
2CornerStone Findings Presentation
- CornerStone Overview
- Findings
- Next Steps
3CornerStone Overview
4CornerStone Goals
- The CornerStone Phase is the foundation of LaRCs
overall Software Process Improvement Initiative - The goals of the CornerStone Phase are
- Develop a plan to improve LaRCs software
development practices - Identify current state of software development at
LaRC - Identify current best practices used in software
development at LaRC - Develop a High Performance Model for LaRCs
software development activities (incorporates the
appropriate elements of the Capability Maturity
Model, ISO 9000-3, Strategic and Quality
Framework, and Baldrige Award Criteria) - Obtain managements support, complete with
resources, to implement LaRCs Software Process
Improvement Plan
5Software Process Improvement Initiative (SPII)
Goals
- Improve the work environment for LaRCs software
community, leading to higher morale and increased
productivity - Develop sustainable mechanisms for continuous
improvement in the productivity and quality of
software developed across LaRC - Increase customer satisfaction with LaRC software
products
6CornerStone Activities
- Lay the Foundation
- Establish Infrastructure
- Plan for the Assessment
- CornerStone Planning and Validation
- Baseline
- Workshop Training
- Customer Workshops
- Supplier Workshops
- Follow-Up Interviews (as needed)
- Best Practices Documentation
- Analyze Workshop Results
- Prepare and Present Findings
- Plan
- Define LaRC SEPG
- Develop SPI Plan
- Review SPI Plan with Sponsors
- Present SPI Plan
- Implement Improvements
7CornerStone Team Members
- Norma Campbell, RTG/FDCD
- Victoria Chung, IOG/ISSD
- Michael Holloway, RTG/FETD
- Chuck Niles, IOG/FSED
- Pam Rinsland, IOG/AESD
- Pat Schuler, IOG/ISSD
- Jim Townsend, RTG/FMAD
- Jim Watson, OSEMA/OMA
- Sue Voigt, SASPG/SSCD
- Consultant Cindy Torpey, ChangeBridge Inc.
8CornerStone Scope
- Centerwide involvement
- Organizations involved in software management,
development, and maintenance - Customers of software projects
- 101 civil servants and contractors interviewed
9CornerStone Principles
- Start with a process framework
- Baseline current state (not audit)
- Conduct interviews and discussions
- Observe strict confidentiality
- Involve senior management
- Work as a Centerwide team
- Focus on LaRC needs
10Capability Maturity Model
11Capability Maturity Model
- Repeatable / Level 2 (All Key Process Areas
covered in interviews ) - Requirements Management
- Software Project Planning
- Software Project Tracking Oversight
- Software Quality Assurance
- Software Subcontractor Management
- Software Configuration Management
- Defined Level (Selected Key Process covered in
interviews) - Training Program
- Software Product Engineering
- Intergroup Coordination
- Peer Reviews
12CornerStone Approach.
- Baseline current state (snapshot in time)
- some improvements are already initiated
- Baseline included all areas which impact the
software development group - some are under our control, others will require
a coordinated effort - Not all points are at the same detail
- some are very specific, other are more general
- Processes were assessed, not specific groups
- especially when we discuss SQA and Training
Program - Identify areas for improvement
- not specify how to
13CornerStone Findings
- Key Process Area
- Purpose and Description
- Best Practices
- Improvement Opportunities
14Requirements Management
- Purpose
- Establish a common understanding of the
customers requirements that will be addressed by
the software project - Description
- Establish and maintain an agreement with the
customer on the requirements for the software
project - The "customer" is a system engineering group,
TAG, a project office, another internal
organization, or an external customer - Agreement covers both the technical and
nontechnical (e.g., delivery dates) requirements
- Agreement forms the basis for estimating,
planning, performing, and tracking the software
project's activities throughout the software life
cycle
15Requirements Management
- Best Practices
- Requirements are prioritized and managed
according to priority - Developer works with customer to interactively
draw out requirements - Operational Concepts Documents help in
understanding the requirements (some projects put
on the web for increased visibility) - Test cases traceable back to requirements
- Explicit contractor task to capture and manage
requirements - Rapid prototyping to validate requirements with
customer
16Requirements Management
- Improvement Opportunities
- Importance of requirements is not realized
- Requirements are neither defined early enough,
nor refined far enough - Inadequate resources dedicated to requirements
management - Software requirements do not track to system
requirements - Dont know how to adequate document requirements
- Customers are not trained in requirements
generation - Changes in requirements handled by overtime masks
over-commitment - Requirements creep is not managed nor controlled
(cost is not understood) - No established policy to guide projects in
requirements management - Role and responsibility of systems analysis is
not clearly understood - Requirements are not consistently documented and
reviewed with the customers - Segmentation of responsibilities between
researchers and software developers leads to
uncertainty in product functionality
17Requirements Management
- Consequences
- Project schedule and cost are significantly
increased when requirements are inadequately
defined and documented - Rework is common due to changing requirements
- Customer satisfaction is decreased when their
requirements are not met
18Software Project Planning
- Purpose
- Establish reasonable plans for performing
software engineering and managing the software
project - Description
- Develop estimates for the work to be performed,
establish the necessary commitments, and define
the plan to perform the work
19Software Project Planning
- Best Practices
- Project cost, effort and duration estimates are
developed and tracked - Software Work Breakdown Structure is generated at
the same level as hardware WBS - Scheduling tools (MS Project, REVIC) are used
- Statements of Work specifies creation of project
plans - Software staged release at planned milestones
- Software manager designated to oversee a software
development project - Software personnel participate in project plans
- SQA involved in project planning
20Software Project Planning
- Improvement Opportunities
- Schedules are driven by externally set
milestones, instead of actual work required - Software Management Plans are not consistently
documented, if at all - No planning metrics are captured across the
organization on which to base realistic future
estimates - Commitments are not honestly negotiated
- Inadequate project planning is compensated for by
overtime - Over-commitment
- No LaRC project policy for managing software
projects - the approach to software project
management is project dependent - The value of software project plans is not fully
understood by developers, managers, or
researchers - Personnel are not aware of available procedures
and guidelines for software estimating - No guidelines are available for document software
development plans
21Software Project Planning
- Consequences
- Software projects do not meet schedules or
budgets (could face cancelation) - Burn out civil servants and contractors
22Software Project Tracking and Oversight
- Purpose
- Provide adequate visibility into actual progress
so that management can take effective actions
when the software project's performance deviates
significantly from the software plans - Description
- Track and review the software accomplishments and
results against documented estimates,
commitments, and plans - Adjust plans based on the actual accomplishments
and results
23Software Project Tracking and Oversight
- Best Practices
- Informal interchange meetings are held frequently
for tracking - Automated tools (MS Project, PC Artemis) are used
to track project progress - Milestones used as checklist
- Formal reviews held at selected milestones
- Dedicated business manager to track software
project status
24Software Project Tracking and Oversight
- Improvement Opportunities
- Corrective actions are not taken in a timely
manner - Software costs are not accurately reflected in
project charge structures - No measurements or lessons learned are captured
and used for future projects - Software size is not estimated or tracked
- Managers are not trained in software project
management and tracking - Technical issues are not managed and communicated
- Software Development Plans are not used to manage
the project - Software project reviews are ad hoc and
ineffective results are not elevated to
management (guidelines needed)
25Software Project Tracking and Oversight
- Consequences
- Products not completed on expected dates
- The real software project status is not known
until it is too late to do anything about it
26Software Subcontractor Management
- Purpose
- Select qualified software subcontractors and
manage them effectively - Description
- Select a software subcontractor
- Establish commitments with the subcontractor
- Track and review the subcontractor's performance
and results
27Software Subcontractor Management
- Best Practices
- Periodic reviews are conducted with contractors
to review progress and communicate status - COTRs and Technical Monitors are designated to
establish and manage the software contract - Source Evaluation Boards select contractors based
on their ability to perform the work - Success has been demonstrated using Performance
Based Contracting - Formal quarterly evaluations are required
- Software services are acquired using standard
NASA contracting regulations
28Software Subcontractor Management
- Improvement Opportunities
- No accountability for poor contractor performance
- COTRs and Technical Monitors are not trained in
software engineering or managing software
development efforts - Performance Based Contracting is misunderstood,
and due to poor training it is ineffectively
applied - Source Evaluation Board frequently recommends
lowest bidder rather than best qualified - Required reviews are not consistently conducted
- Contractor turnover is high and requires frequent
re-orientation resulting in increased costs - No written LaRC policy or guidelines for managing
software contracts - Contractors over-commit and rely on free
overtime - Skill level and training on the contract does not
match what is required
29Software Subcontractor Management
- Consequences
- Frequent dissatisfaction with contractors
products - PBC task generation is more time-consuming
30Software Quality Assurance
- Purpose
- Provide management with appropriate visibility
into the process being used by the software
project and of the products being built - Description
- Review and audit the software products and
activities to verify that they comply with the
applicable procedures and standards - Provide the software project and other
appropriate managers with the results of these
reviews and audits
31Software Quality Assurance
- Best Practices
- SQA contractors advise and give consultation
to projects on software engineering when
requested (on configuration management and
software management plan) - Infractions are reported to participating project
teams
32Software Quality Assurance
- Improvement Opportunities
- Need sufficient staff for uniform coverage of SQA
on Langley software projects - Increase awareness of added value of SQA
practices - Appropriate guidelines for SQA activities
(current LHB not widely accepted) - Review SQA activities on a regular basis
- Track cost and associated return on investments
on SQA tasks
33Software Configuration Management
- Purpose
- Establish and maintain the integrity of the
products of the software project throughout the
project's software life cycle - Description
- Identify the configuration of the software (i.e.,
selected software work products and their
descriptions) at given points in time - Systematically control changes to the
configuration - Maintaining the integrity and traceability of the
configuration throughout the software life cycle - Placed under software configuration management
both the software products delivered to the
customer (e.g., the software requirements
document and the code) and the items identified
with or required to create these software
products (e.g., the compiler)
34Software Configuration Management
- Best Practices
- Configuration Control Boards are established and
used to control changes, assess impacts and
communicate status - Automated tools (ClearCase, PVCS, RCS, LibLink,
CVS, LabView) are used to control software work
products (requirements, change rationale, code
and supporting documentation) - Intent of SCM can be met without the use of
automated tools for projects - Web-based change request system established
(ClearDDTS) - SCM process documented and followed on some
projects
35Software Configuration Management
- Improvement Opportunities
- Rigor of formal SCM is not always appropriate for
projects size, phase and purpose - Change request status is not adequately
communicated across all project personnel - Lack of awareness of existing guidelines and
templates results in significant duplication of
effort - Insufficient funding for SCM staff, tools and
training - SCM typically applied too late in project phase
- SCM plans are not documented and/or followed and
changes are not controlled - Lack of understanding of the benefits of SCM
- Contracts do not always require the appropriate
level of SCM and there is inadequate CM on
deliverables
36Software Configuration Management
- Consequences
- Lack of SCM results in compromised mission
certainty and validity of data and products
37Training Program
- Purpose
- Develop the skills and knowledge of individuals
so they can perform their roles effectively and
efficiently - Description
- Identify the training needed by the organization,
projects, and individuals - Develop or procure training to address the
identified needs - Evaluate current and future skill needs and
determines how these skills will be obtained - Use informal vehicles when appropriate (e.g.,
on-the-job training and informal mentoring)
38Training Program
- Best Practices
- Training Office exists to meet the needs of the
LaRC including the software development community - Some projects and organizations successfully
leverage the capabilities of the Training Office
to meet their needs - Training Office annually surveys LaRC to
establish training needs
39Training Program
- Improvement Opportunities
- Increase is needed in training budget due to
cross training and retraining of LaRC personnel - Inconsistent management support for training
- Insufficient time is allocated for attending
training - Training is not considered a priority
- Need full implementation of Individual
Development Plan (IDP) which ties to the SQF and
the agency strategic plan - No effective mechanism for consistently training
contractors and civil servants working on the
same project - No project training plan
- Travel funds shortage limits training
opportunities and technical exchange
40Software Product Engineering
- Purpose
- Consistently perform a well-defined engineering
process that integrates all the software
engineering activities to produce correct,
consistent software products effectively and
efficiently - Description
- Perform the engineering tasks to build and
maintain the software using the project's defined
software process and appropriate methods and
tools
41Software Product Engineering
- Best Practices
- Operational Concept, Users Manual and Test
Plans/Cases developed prior to code - Apply techniques such as Object Oriented Design
(OOD), reverse engineering, structured
programming, and modularity to reduce code
complexity, increase reusability and improve
maintainability - Automated software development tools (Purify,
Quantify, PureCoverage, Rational Rose, ClearCase,
ClearDDTS, Object Manual, LabView GUI) are used
to design, test, CM, and document software - Provide realistic operational testing scenario
for software - Office automation tools used in innovative ways
for software development, testing and
documentation (databases, spreadsheets)
42Software Product Engineering
- Best Practices (cont.)
- A project has demonstrated LaRCs ability to meet
FAA standards for airworthy flight software - On line Web sites exist that describe software
engineering guidebooks, formal methods and
metrics collection database - Third party tests against requirements agreed to
by users - Continuous rapid prototyping used to demonstrate
feasibility and early progress - Requirements gathering techniques resulted in
improved software product - Centralized facility provides access, guidance
and expertise for domain-specific tools
43Software Product Engineering
- Improvement Opportunities
- Insufficient time allocated for software
engineering tasks (requirements definition,
design, testing, integration, configuration
management, and documentation) - Software engineering activities often hidden
because their value is not recognized, perceived
as overhead by projects, and researchers/projects
do not want to pay for reuse (no corporate view
for long-term investment) - No rewards for good software engineering
- Single point failures on many projects created by
one person and insufficient documentation - Testing address physics only, not software
quality - No LaRC dissemination guidelines for software
exist
44Software Product Engineering
- Improvement Opportunities (cont.)
- Lack of in-house software product engineering
expertise - Need point of contact where software engineering
expertise and guidance can be obtained - Ineffective tool utilization due to poor
communication and awareness - Web sites not advertised
- Software engineering techniques not appropriately
tailored for small projects - No incentive to take research code to production
quality or make it reusable by other projects - Lack of expertise in project planning and
scheduling leads to insufficient time for
fundamental software engineering tasks
45Software Product Maintenance
- Best Practices
- Use portable computer to set up, perform
experimental test, and analyze results - Improvement Opportunities
- Insufficient funding for sustaining and
maintenance of software - Ineffective implementation of CM prohibits quick
minor changes to software for experimental use - Insufficient verification of software prior to
delivery leads to high maintenance cost - Due to insufficient manpower and maintenance
procedures, staff is not adequately notified on
software changes - Constant changes in COTS upgrade decrease
productivity due to perpetual learning curve
46Intergroup Coordination
- Purpose
- Establish means for the software engineering
group to participate actively with the other
engineering groups so the project is better able
to satisfy the customer's needs - Description
- Software engineering group participates with
other project engineering groups to address
system-level requirements, objectives, and issues - Representatives of the project's engineering
groups participate in establishing the
system-level requirements, objectives, and plans
by working with the customer and end users, as
appropriate - Requirements, objectives, and plans become the
basis for all engineering activities
47Intergroup Coordination
- Best Practices
- Workshops and shared facilities bring different
groups together for improved information exchange - Interface Control Documents, frequent meetings
and timely meeting notes ensure effective
exchange of information - Web-based information and email exchange
facilitates technical coordination - Physical co-location of discipline specialists
promotes intergroup coordination during a project - Teamwork training is available
- Communication between software programs used by
different groups achieved by in-house developed
interfaces (scripts, GUIs, wrappers, standard
features, filters)
48Intergroup Coordination
- Improvement Opportunities
- Need to have representatives of all technical
disciplines (including software) involved from
the start of a project (requirements, cost
estimates, operational concepts, requirements
allocation) - Due to the lack of systems engineering performed
at LaRC, software and hardware staff are forced
to fill that role in an ad-hoc manner - Poor communication among groups doing similar
work within LaRC results in duplication of work - No effective mechanism to ensure all disciplines
are represented on project teams - Software specialists not aware of overall project
concept - Perception that software can fix anything,
including poor hardware selection, leads wasted
funds and staff hours - Lack of documented commitments between
engineering groups - Changes are not effectively communicated across
the engineering groups - Contractors resist communication due to
proprietary fear
49Peer Reviews
- Purpose
- Remove defects from software products early in
the development process where it is more cost
effective to remove them - Develop better understanding of software products
and defects that might be prevented - Description
- Peers methodically examine software work products
to identify defects and areas where changes are
needed - Identify and schedule specific products that will
undergo peer review as part of the defined
software process
50Peer Reviews
- Best Practices
- Peer reviews identify problems early in the
lifecycle resulting in saved time and decreased
costs - Post-mortem review is effective mechanism to
capture lessons learned - Informal review by an in-group peer works well on
small projects - Peer reviews captures problems not otherwise
identified - Completion of peer reviews is a phase exit
requirement and indicates readiness to proceed - Peer reviews are effective mechanism to train
staff and indoctrinate new hires - Peer review process is documented (NASA Formal
Inspection Handbook, project documentation) - User requirements are basis for code reviews
- Email and Key Activities are used to document
peer review results
51Peer Reviews
- Improvement Opportunities
- Peer reviews are not commonly used (even unknown
by some projects) in spite of positive return on
investment experienced at LaRC - Current review process frequently omits software
issues - Current formal design review system (PDR, CDR)
often misses large problems - Post-mortem reviews hardly ever held
- Limited time, training, and staff are provided
for performing peer reviews - No data collected on peer review results and
lessons learned (dormant data base)
52Non-CMM Related Concerns
- Improper ISO 9000 implementation could impose
restrictive procedures on research and
significantly impact resources to perform future
research - Numerous management issues were identified such
as - The lack of management awareness of the
pervasiveness and magnitude of work required to
develop software at LaRC - Need investment, encouragement, and reward system
for improvements, best practices, positive
technical transfer - Lack of adequate resources for performing
software development - There is a lack of a central pool of software
developers to service LaRC - Relation of LaRCs Strategic Quality Framework to
real work is poorly defined
53Next Steps
54Critical Point in LaRCs SPI Program
- CornerStone phase is a critical milestone in the
software process improvement program - CornerStone phase baselines our current state and
develops a plan for future process improvement - Proactive management is required to go forward
with successful implementation of the Improvement
Plan
55Upcoming Activities
- Receive feedback on Findings Briefing
- Develop draft Software Process Improvement Plan
- Review SPI Plan with sponsors
- Establish Senior Management Steering Group (SMSG)
- Establish Software Engineering Process Group
(SEPG) - Finalize SPI Plan
- Conduct SPI Plan Briefing
- Implement Improvement Plan
56Management Steering Group Description
- Comprised of the LaRC CIO, ISO Project Manager,
and Division Chiefs who have a stake in the
successful achievement of the Software Process
Improvement (SPI) goals - Monitors and evaluates SPI efforts from the
perspective of the total organization to ensure
that the overall improvement efforts are in
concert with LaRCs mission and goals - Meets regularly with the Software Engineering
Process Group (SEPG) to discuss the progress of
the SPI program, problems, issues and concerns - Works together with the SEPG to identify and
provide the long-term commitments and resources
required to coordinate the development and
maintenance of software engineering processes for
use by current and future software projects
57Suggested Management Steering Group Membership
- Doug Arbuckle/Associate Director, LaRC
- Fay Collier/ISO 9000 Project Manager, LaRC
- Pat Dunnington/CIO, LaRC
- Luat Nguyen/FDCD
- Bill Wessel/OSEMA
- Carl Gray/FSED
- Milt Holt/FETD
- Rob Kudlinski/ISSD
- Lenny McMaster/AESD
- Bill Smith/SSCD
- David Stephens/FMAD
58Responsibilities of the Management Steering Group
(MSG)
- Ensure alignment of Software Process Improvement
(SPI) initiatives with LaRC mission and goals - Approve strategic plan for SPI
- Provide sponsorship, pro-active commitment, and
visible management support - Allocate resources
- Monitor the progress of the SPI initiative
- Provide guidance and direction to the Software
Engineering Process Group (SEPG) - Conduct regular meetings with the SPI Group
- Promote cooperation and cross-functional
communications - Cultivate an enabling environment for continuous
process improvement - Obtain and sustain the support for the SPI
initiative
59Software Engineering Process Group (SEPG)
Description
- Chartered by the MSG, as the focal point for
software process improvement - Coordinate and plan the SPI program activities in
concert with the guidance and direction of the
MSG - Report the progress of SPI related activities to
the MSG - Provide technical support and consultation
- Staffed by experienced software development
practitioners who facilitate the definition,
maintenance, and improvement of the software
engineering processes used within the
organization - Work collaboratively with the projects to resolve
process related problems and assist in the
development, training and implementation of
processes and procedures
60Suggested Software Engineering Process Group
(SEPG) Membership
- Members of the CornerStone team
- Volunteers from interview workshops
- Appointees from participating divisions
61Responsibilities of the Software Engineering
Process Group (SEPG)
- Define and manage the plan for development and
implementation of software process improvements
across the LaRC - Provide a pool of software engineering expertise
and corporate knowledge (Formal part of LaRC
organization with assigned resources and
management commitment) - Provide consultation and guidance on appropriate
level of software engineering implementation and
future directions - Provide and facilitate education on software
engineering to management and staff via
workshops, seminars, symposiums, and setting up
news/user groups and maintain web sight - Provide a repository for reuse code, documents,
tool knowledge, procedures, processes, LaRC best
practices, templates, lessons learned, metrics,
and examples - Facilitate sharing of tools and maintenance of
COTS across projects - Build and reinforce sponsorship for the SPI
program management support at all levels
62Next Steps
- Establish membership of MSG SEPG (by Nov. 12)
- Post Best Practices on Web
- Prioritize Improvement Opportunities
- Develop SPI Project Plan
- Review and Obtain Approval of Plan (by Dec. 19)
- Complete Final Report
- Implement Improvement Plan
63Closing Thought
- Creative thinking may simply be the realization
that there is no particular value in doing things
the way weve always done them. - - R. Flesch, Educator
64Odd and ends follow
- Global Improvement Opportunities
- Seminars (maybe during lunch) on SW topics of
interest - Establish Web site to capture and publicize best
practices - Establish basic training in software engineering
- Short-course training on widely-used COTS.
- Need to provide guidelines and expertise for
appropriate level of software engineeringprocedure
s based on project size, application and
criticality.
65Odds and ends cont.
- Some analysis was started on the Closing Q1 data
that is captured below(not to be used in findings
briefing)Global Improvement OpportunitiesgtProv
ide education on requirement gathering and
documentationgtDo pilot/demonstration projects on
selected software engineering activitiesgtConduct
lesson learned review on project complete and
post results on the webManagement Related
Improvement OpportunitiesgtNeed to overcome
resistance to positive changegtSegmentation of
responsibilities across organizations has led to
lack of single point of accountability for
technical issuesgtImprove management awareness of
the pervasiveness?? importance and magnitude of
work required to develop softwareGlobal
ConcernsgtImproper ISO 9000 implementation could
impose restrictive procedures on research and
significantly impact resources to perform future
researchgtCivil servants should be made
technically knowledgeable in softwaregtSponsor
research in appropriate software engineering
practices for the research environmentNASA
software put in commercial system with no profit
to NASA (companies make profits on NASA
software)
66Responsibilities of the Software Engineering
Process Group (SEPG)
- Establish a repository for SPI products (best
practices, code, and lessons learned) - Establish appropriate organizational metrics and
mechanisms for the collection of metrics - Provide desk-side mentoring to project teams
- Track, monitor and report the status of SPI
initiatives to the MSG - Promote technical awareness and coordinate
training for software engineering with the LaRC
training office - Provide consultation and CMM expertise to the MSG
and the technical working groups which implement
improvements - Compare current practices against the goals and
key practices of the CMM - Keep LaRC apprised of SPI activities and progress
67Center Sponsors
- Kristin Hessenius/Associate Director, LaRC
- Fay Collier/ISO 9000 Project Manager, LaRC
- Pat Dunnington/CIO, LaRC
- Rob Kudlinski/ISSD