? Improving the Current State of Software within NASA

1
? Improving the Current State of
Software within NASA

• May 8, 2000
• Presentation by CIO, CE, and Code Q
• SMC and NASA Administrator
• Version 5
• Briefing to SMC and NASA Administrator CIO_Code
  Q_Chief Eng Talk_V5_March 28 V1.ppt

2
NASA Software Working Group Charge

• Identify software challenges in NASA
• What are our goals
• What are our challenges in meeting those goals
• Recommend plans to address the challenges and
  obtain the goals
• Provide recommendations concerning the
  implementation of software best practices
• Identify methodologies to evaluate and improve
  the quality of software across the Agency

3
Goals

• Primary Goal
• Successfully Develop Software to support NASA
  Enterprises and Missions
• High Quality
• Meet all project requirements
• Safely
• Reliably
• Within approved budget
• Within approved schedule
• Secondary Goal
• Continual Sustained Software Process and Product
  Improvement
• Improve quality
• Reduce cost
• Increase productivity

4
Sources of Challenges Data

• Survey data from individual NASA employees
• Input received from Ames, GRC, GSFC, JPL, LaRC,
  MSFC, HQ
• Data provided from NASA organizations
• NASA CIOs presentations (JPL and HQ)
• Software Engineering Process Group (LaRC )
• Software Assurance Technology Center (GSFC)
• Software Engineering Laboratory (GSFC)
• Flight Software Group (MSFC)
• Mishap reports
• Mars Climate Orbiter Mishap Investigation Board,
  Phase I Report, NASA, November 10, 1999
• "Report on the Loss of the Mars Climate Orbiter
  Mission", JPL D-18441, JPL Special Review Board,
  Nov. 11, 1999
• "Common Errors Afflicting Real Time Control
  System Software", Task 2 Interim Report, April
  30, 1999, Prepared by JSC NX Technology Division,
  Approved by M. Himel/Chief, NX Technology
  Division

5
Findings Sorted into Categories of Challenges
6
Top 10 Software Challenges
7
Sorted List of Software Challenges
8
Findings

• The following give examples highlighting the
  seriousness of the top 10 priority challenges
• Requirements Management
• The flow down of requirements from the
  higher-level document to the software
  requirements document was not done resulting in
  loss of mission
• Software Product Engineering
• Software requirements are not controlled to
  establish a baseline for software engineering and
  management software requirements are poorly
  documented (if at all) and are not always
  reviewed with the customer or kept up to date to
  reflect and communicate changes, and are not used
  as the basis for software management plans,
  products and activities
• Software designs using effective methods are not
  being developed, maintained, documented, and
  verified to assure the requirements are fully
  covered and to form a framework for coding the
  resulting code either does not work properly or
  if it works it has very high maintenance costs do
  to the poor design being hard to modify

9
Findings continued

• Software Product Engineering - continued
• Software code is not adequately documented and
  verified the result is that the software does
  not meet the requirements
• Software testing is performed in an ad hoc
  manner, test procedures do not fully cover the
  requirements if they are written at all software
  products rarely work as expected, lack of
  sufficient testing prior to delivery leads to
  high maintenance costs
• Integration testing is performed in an ad hoc
  manner, integration test procedures do not fully
  cover the requirements if interface requirements
  were written at all partial or total system
  failure is experienced and hug overruns to pay
  for expensive error correction in the latter
  phases of the project
• System and acceptance testing
• Interface with navigation software not tested
• Complete end to end testing including interfaces
  is not done
• Acceptance criteria is either not defined or not
  well defined

10
Findings continued

• Software Quality Assurance and IV
• Software quality assurance staff is 0 to 1 deep
  at some centers
• Testing until run out of budget or time vs. use
  of thorough testing methodologies
• Systems Engineering
• Due to the lack of systems engineering performed,
  software and hardware staff are forced to fill
  that role in an ad hoc manner
• System engineers often have little software
  expertise
• The role and responsibility of system analysis is
  not clearly understood or defined
• The management of complexity is one of the most
  important issues facing NASA missions
• Software Project Tracking and Oversight
• Actual results and performance are not tracked
  against the plans, corrective actions are not
  taken and managed to closure changes to
  commitments are not agreed upon
• Software Reliability, Safety
• Making software safer and more reliable

11
Findings continued

• Software Configuration Management
• Lack of proper configuration management has
  contributed and caused mission failure
• Human Resources
• Software professionals are the first to leave
  the hardest to replace
• Obtaining and keeping Program/Project Managers
  sufficiently knowledgeable of software issues is
  a serious problem
• Software Project Planning
• The way project phases are currently structured,
  emphasis is not placed on software planning which
  leads to huge overruns in testing or high
  maintenance cost that all could be avoided if
  proper time was allotted up front
• Finding knowledgeable and experienced software
  project managers who can apply disciplined
  engineering practices to software development
• Management Support
• Engineers perceive little or no management
  commitment to the software process
• There are no investment, encouragement, or reward
  systems for improvements, best practices, or
  technology transfer

12
Summary

• Conclusions
• Our current software state is an Ad -hoc
  application of engineering
• We must progress to a state of disciplined,
  consistent, repeatable application of software
  engineering principles
• We will not achieve software quality without this
• Need endorsement for the following to effect a
  holistic solution
• Establish SEPGs at each Center
• SWG define a NASA Software Engineering
  Implementation Plan to manage and coordinate
  software improvements efforts across the Agency
• Headquarters review and approve plan and
  disposition SWG recommendations
• Goals
• Successfully Develop Software to support NASA
  Enterprises and Missions
• Continual Sustained Software Process and Product
  Improvement
• See additional findings on the next slides

13
Even Successful Missions may Experience Software
Problems (Excerpt From Another Briefing)

• A few days after the July 4th, 1997 landing, the
  Mars Pathfinder began experiencing total system
  resets, each resulting in losses of data. The
  problem was a logical error in the real-time
  scheduling system---a classic priority-inversion
  problem. Fortunately, this problem was
  repairable from earth.
• A malfunction in one of the on-board computers
  on Clementine on May 7, 1994 caused a thruster to
  fire until it had used up all of its fuel,
  leaving the spacecraft spinning at about 80 RPM
  with no spin control. This made the planned
  continuation of the mission, a flyby of the
  near-Earth asteroid Geographos, impossible.
• The Magellan spacecraft broke Earth lock and
  lost communications several times in August 1990
  (soon after entering Venus orbit). It took over
  six months to identify the source of the problem,
  which was a timing error in the flight software.

14
(Excerpt From Another NASA Findings Briefing)

• Centers are almost universally weak in
• Project planning
• Estimating cost, schedule, and resource
  requirements for project requirements fulfilled
  by software
• Monitoring and control of software engineering
  products
• i.e., tracking progress and taking effective
  corrective actions
• Configuration management is not universally
  applied throughout the software development
  process
• Interface between software and system engineering
  processes is not well defined so agreements,
  audits, and reviews are not well planned or
  performed to achieve the most benefit
• Software Quality Assurance is generally not well
  understood nor is its value appreciated

Findings by Raymond Kile, Authorized Lead
Evaluator Center for Systems Management, Sept 2002
15
Findings (continued)

• Software Contract Management
• Qualified software contractors are not selected
• Constantly receiving poor quality contract
  deliverables
• Software Product Maintenance
• The maintenance phase is rarely covered under the
  project initial budget estimates and plans
• Insufficient maintenance staff are available to
  support ongoing programs
• Peer Reviews
• Required persons were not in attendance at the
  walkthroughs
• "inadequate training of software and other
  personnel in the software walkthrough process"
• Reuse
• Software is re-written many times because the
  software is not developed and documented in such
  a way that it can be re-used.
• Software Coping with Evolving Technology, Tools,
  Methods, COTS and Environment
• Software architectures, testing, and validation
  for intelligent machines
• Tools that would help economically and accurately
  produce software are not funded (must compensate
  for this by manual methods that are prone to
  human error and much more time consuming and
  therefore more expensive)
• Technology assessment and infusion
• There is a huge lack of experience and knowledge
  in testing, integration on projects using
  COTS/GOTS/ middleware/ glueware
• Organization Process Focus
• Software process development and improvement
  activities are not planned, funded, or
  coordinated across the organization
• Strengths are not capitalized on, weaknesses go
  unchecked, those grass roots efforts that are
  trying to make progress are not coordinated
  resulting in duplication of effort

16
Findings (continued)

• Training and Education
• Individuals in the software arena do not receive
  the training necessary to perform their role and
  training is not always provided even if sought
• Integrated Software Management
• A projects defined software process is not a
  tailored version of the organizations standard
  software process since one does not exist
• Intergroup Coordination
• Requirements and engineering roles and
  responsibilities are not clearly defined and
  agreed to
• Need improved communication channels, awareness,
  and understanding between project management,
  hardware developers, and software developers
• Institutional Support / Infrastructure
• There is no place to get mentoring and training
  and advice on all aspects of software
  development
• Need to find mechanisms to penetrate projects, to
  introduce software engineering, and to work in
  partnership with the projects.
• Baseline data / Metrics/Lessons Learned
• Insufficient project tracking of size, cost, and
  error data to accurately do future planning,
  estimating, and process improvement
• Security
• ensure that security measures have been
  implemented on all of NASA's computing systems
• Policies and Standards
• Currently there is no effective enforcement of
  the NASA Software Policies NPD 2820.1
• There are no firm guidance or requirements on
  which software standards should be used in the
  agency
• General CMM related challenges
• Flight Software Group was self-assessed at a
  level 1(of 5) on the CMM at two centers