Title: NASA Software Assurance Training
1NASA Software Assurance Training
Susan J. Sekira GSFC Software Assurance Lead
2Course Agenda
- Course Objectives
- Overview of Software Assurance
- Acquirer Software Assurance
- Provider Software Assurance
- Software Measurement
- NASA Software Assurance Classification Assessment
- Supplementary Information
- Summary
3Course Objectives
4Software Assurance Course Objectives
- To establish a common framework and knowledge
base among the NASA software community - Establish roles and responsibilities
- Enable consistency in implementation and
performance - Eliminate misconceptions about software assurance
- To highlight the benefits of software assurance
throughout the development life cycle - To encourage and strengthen implementation of
software assurance practices
General Audience Project Managers, Software
Managers, Software Engineers, Safety Engineers,
and Systems Engineers
5Overview of Software Assurance
6NASA Software Assurance Challenges
- Performance Based Contracting
- Lack of software assurance requirements in
proposals - Limited insight into interim products
- Increased code size and complexity
- Resources
- Constrained budgets
- Compressed schedules
- Limited availability of trained personnel
- Increased visibility
- Loss of mission
- Loss of credibility
- Achieving high quality software
- (and knowing what that means!)
MARS
7Agency Response and Actions
- Renewed emphasis on process improvement
initiatives (e.g., CMM, CMMI) - Improved and accepted tools and techniques
- Early Software Assurance involvement during
Software Acquisition activities - Redefinition of NASA Software Assurance and
updates to standard - Approved NASA Software Engineering Requirements
NASA Software Assurance Standard was updated in
July 2004!
8What is Software Assurance?
Software Assurance is an umbrella risk
identification and mitigation strategy for safety
and mission assurance of all NASAs software
VV
SQ
IVV
Software Reliability
Software Safety
SQ Software Quality VV Verification and
Validation IVV Independent Verification and
Validation
9Whats in it for Me?
Software Assurance
- Strives to improve the quality of the product
while employing risk mitigation techniques - Focuses on opportunities for early error
detection, problem prevention, and risk
identification and mitigation - Provides project management insight into the
software development processes and products
throughout the life cycle - Reviews and assesses interim products cant
build quality in at the end! - Improves the quality of future products/services
The level of Software Assurance needed is
commensurate with the software classification as
well as software size, complexity, criticality,
and risk
10Value of Software Assurance
Relative Cost of Finding, Fixing Errors Adapted
from Software VV, Steven Rakitin
- Earlier detection is cheaper to fix with lesser
impact to schedules - Increases chances of delivering full
functionality within cost and schedule - Safety issues are identified and addressed early
as part of overall system safety - Software assessments can be used to determine
milestone completion - Metrics provide indicators of the quality and
maturity of the software/system
100
100
60
50
50
40
Cost to Fix
30
20
20
5
1
10
0
Requirements
Design
Code
Test
Maintenance
Phase Error Found Fixed
11Who are the Practitioners?
- Software Assurance practitioners INCLUDE a wide
range of personnel, employed throughout the
software development life cycle
Software Managers
Safety Engineers
Software Engineers
Systems Engineers
Software Quality Personnel
IVV Personnel
Software Assurance personnel arent just those
Software Quality folks!!
12The Disciplines of Software Assurance
13Software Quality
- Assures that quality is built into the software
through the functions of - Software quality assurance
- Software quality engineering
- Software quality control
- Ensures conformance of software life cycle
processes and products to requirements,
standards, and procedures - Performs process and product activities
throughout the life cycle to provide objective
insight into the maturity and quality of the
software processes and products - Promotes continuous process improvement
14Sample Process and Product Activities
Process
Product
- All plans (e.g., Configuration Management Plan,
Software Management Plan) and procedures are
implemented according to specified standards and
procedures. - Engineering peer reviews and management reviews
are conducted and action items are tracked to
closure - Tests are planned, documented, and conducted
using approved test procedures and tools - Project risks are documented and addressed in
accordance with the risk management plan
- All required plans are developed in accordance
with specified requirements, standards, or
procedures - All software requirements are documented and
traceable from system requirements to design,
code, and test - Software development records are maintained and
up-to-date - Configuration baselines are managed, maintained,
and accurate - Software quality metrics are in place and used to
manage the software development effort
SQ identifies strengths, weaknesses, and areas
for improvement!
15Software Safety
Software Safety is a systematic approach to
identifying, analyzing, tracking, mitigating and
controlling software hazards and hazardous
functions (data and commands) to ensure safer
software operation within a system.
Software Safety entails
Testing of software safety critical components on
actual hardware to ensure that the safety
requirements were sufficiently implemented and
that applicable controls are in place to verify
all safety conditions
Ensuring that software safety requirements are
clearly identified, documented, traced and
controlled throughout the software lifecycle
Analysis of the consistency, completeness,
correctness and testability of software safety
requirements
Continuous analysis of proposed changes on system
safety
Software Safety is a function of System Safety!
16What is Safety-Critical Software?
Does the software meet any of the following
criteria
- Reside in a safety-critical system (as determined
by a hazard analysis) AND at least one of the
following - Cause or contribute to a hazard
- Provide control or mitigation for hazards
- Control safety-critical functions
- Process safety-critical commands or data
- Detect and report, or take corrective action, if
the system reaches a specific hazardous state - Mitigate damage if a hazard occurs
- Reside on the same system (processor) as
safety-critical software
17What is Safety-Critical Software? cont.
Or does the software meet any of the following
criteria
- Process data or analyze trends that lead directly
to safety decisions (e.g., determining when to
turn power off to a wind tunnel to prevent system
destruction) - Provide full or partial verification and
validation of safety-critical systems, including
hardware and software subsystems
If the software is determined to be
safety-critical, then the project must adhere to
the NASA-STD-8719.13, NASA Software Safety
Standard
18Safety-Critical Software Examples
- At GSFC, the following software is considered
safety-critical - Example 1 Software on the Hubble Robotic
Vehicle (HRV) that controls the robotic arm for
servicing the spacecraft - Example 2 Guidance and navigation software on
HRVs De-Orbiting Module (DM) used to dock the DM
and HST
NOTE Engineering telemetry data used to provide
safety information, but not required for safety
or hazard control, is not safety-critical!!
19Software Reliability
- Software Reliability is concerned with
incorporating and measuring reliability in the
products produced throughout the life cycle
Specifically
- Systems Engineering defines software requirements
as they contribute to system robustness - Software Engineering develops software containing
required redundancy and fault tolerance AND
measures and analyzes softwares ability to
withstand errors - Software Quality assures quality metrics/measures
are documented, monitored, analyzed and tracked
(e.g., error density) AND verifies that
reliability requirements have been successfully
demonstrated
20Verification and Validation (VV)
- Software Verification and Validation
- Ensures that software being developed or
maintained satisfies functional and performance
requirements - Ensures that each phase of the development
process yields the right products - Every participant in the software life cycle
process plays a role in some aspect of VV! - VV activities include, but are not limited to
- Analysis of system and software requirements
- Engineering peer reviews (e.g., code
walkthroughs) - Test planning and test execution
- Audits/assessments (e.g., baseline management)
21Independent VV IVV
- IVV is Verification and Validation performed by
an organization that is technically,
managerially, and financially independent of the
development organization - IVV focuses on mission critical software,
provides additional reviews and analyses, and
provides in-depth evaluations of life cycle
products that have the highest level of risk
Examples
- Validation of design to meet system
needs/requirements - Traceability of safety critical requirements
- Code analysis of mission-critical software
components - Design analysis of selected critical algorithms
22So what do I need to do?
- As a Manager, ask the following
- How much software is predicted to be on the
project? How much will be acquired vs. developed
in-house? - What is the softwares classification (i.e.,
Class A G or Exploratory)? - What is its criticality? What functions will
software control? What hazard controls and
mitigations? - What metrics are being collected? Can they help
me measure the maturity and quality of the
software product? - Have I carefully planned for software assurance?
Are SQ, IVV, and Software Safety personnel
on-board? - Are my contracts/MOUs written to assure the
safety and quality of the software?
23Getting Started
- The NASA Software Assurance Standard
NASA-STD-8739.8 specifies requirements for
software assurance for use by NASA projects,
programs, and facilities - This standard
- Provides a software life cycle perspective for
the minimum required software assurance
procedures that contribute to quality software - Provides specific requirements for each
discipline of software assurance - Provides the Acquirer and Provider
requirements for software assurance to obtain the
most cost effective, best quality, and safest
products
24Acquirer vs. Provider
Acquirer (Usually NASA or an Organization within
the Agency) Specifies the requirements and
accepts the resulting software products
Provider(May be a contractor, a university, a
separate organization within NASA, or within the
same organization as the Acquirer) Designs,
develops, implements, tests, operates, and
maintains software products
Software Assurance begins with the Acquirer!
25Acquirer Software Assurance
26Acquirer Software Assurance
- Getting started
- Identify a person responsible for software
assurance (e.g., a software assurance manager) - Secure budget/resources for software assurance
EARLY on (just like any other functional area) - Participate in the Initiation phase of the
program/project to ensure appropriate
oversight/insight into requirements, including
needed deliverables
Notify your Safety and Mission Assurance
Organization to arrange for Software Assurance
Expertise
27Acquirer Software Assurance
- Key Activities
- Performs Software Assurance Classification
Assessment to identify and evaluate the
characteristics of software in determining the
softwares classification - Applies software classification to tailor
software assurance requirements - Develops SA input to the Request for Proposal
(RFP) - Reviews proposals for compliance to SA
requirements - Establishes and maintains a software assurance
plan for the acquirer - Ensures consistency between acquirer and provider
SA plans - Ensures SA training for both the acquirer and
provider
28Acquirer Software Assurance
- Key activities (cont.)
- Provides surveillance to assure both the acquirer
and provider SA organizations perform to their
plans and procedures - Assures that problem reports, action items, and
test anomalies are documented, addressed,
analyzed, and tracked to closure - Assures that software products are reviewed and
any metrics collected, analyzed, and trended - Reports at all formal software reviews (e.g.,
CDR) - Fosters effective communication with project
management, other project assurance personnel,
and the provider
29Acquirers Software Assurance Deliverables
- SA Input to RFP
- Software Assurance Plan
- Addresses all SA disciplines
- Outlines activities and deliverables
- Software Assurance Status Reports
- Results from software assurance activities
- Issues/risks/recommendations
- Software Assurance Records
- Reports
- Analyses
- Metrics
- SA Input at Formal Reviews
30Acquirers Role by Phase
- INITIALIZATION,PRE-AWARD
- Secure SA expert/resources
- Conduct software classification assessment
- Specify SA reqts for inclusion in RFP -
including provider deliverables for SA - Plan acquirers SA activities prepare
Preliminary SA Plan
- RETIREMENT
- Assure preparation, approval, and execution of
retirement plan - Ensure plans for archival or disposal of SA
records
- MAINTENANCE
- Ensure SA processes in place
- Assure Metrics are transferred to Maintenance
Organization and maintained
- POST-RFP PRE-AWARD
- Evaluate proposal for SA requirements
- Conduct Pre-award Surveys
- Participate in contract negotiations
ACQUIRERS ROLE
- POST-AWARD PRE-IMPLEMENTATION
- Ensure provider SA plan meets requirements
- Verify consistency between acquirer and
provider SA plans - Ensure trained SA personnel
- OPERATION
- Review SA plan
- Ensure SA processes in place
- Conduct periodic audits of OPS
- CONTRACT
- IMPLEMENTATION
- Conduct surveillance
- Assure CM
- Assure provider SA
- Ensure acquirer performs tasks
- ACCEPTANCE
- Conduct Audits prior to delivery
- Assure Facility readiness
- Assure acceptance documentation
- Capture Lessons Learned
31Provider Software Assurance
32Provider Software Assurance
- Key Activities
- Identifies one or more persons to direct and
manage a software assurance program (e.g., a
software assurance manager) independent from
the project - Develops a software assurance program that
includes SQ, Software Safety, Software
Reliability, VV, and IVV (when required) - Establishes and maintains a working relationship
with project management and the acquirer - Establishes and maintains a software assurance
plan that conforms to IEEE STD 730-2002 - Conducts periodic reviews, audits, and
assessments of the development processes and
products
33Provider Software Assurance
- Key activities (cont.)
- Prepares and maintains software assurance records
from software assurance activities - Participates in formal and informal reviews
- Prepares software assurance status reports,
highlighting - Assurance accomplishments from reviews, tests,
etc. - Significant problems, their status, solutions,
and remedial and preventive actions - Trends in software quality metric data
- Plans for upcoming software assurance activities
- Recommendations and lessons learned
- Flows down software assurance requirements to any
subcontractors and assures compliance
34Provider Deliverables
- Sample Software Deliverables
- Software Management Plan
- Configuration Management Plan
- Software Requirements Specification
- Requirements Traceability Matrix
- Test plans and procedures
- Users Guides
- Etc.
- Software Assurance Deliverables
- Software Assurance Plan
- Software Assurance Status Reports
- Software Assurance Records
See NPR 7150.2 for more on software products
and required content
35Providers Role by Phase
- INITIALIZATION,PRE-AWARD
- Draft SA approach
- Respond to SA requirements in RFP
- RETIREMENT
- Send final SA records to project
-
- MAINTENANCE
- Ensure SA processes in place
- Conduct periodic audits or assessments
- POST-RFP PRE-AWARD
- Address SA questions from acquirer
- Participate in contract negotiations for SA
PROVIDERS ROLE
- POST-AWARD PRE-IMPLEMENTATION
- Develop SA plan
- Review Acquirer SA plan
- Ensure trained SA personnel
-
- OPERATION
- Develop New/updated SA plan
- Ensure SA processes in place
- Conduct periodic audits of OPS
- CONTRACT
- IMPLEMENTATION
- Establish communication with acquirer, project
team - Maintain SA plan
- Conduct /participate in reviews, inspections,
audits and prepare SA reports - Collect, utilize metrics to identify trends,
assess quality - Deliver SA reports, maintain records
- ACCEPTANCE
- Conduct audits prior to delivery
- Assure acceptance documentation
- Prepare Lessons learned
36Communication is Essential!
Acquirer SA
Provider SA
- Verify Provider SA Plan is complete and
compatible with Acquirer SA Plan - Review Provider plans and procedures
- Review Provider SA reports and records from
surveillance activities - Evaluate software products and deliverables
throughout life cycle - Prepare SA Status reports for Project
- Establish and maintain working relationship with
Project, software assurance personnel, and the
Provider
- Develop Provider SA Plan per Acquirer
requirements - Review Acquirer plans and procedures
- Flow down requirements to any subcontractors
- Prepare and maintain SA records (accessible by
Acquirer) - Prepare SA Status reports for Project and
Acquirer - Establish and maintain working relationship with
Project, software assurance personnel, and the
Acquirer - Share issues/concerns/risks
Software Assurance Plan
Plans and Procedures
Software Products and Deliverables
Software Assurance Records
Software Assurance Status Reports
Meetings, Telecons, and and Project Reviews
37Software Measurement
when you can measure what you are speaking
about, and express it in numbers, you know
something about it when you cannot express it in
numbers, your knowledge is of a meager and
unsatisfactory kind
Lord Kelvin, 1889
38Why Measure?
- Measurements
- provide meaningful information to enable informed
control decisions - provide data from which to model and predict
future software trends
- Are we there yet?
- How much time and effort required to detect
remaining errors? - To what extent is product error free?
- What will it take to get there?
- Why did/didnt we get there?
- Post mortem analysis lessons learned
39Sample Metrics
Start simple with only a few metrics
- Requirements
- Ambiguity weak phrases
- Completeness TBD TBA TBR
- Volatility excessive requirement changes
- Design Implementation
- Structure/Architecture complexity size
- Verification (e.g., formal reviews, peer reviews,
testing) - Defect density (defects per 1000 lines of code)
- Problem report tracking open, closed, severity
40Requirements Metrics Completeness Volatility
Analysis
Modifications to Requirements
Total Number of Requirements
450
1000
400
900
New
Modified
800
350
Deleted
700
300
600
250
Quantity
Quantity
500
200
400
150
300
100
200
50
100
0
0
1Q94
2Q94
3Q94
4Q94
1Q95
2Q95
3Q95
1Q94
2Q94
3Q94
4Q94
1Q95
2Q95
3Q95
Calendar Quarter
Calendar Quarter
CRR Looks Good! (Stable)
CRR Excessive Changes! NOT Stable
Combination of BOTH views indicates risk area -
requirements are NOT YET stable
41Defect Trending
Problem --gt Opening Errors Continues sharp climb
Expected
42Hints for Successful Measurement
- Define your measures of success early in your
project and track your progress toward them - Collect only what you need, and use what you
collect - Use more than one metric to interpret the entire
picture - Use defect data trends to help you decide when to
release a product - Measure complexity to help you optimize design
decisions and create a more maintainable product - Ensure reasonable data collection, interpretation
of data - Categorize defects to identify product and
process weaknesses
43NASA Software Assurance Classification Assessment
44SA Classification Assessment Process
- The Software Assurance Classification Assessment
was developed by NASA to identify and evaluate
the characteristics of software to determine the
softwares classification and level of software
assurance to be applied - This assessment is conducted for all NASA
software - The process entails 4 basic steps
- Perform the Software Safety Litmus Test to
determine if the software has safety implications
for the system, property, humans, or the
environment - Determine the software engineering class of
software (Class A-H) - Determine the software criticality score based on
project/mission/software characteristics - Classify the software and software assurance
effort based on the results from Steps 1-3
45 SA Classification Assessment (cont.)
- After each step of the process, record the
results in a Software Assurance Classification
Report and maintain as a quality record - Provide report to the SMA Director and Systems
Management Office (SMO) - Report contents include
- Project name and date
- Names of software assurance manager and
- project manager
- Result of software safety litmus test
- Software Class (A-H)
- Software Criticality score
- Software Assurance Effort/Priority
46Software Assurance Classes
- Class A Human Rated Software Systems
- Class B Non-Human Space Rated Software Systems
- Class C Mission Support Software
- Class D Analysis and Distribution Software
- Class E Development Support Software
- Class F General Purpose Computing Software
(Multi- Center or Multi-Program/Project) - Class G General Purpose Computing Software
(Singe Center or Project) - Class H General Purpose Desktop Software
47Tailoring Guidelines for SA Requirements
- The level of Software Assurance needed is
commensurate with the software classification,
criticality, unique software characteristics
(e.g., software size, complexity), and perceived
risk - Based on the NASA SA Standard and NPR 7150.2,
tailoring guidelines are being developed to
assist assurance practitioners in defining the
software assurance requirements required to meet
specific project/mission/software objectives - The tailoring guidelines will be provided as part
of the NASA Software Assurance Guidebook
(currently under revision)
48Supplementary Information
49GSFC Contacts
- Software Working Group (SWG) Representatives
- Sally Godfrey 301 286-5706
- Susan Sekira 301 286-6160
- Center Software Assurance Lead (Code 303)
- Susan Sekira 301 286-6160
- Software Assurance Technology Center (Code 304)
- Al Gallo, Manager 301 286-3756
- Systems Safety and Reliability Office (Code 302)
- Karen Fisher, Chief 301 286-7123
- Flight Software Branch (Code 582)
- Elaine Shell, Head 301 286-2628
50Recommended Web Sites
- NASA Software Assurance
- http//software.nasa.gov/
- NASA Software Working Group
- http//nasa-software-pbma-kms.intranets.com/
- NASA Process Based Mission Assurance (PBMA)
- http//pbma.hq.nasa.gov/pbmamaster.html
- NASA Technical Standards
- http//standards.nasa.gov/
- Carnegie Mellon- Software Engineering Institute
- http//www.sei.cmu.edu/
- Software Engineering Laboratory
- http//sel.gsfc.nasa.gov/
51More Web Sites
- Software Assurance Technology Center
- http//satc.gsfc.nasa.gov/
- NASA Independent Verification and Validation
- http//www.ivv.nasa.gov
- GSFC Engineering Process Group
- http//software.gsfc.nasa.gov
- GSFC Software Assurance
- http//sw-assurance.gsfc.nasa.gov
52GSFC SA Groups and Initiatives
- Engineering Process Group (EPG)
- Center-wide team that establishes and
continuously improves system and software
processes and products - Software Assurance Technology Center (SATC)
- Center of Excellence that provides project
support, outreach, and research in software
metrics and software assurance tools and
techniques - Quick-Look Project Team
- Team of software experts organized to take a
quick look at potential (or impending) software
problem areas for a specific project
53GSFC Procedures and Products
- GPG 8700.5, In-House Development and Maintenance
of Software Products - 580-PG-8730.3.1, ISC Product Development Handbook
- GPG 8700.6, Engineering Peer Reviews
- 300-PG-7120.2.2, Mission Assurance Guidelines
Tailoring - 303-PG-7120.2.1, Developing and Implementing
Software Quality Programs - Also available work instructions, checklists and
templates for - Software Quality (SQ) process and product
assessments - SQ Checklists for evaluating processes and
products - FSW Product Plans and Test Plan Templates
- FSW Coding Standards for Ada and C
- FSW CCB Process and Policy
- FSW Development Life Cycle
- ARM tool, code analysis and metrics,
reliability tools (via SATC)
54Additional Training Opportunities
- SOLAR Training https//solar.msfc.nasa. gov/
- Software Assurance
- Software System Safety
- NASA Engineering Training Program (NET)
http//net.larc.nasa.gov/ - Introduction to CMMI
- Intermediate CMMI
- Defining World-Class Processes
55Recommended Reference Documents
56Additional References
- Conte, Dunsmore, Shen,Software Metrics
Engineering Metrics and Models,
Benjamin/Cummings, 1986 - Leveson, Nancy G., SAFEWARE System Safety and
Computers, Addison-Wesley Publishing Company,
Inc., 1995 - Lyu, Michael R., Handbook of Reliability
Engineering, IEEE Computer Society Press, McGraw
Hill, 1995 - Shulmeyer, Gordon G., and James I. McManus,The
Handbook of Software Quality Assurance,Prentice
Hall, 1999 - Kan, Stephen H., Metrics and models in Software
Quality Engineering,Addison Wesley, 2002
57Summary
58Benefits of Software Assurance
- Provides insight into the software development
processes and products throughout the life cycle - Focuses on opportunities for
- Early error detection
- Problem prevention
- Risk identification and mitigation
- Increases chances of delivering full
functionality within cost and schedule - Enables improvement of future software products
and services!!
59Example SA Activities
- Software Assurance Classification Process
- Proposal and contract evaluations
- Requirements traceability
- Design and code analyses
- Engineering peer reviews
- Test planning and test execution
- Software measurement and trends
- Audits and assessments
Team work and communication!!
60Keys to a Successful SA Program
- Identify a person responsible for directing and
managing the Software Assurance Program - Define software assurance requirements early in
the life cycle - Develop a Software Assurance Plan
- Monitor processes throughout the system and
software development life cycle - Evaluate software and deliverables to assure that
quality and safety are being built into the
products - Ensure compliance with established standards and
procedures - Establish metrics to help measure quality
61More Keys
- Assure that problems and risks are documented,
reported, addressed, and tracked to closure - Prepare and maintain software assurance records
and status reports - Capture Lessons Learned to improve the quality of
future products/services!
62Backup Slides
63VV Activity Matrix
64Fundamental Differences
SQ
IVV
- Provides Center-level services
- Focuses on ALL Project software
- Emphasizes compliance to standards and
procedures - Reviews, monitors and audits all Project
processes and products for completeness and
accuracy - Matrixed to the Project as part of the Project
Team and provides daily insight/oversight - Reports to Project and Center Director through
SMA
- Provides Agency-level services
- Focuses on MISSION CRITICAL Project software
- Emphasizes completeness and correctness of the
product - Reviews, analyzes, and provides in-depth
evaluations of life cycle products which have
the highest risk - Independent from the Project and provides
analyses and evaluations per IVV priorities - Reports to Project, GPMCs, and NASA Headquarters
65Frequently Asked Questions
- How do we levy software assurance requirements on
an existing software development activity? - How do these requirements apply to a small
development effort? - Are these requirements applicable to in-house
work? - Who determines the software classification for my
software? - Do I need to document the classification?