Technical Reviews Module Space Systems Engineering, version 1.0

1

Technical Reviews Module Space Systems
Engineering, version 1.0

2
Module Purpose Technical Reviews

• To understand the purpose and value of conducting
  technical reviews.
• To discuss the timing of technical reviews over
  the course of a projects life cycle.
• To consider the entrance criteria and success
  criteria for the standard project technical
  reviews.
• To understand when a technical review is
  complete.

3
Definition of Technical Reviews

• Technical reviews are scheduled within a project
  or program to communicate an approach,
  demonstrate an ability to meet requirements, or
  establish status.
• Technical reviews often serve as control gates
  for management to make a go/no-go decision on a
  project.
• Control gates involve formal examination of a
  projects status in order to obtain approval to
  proceed.
• Technical reviews may be held for subsystems too
  depending on their size and complexity. Subsystem
  reviews precede their corresponding system
  review.
• When do technical reviews occur?
• Throughout the entire life cycle of a project, as
  shown on the following life cycle chart.

4
Reviews in a NASA Projects Life Cycle
5
The Value of Technical Reviews

• Technical reviews are key development milestones
  used to measure progress, assess project maturity
  and to infuse lessons from the past. They
• Provide confirmation of completed effort and
  readiness to commit additional resources for the
  next phase.
• Encourage and establish project discipline with
  an integrated project team perspective.
• Identify risks and review mitigation options.
• Describe plans and priorities for the next phase.
• Technical reviews give everyone (design team,
  non-advocate discipline experts, customer and
  consumer) the opportunity to agree on the current
  project baseline (requirements, interfaces,
  allocations, margins, schedules, risks, budgets,
  etc.).
• Functional, resource and performance allocations
  and assumptions are described and confirmed.
• System development constraints and interfaces are
  described and agreed to by both sides.
• Risks and development problems are identified and
  mitigation options discussed.

6
More Values of Technical Reviews

• A significant, and perhaps unexpected, value to
  the development team is the preparation for a
  review. Usually everyone is busy in their own
  domain - preparing for a review forces the team
  to describe what they have done and understand
  what others are doing.
• While there is great value in the preparation,
  there is also unique value in the execution. One
  Air Force officer once took Dwight D.
  Eisenhowers famous quote
• Plans are nothing. Planning is everything.
• too seriously and on the day before it was
  scheduled cancelled the PDR of a 100 million
  space mission. After a minor revolt, the PDR was
  reinstated and the team benefited from the
  preparation and the review.
• Periodic project reviews are held to demonstrate
  that the appropriate products, accomplishments
  and plans have been completed before proceeding
  to the next phase.
• Appropriate products, accomplishments and plans
  are based on the lessons of hundreds of past
  projects (best practices). NASA, for example,
  describes standard entrance and success criteria
  for all standard milestone reviews.

7
NASAs Minimum Set of Technical Reviews

• Mission Concept Review (MCR),
• System Requirements Review (SRR) and/or Mission
  Definition Review (MDR),
• Preliminary Design Review (PDR),
• Critical Design Review (CDR),
• System Integration Review (SIR),
• Test Readiness Review (TRR),
• Operational Readiness Review (ORR),
• Flight Readiness Review (FRR),
• Post-Launch Assessment Review (PLAR),
• Critical Event Readiness Review (CERR), and
• Decommissioning Review (DR).
• While this may seem like a large number of
  technical reviews, each has its own focus and
  proven value.
• In addition, since mission lifetimes are several
  years, typically there is 6 months or more
  between major reviews.

8
The Primary Questions of Each Review

• Mission Concept Review - Does the proposed
  concept meet the mission need and objectives?
• System Requirements Review and/or Mission
  Definition Review - Do the functional and
  performance requirements and the selected concept
  satisfy the mission?
• Preliminary Design Review - Does the preliminary
  design meet all the system requirements within
  acceptable cost, schedule, and risk?
• Critical Design Review - Is the system design
  mature enough to proceed with full-scale
  fabrication, assembly, integration and test?
• System Integration Review - Are the system,
  facilities, personnel, plans and procedures ready
  for system integration?
• Test Readiness Review - Is the project ready to
  commence with verification testing?
• Operational Readiness Review - Are all systems
  hardware, software, personnel, and procedures in
  place to support operations?
• Flight Readiness Review - Is the system ready for
  launch? Are the ground facilities and personnel
  ready to support launch?
• Post-Launch Assessment Review - After launch and
  deployment, are the spacecraft systems ready to
  proceed with routine operations?
• Critical Event Readiness Review - Is the project
  ready to execute the missions critical
  activities during flight operation?
• Decommissioning Review - Is the system ready to
  be terminated?

9
Pre-Phase A Reviews

• Mission Concept Review (MCR)
• A validation that the mission has clearly
  established needs, objectives, and top-level
  functional/performance requirements, and that at
  least one way of conducting the proposed mission
  is realistic and attainable within existing or
  projected technology and ROM cost.
• An internal Center review(s) of all Pre-Phase A
  activities and products should be conducted prior
  to forwarding the Pre-Phase A report to
  Headquarters. Technical, management, resources,
  and scientific personnel should conduct the
  review.
• Peer Reviews
• Used informally during Pre-Phase A and Phase A.
• The peer group is composed of individuals
  selected from outside the project according to
  their expertise in the applicable disciplines.
• Throughout Pre-Phase A and Phase A, peer reviews
  should informally check the evolving mission
  concept against objectives, requirements, and
  constraints.
• Peer reviews help you take advantage of other
  engineering experience from colleagues who have
  worked on different missions. They can point out
  issues they confronted that may be similar for
  your mission concept.

10
Phase A Reviews

• System Requirements Review (SRR)
• The primary focus of the SRR is to verify the
  realism of the functional and performance
  requirements, ensure their congruence with the
  mission and system configuration, and ensure the
  mission objectives can be satisfied.
• The SRR encompasses all major participants (NASA
  and contractors), and is chaired by the Project
  Manager.
• A product from the SRR is the project system
  specification that is formally baselined and
  placed under configuration management control.
• Mission Definition Review (MDR)
• A validation that the mission objectives can be
  satisfied, the partitioning of the functionality
  to each of the systems is adequate, the top-level
  performance requirements for each system have
  been defined, and the technology required to
  develop the systems and implement the mission is
  attainable.
• The MDR is keyed to the end of Phase A and
  evaluates the mission definition, system design,
  operational concepts, schedule, and cost
  estimates.

11
Example Entrance Criteria System Requirements
Review
1. Successful completion of the Mission Concept
Review (MCR) and responses made to all MCR
Requests for Actions (RFAs). 2. A preliminary
SRR and/or MDR agenda, success criteria, and
charge to the board have been agreed to by the
technical team, project manager, and review chair
prior to the SRR and/or MDR. 3. The following
technical products for hardware and software
system elements are available to the cognizant
participants prior to the review a. System
Architecture. b. System requirements document. c.
System software functionality description. d.
Updated concept of operations. e. Updated mission
requirements, if applicable. f. Baselined
SEMP. g. Preliminary system requirements
allocation to the next lower level system. h.
Updated cost estimate. i. Technology Development
Maturity Assessment Plan. j. Preferred system
solution definition including major trades and
options. k. Updated risk assessment and
mitigations. l. Updated cost and schedule
data. m. Logistics documentation (preliminary
maintenance plan, etc.). n. Preliminary human
rating plan, if applicable. o. Software
Development Plan (SDP). p. System safety and
mission assurance plan. q. Configuration
management plan. r. Project management plan. s.
Initial document tree. t. Verification and
validation approach. u. Preliminary hazard
analysis (PHA).
12
Example Success Criteria System Requirements
Review

1. The resulting overall concept is reasonable,
   feasible, complete, responsive to the mission
   requirements, and is consistent with system
   requirements and available resources (cost,
   schedule, mass power, etc.).
2. The project utilizes a sound process for the
   allocation and control of requirements throughout
   all levels, and a plan has been defined to
   complete the definition activity within schedule
   constraints.
3. Requirements definition is complete with respect
   to top level mission and science requirements,
   and interfaces with external entities and between
   major internal elements have been defined.
4. Requirements allocation and flow down of key
   driving requirements have been defined down to
   subsystems.
5. System and subsystem design approaches and
   operational concepts exist and are consistent
   with the requirements set.
6. The requirements, design approaches, and
   conceptual design will fulfill the mission needs
   within the estimated costs.
7. Preliminary approaches have been determined for
   how requirements will be verified and validated
   down to the subsystem level
8. Major risks have been identified, and viable
   mitigation strategies have been defined.

13
Constellation Program Office Schedule of SRRs
You are Here!
2007
5/23
March
April
May
Provide Issue list w/ Resolution Plan Impacts
3/29
Rev. Issue Table, POC, Sched.
4/6
4/9
5/14
4/16
Develop PBS Presentation
Provide Status on Issue Resolution (every Mon. _at_
Cx SEI 830 telecon)
4/23
4/30
(send updates to Cx SEI COB every other day in
May)
5/10
Provide Issue list w/ Resolution Plan Impacts
4/16
Provide Status on Issue Resolution (every Mon. _at_
Cx SEI 830 telecon)
4/23
4/30
(send updates to Cx SEI COB every other day in
May)
CEV SRR
5/10
Provide Issue list w/ Resolution Plan Impacts
4/23
4/30
Provide Status on Issue Resolution (every Mon. _at_
Cx SEI 830 telecon)
4/16
(send updates to Cx SEI COB every other day in
May)
MO SRR
Provide Issue list w/ Resolution Plan Impacts
Develop PBS Presentation
GO SRR
Provide Status on Issue Resolution (send updates
to Cx SEI COB every other day in May)
Provide Issue list w/ Resolution Plan Impacts
Issue ID, Form Entry
Develop PBS Presentation
EVA SRR
Provide Status on Issue Resolution (send updates
to Cx SEI COB every other day in May)
14
Constellation Program Office Status - Closure of
SRR

• RIDs
• Total RID Count 6,283
• 6225 Closed
• 58 Open
• Majority closed prior to PBS
• All RIDs should be closed prior to CxP SDR

• Board AI
• Total AI Count 48 Total
• 9 Closed
• 39 Open
• 30 are Past Due
• 9 are in work, Not Due Yet

• TBD/TBR
• Total Count 2,532
• Received plans for 2,064 TBRs/TBDs in 72 of 95
  documents (76 of docs.)
• Need burn-down plans for 196 TBRs/TBDs in 16
  documents (17 of docs.)
• 7 documents (274 TBRs/TBDs) will not be
  updated/baselined until post CxP PBS. These
  TBRs/TBDs will not be closed prior to CxP PBS

15
Preliminary Design Review (PDR)

• The PDR is not a single review but a number of
  reviews starting with the specific component
  PDRs, followed by the system-level review.
• The PDR establishes the design-to baseline and
  ensures that it meets the program, project ,
  system, subsystem, or specific component baseline
  requirements.
• The PDR process should
• Establish the ability of the selected design
  approach to meet the technical requirements
  (i.e., Verifiability/ Traceability)
• Establish the compatibility of the interface
  relationships of the specific end item with other
  interfacing items
• Establish producibility of the selected design
• Establish the operability of the selected design
• Assess compliance with reliability and system
  safety requirements
• Establish the feasibility of the approach
• Address status, schedule and cost relationships.

16
When is a Review Complete?

• Reviews are considered complete when the
  following is accomplished
• Agreement exists for the disposition of all
  Review Item Discrepancies (RID) and Request for
  Actions (RFA).
• The review board report and minutes are complete
  and distributed.
• Agreement exists on a plan to address the issues
  and concerns in the review boards report.
• Agreement exists on a plan for addressing the
  actions identified out of the review.
• Liens against the review results are closed, or
  an adequate and timely plan exists for their
  closure.
• Differences of opinion between the project under
  review and the review board(s) have been
  resolved, or a timely plan exists to resolve the
  issues.
• A report is given by the review board chairperson
  to the appropriate management and governing
  program management committees charged with
  oversight of the project.
• Appropriate procedures and controls are
  instituted to ensure that all actions from
  reviews are followed and verified through
  implementation to closure.

17
Example Agenda for a Project System Design Review
by a Standing Review Board (SRB)

• Purpose of Review Charge to SRB SRB Chair
• Project Overview Status Project Manager
• System Engineering Status Project SE
• Requirements VV plans
• Trade studies
• Technical margins
• WBS-level 2 Design State Status for each
  area WBS Managers
• System Design
• Key Requirements
• Trade Studies
• Technology Readiness
• Acquisition Strategy Long Lead
• Logistics Facilities
• Challenges Risks
• Integrated System (e.g., power) State Status
  for each area Discipline Leads
• Integration Test Integration Manager
• Safety Mission Assurance (SMA) SMA Manager
• Human Rating Project HR Rep
• Risk Risk Manager

18
Pause and Learn Opportunity

• Student role-play
• You are the chief systems engineer for a New
  Frontiers-class mission to Europa. Your PDR is
  scheduled in 3 months.
• What do you do?
• What do you ask the development team to do?
• What benefits would you expect from having a PDR?
  
• How might it waste time?
• Or
• Read and discuss the Crosslink article The Role
  of Independent Assessments for Mission Readiness
• (Crosslink_Independent Review.pdf)

19
Module Summary Technical Reviews

• Technical reviews are key development milestones
  used to measure progress, assess project maturity
  and to infuse lessons from the past. They
• Provide confirmation of completed effort and
  readiness to commit additional resources for the
  next phase.
• Encourage and establish project discipline with
  an integrated project team perspective.
• Identify risks and review mitigation options.
• Describe plans and priorities for the next phase.
• There are 11 reviews in the minimum set of
  technical reviews for a NASA robotic mission.
  These reviews cover the entire mission life
  assessing the concepts and designs early
  readiness for test, flight and operations in
  mid-life and plans for disposal at the missions
  end.
• These reviews are held to demonstrate that the
  appropriate products, accomplishments and plans
  have been completed before proceeding to the next
  phase.
• Appropriate products, accomplishments and plans
  are based on the lessons of hundreds of past
  projects (best practices).

20
Backup Slidesfor Technical Review Module
21
Major Project Reviews Precede Each Key Decision
Point
FORMULATION
IMPLEMENTATION
A
B
C
D
E
F
Pre-A
Concept Technology Development
System Assembly, Test, Launch
Concept Studies
Preliminary Design Technology Completion
Final Design Fabrication
Closeout
Operations Sustainment
Project Phases
C
D
E
B
A
F
Key Decision Points
Mission Concept Review
Systems Requirements Review
Major Reviews
Mission/System Definition Review
Preliminary Design Review
Critical Design Review
Systems Integration Review
Independent Cost Estimates
Operational Readiness Review
Flight Readiness Review
Post Launch Assessment Review
Decommissioning Review
22
Purpose of Technical Reviews

• A technical review is an evaluation of the
  project, or element thereof, by a knowledgeable
  group for the purposes of
• Assessing the status of and progress toward
  accomplishing the planned activities.
• Validating the technical tradeoffs explored and
  design solutions proposed.
• Identifying technical weaknesses or marginal
  design and potential problems (risks) and
  recommending improvements and corrective actions.
• Making judgments on the activities readiness for
  the follow-on events, including additional future
  evaluation milestones to improve the likelihood
  of a successful outcome.
• Making assessments and recommendations to the
  project team, Center, and Agency management.
• Providing a historical record that can be
  referenced of decisions that were made during
  these formal reviews.
• Assessing the technical risk status and current
  risk profile.

23
Need for Management Reviews

• The progress between life-cycle phases is marked
  by key decision points (KDPs). At each KDP,
  management examines the maturity of the technical
  aspects of the project. For example, management
  examines whether the resources (staffing and
  funding) are sufficient for the planned technical
  effort, whether the technical maturity has
  evolved, what the technical and non-technical
  internal issues and risks are, or whether the
  stakeholder expectations have changed. If the
  technical and management aspects of the project
  are satisfactory, including the implementation of
  corrective actions, then the project can be
  approved to proceed to the next phase.

24
Blue text indicates changes from last update