Revitalization of Systems Engineering

1
Revitalization of Systems Engineering

• Kristen Baldwin
• Acting Director, Systems and Software Engineering
• Office of the Deputy Under Secretary of Defense
• (Acquisition and Technology)
• presented to
• NDIA Manufacturing Division Meeting
• August 13, 2008

2
ATL Organization
USD, Acquisition Technology Logistics
DUSD, Acquisition Technology
Dir, Joint Advanced Concepts
Dir, Systems and Software Engineering
Dir, Portfolio Systems Acquisition
Defense Acquisition University
Industrial Programs
Defense Procurement and Acquisition Policy
Defense Contract Management Agency
Small Business Programs
3
Briefing Outline

• Where is DoD Systems Engineering now?
• SE Revitalization Status
• Analysis of Failure
• Key near-term initiatives
• Where does SE need to be?
• Strategic Direction
• Enhanced SE

Institutionalize SE
Key Initiatives
Revitalize SE
2003
2015
2010
2008
4
SSE Functions

• Acquisition Program Support
• Decision Support to Senior DoD Leadership
• Mentoring Acquisition Programs
• Technical Reviews of Key Program Documentation
• Policy, Guidance, Best Practices
• Acquisition Workforce Education and Training
• SE, Test, Production Quality and Manufacturing,
  Software
• Systemic Root Cause Analysis
• Discipline Expertise
• Risk ? CMMI ? Modeling/Simulation
• Reliability ? Software ? Assurance/Cyber
• Architectures ? Energy ? Comm/Networking
• Test/Eval ? Safety ? System of Systems

5
Revitalizing DoD SE

• Issued DoD-wide SE policy focused effort on up
  front, sound technical planning
• Issued guidance on SE, test and evaluation (TE)
  and software
• Revised SE TE curricula at Defense Acquisition
  University
• Established SE Forumsenior-level focus across
  DoD
• Instituted system-level assessments in support of
  OSD major acquisition program oversight role
• Executed 83 reviews since March 2004
• Integrated, software, system assurance, energy,
  MS and system of systems into SE revitalization
  efforts
• Established closer working relationships with
  industry and academia

Prescribing Good Practices Is Not Enough
6
Analysis of Program Failure

• Acquisition cost growth over 11 years (SAR data
  FY 19952005)
• Estimation changes 201B
• Engineering changes 147B
• Schedule changes 70B
• Over the past 10 years, DoD systems have
  experienced a 33 cost growth due to RDTE
  mistakes
• DoD IOTE results, FY2001-2007
• 35 systems mix of ACAT II, 1C, 1D across 3
  Services
• 57 of programs found to be Not Suitable or
  Effective
• Approx. 49 were deemed Not Suitable or
  partially NS
• Approx. 34 were deemed Not Effective or
  partially NE

7
Analysis of Program Failure

• Specific Causes of Program Failure
• (Systemic Root Cause Analysis findings)

We do not start programs right Insufficient requirements analysis and definition at program initiation Lack of rigorous SE approach Optimistic/realistic reliability growth not a priority during development Inadequate software architectures, design, development discipline, and organizational competencies We do not manage programs right Insufficient trade space Insufficient risk management Inadequate IMP, IMS, EVMS Most programs lack quantifiable entrance/exit criteria Maturing suitability (e.g., RAM) is not always a priority
8
Key Initiatives2008-2009
9
SSE FY08/09 Thrust Areas

• Systemic Root Cause Analysis the collection and
  analysis of systemic program performance issues,
  determination of root causes, and development of
  corrective action
• System Assurance the reduction of vulnerability
  to malicious intent in our systems, considering
  the full spectrum security of information,
  technology and hardware/software components
• Software Engineering Competency the focus on
  software engineering as a critical element of
  complex systems acquisition, and strategic
  initiatives to ensure future Defense software
  demands can be met by government and industry
• Systems of Systems augmenting acquisition and
  engineering practices to better plan, develop,
  and manage interdependent systems of systems
• Early Systems Engineering the institution of
  technical management and engineering practices
  prior to program initiation (MS B) in order to
  enable risk informed, balanced acquisition and
  budgeting decisions

10
SSE FY08/09 Thrust Areas (contd)

• Revitalizing Developmental Test Evaluation
  comprehensive strategy for early TE involvement
  and integrated testing to manage technical risks
  across the acquisition system, sustaining systems
  and capabilities, and rebuilding government TE
  expertise.
• System Engineering Research UARC establishment
  of a system engineering research program
• Energy reducing DoD energy consumption across
  our operations, to include force development,
  deployment, and support
• Reliability ensuring rigor, discipline and an
  integrated approach in systems engineering
  practices and TE strategies for improving the
  reliability of systems
• Safety integrating safety process advances into
  Department acquisition policies
• Modeling and Simulation ensuring rigor in MS
  policy and guidance to advance the
  state-of-practice execute acquisition MS Master
  Plan

11
Manufacturing Readiness Levels

• MRL/EMRL Background
• MRLs documented by a body of knowledge, the MRA
  Deskbook and piloted in selected Air Force
  acquisition programs
• EMRLs used as exit criteria on ACAT ID programs
  since 2002
• ODDRE and OSD SSE collaborating to
• Transition from EMRL to robust MRL body of
  knowledge
• Align metrics with new DoD life cycle management
  framework
• Leverage acquisition milestones and SE technical
  reviews to assess manufacturing maturity
• Draft policy to integrate manufacturing into
  acquisition decisions
• Doubtful that it will go into the pending DoDI
  5000.02 update
• ACTION OSD SSE has requested the NDIA
  Engineering and Manufacturing Committees to
• Review MRL criteria and its time phasing across
  the acquisition lifecycle
• Better define production relevant and
  representative environments
• Assess use of MRAs/Production Readiness Reviews
  on supplier base
• Provide comments on the MRA Deskbook

12
Manufacturing Readiness Levels (cont.)

• Goals
• Increase emphasis on manufacturing throughout the
  acquisition lifecycle
• Ensure considerations of manufacturing and
  producibility through established service
  mechanisms
• Revised DoDI 5000.02 mandates consideration of
  manufacturing maturity during each acquisition
  phase and at each milestone
• Areas to continue to work
• Appropriate assessment of manufacturing at MS A
  and MS B
• Ensuring manufacturing assessments are not
  stand-alone
• Update engineering training guidance
  accordingly

13
Integration Readiness Levels

• Background
• Systemic analysis has shown integration issues in
  45 of PSRs
• Contributes to cost and schedule issues
• DUSD(AT) requested SSE study and develop
  integration readiness levels for use in
  acquisition
• SSE initiating a collaborative effort to develop
  quantifiable integration metrics
• Product Draft set of IRLs in 120 days
• ACTION
• Identify existing IRL activities/products and
  interest to support comprehensive study

14
Human Systems Integration
Hardware
Total System Approach
Software
Human
Human Systems Integration (HSI). The
interdisciplinary technical and management
processes for integrating human considerations
within and across all system elements an
essential enabler to systems engineering
practice.
15
Recent HSI Guidance and Direction

• H.R. 1585 National Defense Authorization Act for
  FY 2008 Section 231
• Directs the Secretary to (1) coordinate and
  manage human systems integration activities
  throughout DOD acquisition programs and (2)
  designate a senior DOD official to be responsible
  for such effort.
• USD(ATL) memorandum, Apr 3, 2008
• Designates DUSD(AT) as the senior official
  responsible for coordination and management of
  HSI activities throughout DoD acquisition
  programs, with DUSD(ST), Director, Biological
  Systems as the co-lead for ST
• FY09 HASC language
• directs the USD(ATL) to develop a comprehensive
  plan for funding and implementing HSI through all
  phases of science, research, and acquisition.
  Report due by March 15, 2009
• ACTION We are soliciting input from DoD and
  Industry partners to help respond to
  Congressional Direction. Specifically
• What are key HSI activities, initiatives,
  capabilities that exist today
• Where gaps or issues exist

16
Our Challenge Institutionalization2010-2015
17
SSE Strategic Direction

• Cut cycle time to field capability by embedding
  systems engineering engagement with requirements
  and early phase acquisition
• Change engineering culture, both vertically and
  horizontally, across Government and industry
• Implement Human Capital Strategy to assure DoD
  workforce competency
• Advance Engineering State-of-Practice through
  research transition
• Implement comprehensive solution for system
  assurance/program protection/cyber security

18
Improve Knowledge through Technical Foundation
MDD
MS A
MS B
Business Decisions
Agreement to pursue a materiel solution
Selection of a preferred solution
Engineering Support
Preferred System Analysis
National Research Council
Formal Program Start
Uncertainty
Preferred System Concept
PDR
Technology Maturation And Prototyping
Pre-Milestone A and Early-Phase Systems
Engineering Jan 2008
System Level Specs
AoA
CDR
Preliminary Design
Completed Design
Technology Development
Material Solution Analysis

• Systems Engineering is effective when it informs,
  
• and is informed by, other Acquisition process
  owners

19
Questions?
20
Systems Engineering Research (SER) University
Affiliated Research Center (UARC)

• Overview SER UARC implementation is intended to
  continue efforts to advance practice of SE across
  DoD enterprise
• Approved by the DDRE, October 2007
• Committed funding of 10M over 5 years
• Mission Research and analyze advanced and
  emerging systems engineering practices and
  relevant technologies to address the full
  spectrum of DoD systems across the Department,
  from capability areas, enterprise systems,
  systems of systems, and interoperability down to
  subsystems and configuration items with the goal
  of ensuring consistency and systems engineering
  excellence throughout the acquisition cycle.
• Status Request for Proposal nearing completion
• All domestic institutions with engineering
  programs notified (gt700)
• Draft RFP reviewed by over 30 universities
• Key Dates (planned)
• RFP release May 2008
• Contract award July/August 2008

21
Initial SE Research Tasks

• Examine and recommend areas for advancing current
  SE methods, processes, and tools as they are
  applied across the DoD acquisition life cycle
  focusing on three different development
  environments individual weapons systems, SoS,
  and network-centric systems
• Characterize SE effectiveness within the context
  of DoD acquisition and identify methods of
  measurement suitable for application to project
  execution organizations (e.g., defense
  contractors), program management organizations
  (e.g., project managers and program executive
  offices), and oversight organizations (e.g.,
  OUSD(ATL))

22
SoS SE Guide (SEG) v 1.0 Revision

• Draft SoS SEG version 0.9 released December 2006
• March September 2007 -- Pilot Phase
• Structured review of version 0.9 with SoS SE
  practitioners and researchers
• Results offer boots on the ground basis for
  guide revision
• Draft SoS SEG version 1.0 released for comment
  December 10, 2007
• Available at http//www.acq.osd.mil/sse/ssa/guida
  nce.html
• Formal comment acceptance closed on February 14,
  2008
• 600 comments were adjudicated by April 1
• Final release of version 1.0 by June 2008

23
View of SoS Systems Engineering
Assessing SoS Performance
SoS
Systems
Multiple, possibly concurrent increments
24
DoD Software Core Competency

• DUSD(AT) vision for software engineering (SWE)
  leadership and world class capability
• Issue Software Growth
• Software Requirements Growth 80 of system
  functionality CSIS
• Software Size Growth 20 million ESLOC CSIS
• Software Overruns 50 of SW efforts Standish
  Group
• Issue SE and SW have not been well integrated
  on projects
• Top Software Issues and Project Areas

Requirements SE/SW Integration Human Capital
SW Sustainment SW Testing SW Quality Metrics
SW Estimation
25
System Assurance

• We continue to be concerned with assurance of our
  critical DoD assets
• Critical information
• Critical technologies
• Critical systems
• Observations
• Increasing numbers of network attacks (internal
  and external to DoD)
• Broader attack space
• Trends that exacerbate our concerns
• Globalization of our contracts, expanding the
  number of international participants in our
  system developments
• Complex contracting arrangements that further
  decrease transparency below prime, and visibility
  into individual components

These trends increase the opportunity for access
to our critical assets and for tampering
25
26
System Assurance for the Program Manager
Lack of focused guidance to counter the threat
27
Acquisition Path Forward
Raise the bar Awareness - Knowledge of the
supply chain - Who has access to our critical
assets Protection - Protect critical assets
through security practices - Design our systems
for assurance

• Create a framework to integrate multiple security
  policies and oversight
• Leverage Program Protection requirement for all
  acquisition programs as set by DoDD 5200.39
  policy
• Integrate all assurance oversight, planning, and
  risk mitigation activity at the system level
• Develop Guidance on Engineering for System
  Assurance
• Guidebook on Engineering for Assurance for
  program managers/engineers
• Defines how assurance can be incorporated into
  system engineering and design
• e.g. Isolation, Redundancy, Quality, and Fault
  Analysis

28
Leadership Direction for SE

• ATL Source Document Objectives
• Capabilities Balance and rationalize
  requirements Review key capability areas to seek
  greater efficiency Evaluate mission capability
  gaps against technology opportunities
• Programs Accurately price programs and insist
  schedule and budget reflect realistic pricing,
  recognizing technical and integration risks
  Build prototypes and conduct experiments to
  provide options Arm the Program Manager with
  tools
• People Play an appropriate stewardship role for
  the science and engineering community
• Strategic Objectives identified by the SSE Forum
• Enhance SE pre-MS B to ensure programs succeed
• Transform Architecture to support Systems
  Engineering Enterprise
• Establish a Human Capital Strategy for SE
• Assess needs for additional SE Tools

We believe we have gone too far in reducing
Systems Engineering capability we need to
re-build. Honorable Secretary England, 7
November 2007, PEOSYSCOM Conference Keynote
29
SSE FY10-15 Program Outcomes (1/2)

• Create enhanced Systems Engineering capability
  (pre-MS A and between MS AB)
• Provide SSE resources (support, tools) for
  Technology Development phase
• Integrate SSE with Requirements, Programming and
  Management
• Support pre-program and capability analyses
  (AoAs)
• Establish ATL matrix support to CPMs and SoSs
• Develop a Human Capital Strategy for System and
  Software Engineering (SSE)
• Advance Systems Engineering state of practice
  through research by leveraging and transitioning
  UARC recommendations
• Revitalize Developmental Test and Evaluation to
  support faster fielding and reliability growth
  testing

30
SSE FY10-15 Program Outcomes (2/2)

• Transition and promulgate SSE tools, practices
• Institutionalize PSRs, SEPs, Software,
  Reliability and TE practice, etc. among
  Services, Agencies, Industry Transition Partners
• Transform lessons learned to predictive measures
• Transparent data, correlate program findings with
  external databases (e.g., DAMIR, PoPs, PAE)
• Implement corrective actions
• Value Energy in acquisition processes
• Institutionalize energy in Milestone reviews and
  portfolio management
• Ensure fuel demand considerations in AOA and all
  program analysis
• Integrate assurance and security practice into SE
• Transition from FY08/09 cyber and system
  assurance pilot activities