INTEGRATED SYSTEM EVALUATION PLANNING OVERVIEW

1
INTEGRATED SYSTEM EVALUATIONPLANNINGOVERVIEW
UPDATED As of 26 Jan 05
TEMAC TE Refresher Course
2
Outline

• Introduction
• Value of Integrated System Evaluation
• Integrated System Evaluation Strategy Formulation
• objective
• concept
• focus area examples
• Summary

3
(No Transcript)
4
Objectives

• Overview of Evaluation Planning
• Emphasize whats important

5
Evaluation Take Aways

• Early Evaluation Planning is essential
• Evaluation Strategy drives TE resources and
  execution
• Evaluations Assessments must be
• objective
• balanced
• unbiased
• Evaluation Assessments support both combat and
  materiel development efforts

6
Evaluators RoleEarly Involvement

• Ensure the following in all requirements
  reviews
• Audit trail back to the Mission Need
• Consistency
• Achievability
• Testability
• Measurability
• Adequately address testing in all proposals and
  contracts
• All essential testing in the TEMP
• How Government will witness testing
• Access of all test data and records
• Conduct of assessment conferences
• Corrective action process
• Demonstration of corrective actions

7
Evaluators Role Evaluation Planning

• Lead all aspects of system evaluation planning
• Develop the Integrated System Evaluation
  Strategy
• Represent AEC at the IPT and TE WIPT meetings
• Obtain TE WIPT buy-in of the Integrated System
  Evaluation Strategy
• Resolve TE WIPT concerns relative to system
  evaluation
• Present the Integrated System Evaluation
  Strategy to ATEC leadership
• Provide Integrated System Evaluation Strategy
  input to the TEMP
• Write the System Evaluation Plan
• Update the Integrated System Evaluation
  Strategy, as required
• Update the SEP, as required

8
Evaluators RoleTest/Simulation Execution
Planning

• Ensure that the Test/Simulation Execution
  Strategy supports the
• Integrated System Evaluation Strategy
• Obtain TE WIPT buy-in of the Test/Simulation
  Execution Strategy
• Resolve TE WIPT concerns relative to
  Test/Simulation Execution
• Ensure that all EDPs adequately support the SEP

9
Role of the EvaluatorEvaluation Execution

• Execute the continuous evaluation process
• Provide periodic system assessments to the
  acquisition and combat
• developer communities
• Ensure that all corrective actions are
  identified, fixed, and demonstrated
• Assess the readiness of the system prior to IOT
• Write the System Evaluation Report (SER)
• Assess the readiness of the system for materiel
  release

10
Supporting the Soldier
This is what I need
This is what I built
Program Manager
Combat Developer/ Functional Proponent
Acquisition
Requirements
This is how well it works
Effective Suitable Survivable
Test and Evaluation
Tester Evaluator
10
11
Evaluation Considerations
Hardware
Interoperability
Logistics
Warfighting System
Software
Tactics, Techniques, Procedures
Organization and Tasks
Training
11
12
TE Planning Process
13
(No Transcript)
14
Integrated System Evaluation Effort
15
Developmental TestingNew Paradigm
2012
DTOptimized (CT, GT, Exp., Analysis, LFTE,
MS) dt
2005
To Minimize Acquisition Cycle Time, the Army
Must Optimize Developmental Testing
Primary TE Purpose is to Demonstrate a System
Works
16
Value of Test and Evaluation Concept Refinement
Technology Development Phases

• Key Evaluation Support Objective
• Involvement in development of user requirements
• Assist acquisition community in risk reduction
  through accurate and timely system evaluations
• Early identification of system potential and
  Military Utility
• Key Evaluation Support Contributions
• Develop integrated system evaluation strategy
  early
• Assist in TEMP development
• Provide evaluator expertise to CBTDEV/MATDEV
  studies analyses
• Develop evaluation plans for ACTD/ATD/TD
  demonstrations AWE experiments
• Provide quick evaluation on rapid acquisition
  initiative candidates
• Perform early system assessments on operational
  impact of technical approaches

Early evaluator involvement reduces risk
regarding requirements, technology, and schedule
16
17
Value of Test and EvaluationSystem Development
Demonstration Phase

• Key Evaluation Support Objective
• Optimize evaluations scope and data requirements
  to
• support Acquisition Strategy
• Provide decision makers with identification of
• system performance areas of risk
• Execute corrective action process
• Support LRIP decisions
• Assist in posturing system for successful Ots
• Key Evaluation Support Contributions
• Lead DT and OT input to the TEMP
• Develop System Evaluation Plan (SEP)
• Witness and/or monitor contractor conducted
  testing
• Provide periodic assessments on system
  demonstrated performance
• Assist in identification of system requirements
  that need refinement
• Assess reliability survivability, vulnerability,
  and logistics supportability system capability,
  interoperability, and integration capability

17
18
Value of Test and EvaluationProduction
Deployment Phase

• Key Evaluation Support Objective
• Provide system evaluations that accurately
  reflect system maturity and performance
  capability prior to
• full-rate production decision
• Quick Look Assessment Report
• Support the Materiel Release process
• Key Evaluation Support Contributions
• Conduct system evaluations to estimate mission
  effectiveness, suitability, and survivability
• Evaluate changes in production configuration
• Identify needed changes to logistics support and
  TTPs

19
Value of Test and EvaluationOperations Support
Phase

• Key Evaluation Support Objective
• Provide evaluation of system performance
  resulting from increment changes, P3I, or
  operations in deployed units to determine full
  operational capability
• Key Evaluation Support Contributions
• Conduct system assessment(s) to support Materiel
  Release
• Monitor production acceptance testing and
  evaluate results
• Assist CBTDEV identify and develop changes to
  system requirements

20
(No Transcript)
21
Good System Evaluation Strategy Characteristics

• Means
• Brings about TE community resources,
  understanding, and agreement
• Ends
• System Assessments, reporting periodically on the
  systems development status, readiness for OT,
  and potential mission impacts
• System Evaluation Reports, reporting on the
  Systems Effectiveness, Suitability, and
  Survivability at acquisition milestone decision
  points
• Ways
• A Roadmap of the continuous comprehensive
  evaluation which is
• Adaptable to PM/MATDEV modifications and user
  requirement changes and
• Flexible to accommodate and integrate test
  /simulation events also
• Provides for the integration of data from system
  development demonstration through production
  and deployment and
• Remains stable under inevitable program and user
  changes

Integrated System Evaluation Strategy drives TE
resources, timelines, and data sources!
21
22
Integrated System Evaluation Objectives

• Hardware and software functionality of the
  system works
• System design is technically sound (suitable
  and survivable)
• Critical problems are identified, corrected,
  and demonstrated prior to IOT
• Soldier is integrated into the system, and the
  system is integrated into the unit
• as early as possible during system
  development and development phase
• Potential impacts of the systems capability
  and limitations on the Mission
• Minimize the risks in transitioning the system
  from development to fielding
• Total war fighting system is ready for IOT
• Evaluate the systems effectiveness,
  suitability, and survivability
• Evaluate the systems contribution to enhancing
  and sustaining the Mission

23
Definitions (DA PAM 73-1)

• Mission Effectiveness The capability of the
  operational unit (that is,
• military units and soldiers) to accomplish
  the critical mission tasks required
• to perform its assigned missions, as
  described in the MNS and ORD.
• Capability is the ability of typical
  operators and maintainers to accomplish
• needed critical mission tasks.
• Mission Suitability The design
  characteristics (such as, MANPRINT,
• RAM, integrated logistics, and tactical
  interoperability) needed to enable and
• sustain critical mission task
  accomplishment. Sustainability addresses the
• ability of the system to achieve and remain
  in an operable and committable
• state (that is, operational availability)
  during the course of conducting its mission(s).
• Mission Survivability The design
  characteristics needed to enable systems
• and operational units to avoid, evade, and
  withstand the effects of the threat in order
• to increase mission effectiveness.

24
Mission Oriented Integrated System Evaluation
Concept
Mission Capabilities
25
Integrated System Evaluation Model
Does the System Enhance Mission Capability
Outcomes?

• System Capability
• Effectiveness Requirements
• HW Functionality
• SW Functionality
• Technical Interoperability
• Suitability Requirements
• Soldier Interface
• Training
• Manpower
• Personnel Skills
• Safety Health Hazards

• System Suitability Survivability
• Suitability Requirements
• Reliability, Maint, Logistics
• Manpower Personnel
• Soldier Interface
• Training
• Safety Health Hazards
• Survivability Requirements
• Susceptibility
• Vulnerability
• Recoverability

Does the System Enhance Battlefield Functional
Area Capability Outcomes?
Does the System Enable Battlefield Functional
Area Capability Sustainment?
26
Transition Hypothesis
Integrated System Evaluation

• IF system capability, supportability,
  survivability, and suitability design are
  adequately demonstrated and ensured during
  development
• Then there is potential for success during IOT
• Otherwise fielding needed capabilities to the
  Warfighter is delayed

Decision Risk Alpha - The risk associated with
deciding that the system is not ready for OT,
when it really is ready. Beta - The risk
associated with deciding that the system is ready
for OT, when it really isnt ready.
27
Capability

• Has to be discussed in context of the mission
• Mission Capability to enhance the outcomes of
  mission operations (e.g. increase the combat
  effectiveness of combined arms teams)
• Battlefield Functional Area Capability to
  enhance the outcomes of mission tasks (e.g.
  provide threat indications and warnings)
• System Capability to perform the functions
  designed into the system (e.g. detect, locate and
  identify SIGINT emitters within the corps area of
  interest)
• Has to be defined in detail
• What outcomes are expected?
• Under what conditions and environments?
• What primary measures of goodness will be used?

28
Evaluation Focus

• What are the issues (topics) you want to address?
• Why are these important?
• What do you want to measure?
• Under what conditions do you want to measure it?

System Soldiers with the Equipment and
support, conducting their missions
System gt Equipment
29
System Capability

• Identify and Describe
• Critical systems/sub-systems to be evaluated
• Intended capability of each critical
  systems/sub-system
• Link critical systems/sub-systems to BFA
  Capabilities to be evaluated (e.g. Hardware
  Performance, Software Performance, Technical
  Interoperability, Human Factors Design)
• Measures of Performance
• Environments, factors, conditions, and data
  elements to be collected

30
BFA Capability

• Identify and Describe
• COI and additional issues to be addressed
• The BFA capabilities to be enhanced
• Expected operational outcomes of each capability
  (e.g. task performance, messages sent, reports
  completed, information disseminated, support
  provided, precision fire executed, ballistic
  hits, etc.)
• Link BFA Capabilities to Mission Operations
• MOE (annotate COIC)
• Factors, conditions, and data elements to be
  collected

31
Critical Missions

• The critical tasks and functions that must be
  performed by the soldiers or units to accomplish
  their assigned mission operations
• May be dependent on the Battlefield Operating
  Systems
• Do not specify who or by what means will be
  employed in performing the tasks
• Do not specify how the tasks will be performed
• Should be independent of conditions
• Provide a common language and reference system
  for evaluation of effectiveness

32
System Suitability and Survivability

• Identify and Describe
• Set of systems or sub-systems critical to
  sustaining BFA capability
• Requirements to be evaluated (e.g. suitability
  and survivability)
• MOP annotate Key Performance Parameters (KPPs)
  and Critical Technical Parameters (CTPs)
• Environments, factors, conditions, and data
  elements to be collected

33
BFA Sustainability

• Identify and Describe
• Expected impact on BFA capability outcomes over
  time
• COI and additional issues to be addressed
• MOE (annotate COIC)
• Factors, conditions, and data elements to be
  collected

34
Mission Capability

• Identify and Describe
• COI and additional issues to be addressed
• Types of Mission Operations expected to be
  conducted
• Physical, military, and civil environments
• Mission capabilities to be evaluated
• Expected operational outcomes of the Mission
  Operations
• (e.g., commanders intent or objectives as
  stated in the Operational Order)
• Measures of Effectiveness
• Factors, conditions, and data elements to be
  collected

35
Data Source Matrix (DSM)
Example
EVENTS
EVALUATION FOCUS AREA
ISSUE
PQTC/G
LUT
PVT
AWE
FDT/E
MS
MEASURE
IOT
MOE 1-1 Force Exchange Ratio
Combat Multiplier ?
Mission
P S
Effectiveness
Develop Report on High Value Targets ?
MOE 2-1 Target Acquisition Rate MOE 2-2 Percent
of Reported Targets Engaged
S P P
Mission Task Performance
S S S P
System Effectiveness (Performance)
MOP 1-1 Target Location Error as a function of
flight size
P S S S S
System
Geometric Accuracy?
System Suitability (Reliability)
Demonstrate Reliability Requirement ?
MOP 1-2 Mean time between hardware failures
S S P P
Legend P - Primary Source (Provides All
Essential Data to Answer the Measures) S -
Secondary Source (Supplemental or Acts as a
Backup)

• Shows the contributions of each data source to
  the evaluation focus areas
• Enables proper scoping of the events
• Depicts that each focus area measure must have
  one or more primary data sources

35
36
Evaluation Focus Areas
System Capability Leading to Increased BFA
Capability
Example
Mission Level
System Level

• Hardware functionality
• Technical interoperability
• Software functionality
• Software interoperability
• Software errors
• Software usability
• Software growth
• Lethality

• Can the commander's mission objectives be
  accomplished with a high degree of success?
• Can the mission task objectives, outcomes, or
  products be accomplished with a high degree of
  success?
• What are the critical technical concerns
  affecting mission task accomplishment?
• What are the critical software interoperability
  concerns affecting mission accomplishment?

36
37
Evaluation Focus Areas
System Suitability Leading to Increased BFA
Capability
Example
System Level
Mission Level

• MANPRINT design characteristics
• MANPRINT design concerns
• Tactical interoperability
• Tactics and procedures
• Doctrine

• What are the critical MANPRINT concerns affecting
  mission task accomplishment?
• What are the critical tactical interoperability
  concerns affecting mission task accomplishment?

38
Evaluation Focus Areas
System Reliability Leading to Increased BFA
Sustainment
Example
Mission Level
System Level

• Reliability Characteristics
• Requirements Documents
• Critical Failure Modes
• Corrective Action
• Reliability Growth

• Can the system complete the mission without a
  critical downing event?
• Can the unit complete the mission without
  dropping below a critical readiness state?
• Can the system and unit be sustained during
  continuous operations IAW the OMS/MP?
• What is the impact of the systems reliability on
  meeting the commanders mission objectives and
  mission tasks?
• What types of downing events occur most
  frequently during the mission?

38
39
Evaluation Focus Areas
System Maintainability Leading to Increased BFA
Sustainment
Example
Mission Level
System Level

• Maintainability Concept
• Maintenance burden on manpower and logistics
  supply
• Maintainability Characteristics
• Maintenance Allocation Chart
• validation
• BIT/BITE Performance
• Fault Isolation/Location
• performance
• MANPRINT design
• Maintainability Growth

• Given the maintenance concept, can the system be
  restored to an operationally effective capability
  before mission completion?
• Given the maintenance concept, can the unit be
  restored to an operationally effective readiness
  state before mission completion?

40
Evaluation Focus Areas
Logistics Design Leading to Increased BFA
Sustainment
Example
Mission Level
System Level

• Logistic Concept
• Supply concept
• Logistic Characteristics
• Logistic design
• MANPRINT design
• Integrated Logistics Support Package
• Transportability
• Deployability

• Can the logistic concept enable the system to be
  restored to an operationally effective capability
  before mission completion?
• Can the tactical logistic concept enable the unit
  to be restored to an operationally effective
  readiness state before mission completion?

41
Evaluation Focus Areas
System Survivability Leading to Increased BFA
Sustainment
Example
System Level
Mission Level

• Hardware survivability characteristics
• Susceptibility
• Vulnerability
• Recoverability
• Soldier Survivability
• Information Assurance
• NBC
• EMI
• EW
• Nuclear Weapons Effects
• E3
• Countermeasures

• Can the system complete the mission without a
  critical survivability event?
• How long can the system be expected to
  participate in the mission without experiencing a
  significant survivability event?
• What are the units mission susceptibilities
  vulnerabilities?
• Can the system recover from a from a significant
  survivability event during the mission?
• What types of survivability events occur most
  frequently during the mission?

41
42
Evaluation Focus Areas
System Suitability Leading to Increased BFA
Sustainment
Example

• MANPRINT design characteristics
• MANPRINT design concerns

• What are the critical MANPRINT concerns affecting
  mission sustainment?
• What are the critical MANPRINT concerns affecting
  mission supportability and logistics?

43
Using MS Results in Evaluation
MS Applications

Post-Test
Test
Pre-test
Estimate performance
E
Create loading with
Assess vulnerability
envelope.
n
simulators and
and lethality.
g
Plan rehearse data
stimulators.
Type

r
Examine alternative
acquisition.
g
Drive instrumentation.
environments.
of

Evaluation
Planning.
C
Scenario driver for
Supplement test results.
MS

o
command, control,
m
Develop test
Examine effects of test
b
communications and
scenarios and
limitations on
a
intelligence.
mission profiles.

evaluation.
t
Predict
Observe
Apply
Model
Test
Model
43
44
MS Documentation
AccreditationPlan
AccreditationReport
Defines AcceptabilityCriteria
Provides evidence MS meets Acceptability
Criteria Report to Accreditation Authority
VV Plan
Report signed by the Accreditation Authority
Accreditation Plan and Accreditation
Report Signature Authority is Approver of the
Document in Which the MS is to be Used
44
45
VVA Responsibilities
  Signature authority is the approver of the
document in which MS is used.
46
Summary

• TE Community Must Get Involved Early
• Evaluator Must Know the System as well as the PM
• Create an Integrated System Evaluation Strategy
  that
• Is guided by the ICD, CDD, CPD, and COI
• Includes both mission and system level focus
• Links Mission Capability to BFA Capability and
  Sustainment
• Links BFA Capability to System Capability
• Links BFA Sustainment to System Suitability
  Survivability
• Integrates what is learned from system
  development and demonstration into production and
  deployment
• Remains steady over the acquisition life cycle
• Speaks the users language

47
Request for Feedback
This module is a part of the TE Managers
Committee (TEMAC) TE Refresher Course. Request
your feedback. Please contact TE
Management Agency Office, Chief of
Staff Army 200 Army Pentagon
[phone] DSN 225