EE421

1
EE-421

• TEST AND EVALUATION
• FOR
• SUCCESSFUL PROJECTS

2
(No Transcript)
3
Test Evaluation Process
PRE-TEST ANALYSIS EVALUATION PLAN
ID CRITICAL ISSUES AND DATA RQMTS
ISSUE TO BE DECIDED
E X P E C T E D O U T C O M E S
STEP 1
STEP 2
MILITARY NEEDS ORD ANALYSIS OF
ALTERNATIES PROTOTYPES SYSTEM DESIGN SYSTEM
SPECIFICATION PRODUCTION ARTICLES
HW/SW THREATS/TARGETS SOFTWARE UPDATES MAJOR
MODS
F E E D B A C K S
TEST PLANNING RUN SIMULATION CONDUCT
TEST GATHER DATA ANALYZE DATA TEST REPORT
STEP 3
BALANCE TE RESULTS WITH OTHER PROGRAM INFO
POST-TEST SYNTHESIS COMPARE RESULTS/PREDICTIONS EV
ALUATION REPORT
DECISION
STEP 4
STEP 5
(Modeling Simulation used throughout)
4
SYSTEM ENGINEERING PROCESS
P
R
O
C
System Analysis
E
and Control
S
(Balance)
S
Requirements

Analysis
I
N
Requirements
P
Loop
Functional Analysis/
U
Allocation
T
Design
Loop
VERIFICATION
Synthesis
PROCESS OUTPUT
5
TEST RESOURCES
ENGINEERING MANUFACTURING DEVELOPMENT
PROG DEFINITION RISK REDUCTION
CONCEPT EXPLORATION
PRODUCTION, FIELDING/ DEPLOYMENT AND
OPERATIONAL SUPPORT
MODELING AND SIMULATION
MEASUREMENT FACILITY/LABORATORY
INTEGRATION LABORATORY
HARDWARE IN THE LOOP
INSTALLED SYSTEM
TEST/TRAINING RANGE
6
MODELING SIMULATION

• CONCEPT
• MODEL - REPRESENTATION OF SYSTEM OR PROCESS
• SIMULATION - APPLICATION THROUGH TIME
• INCREASED EMPHASIS
• REDUCE TIME COST OF WEAPON DEVELOPMENT
• MS CAN BE USED IN ALL PHASES OF DTE AND OTE
• METHODOLOGY MODEL - TEST - MODEL

7
STEP(SIMULATION, TEST AND EVALUATION PROCESS)
TEST - FIX - TEST
(YESTERDAY) MODEL - TEST - MODEL (TODAY)
STEP CONCEPTS 1. first MODEL 2. next
SIMULATE 3. then TEST 4. now ITERATE test
results back into model. RESULTS 1. BETTER
UNDERSTANDING OF SYSTEM PERFORMANCE 2.
BETTER MODELS (WITH POTENTIAL REUSE IN OTHER
PROGRAMS
8
TEST EVALUATION CHECKLIST

• TAILOR TE FOR SPECIFIC APPLICATION
• TAKE ADVANTAGE OF PREVIOUS TESTING
• DOCUMENT TE
• EXAMINE AND UNDERSTAND
• OPERATIONAL REQUIREMENTS
• OPERATIONAL ENVIRONMENT
• SUPPORT REQUIREMENTS

9
TESTER
PROGRAMMANAGER
HEAD-QUARTERS
USER
10
TESTER
PROGRAMMANAGER
HEAD-QUARTERS
USER
11
TESTER
PROGRAMMANAGER
HEAD-QUARTERS
USER
12
TESTER
PROGRAMMANAGER
HEAD-QUARTERS
USER
13
TYPICAL TEST SEQUENCE
INTEGRATION TEST
COMBINED TEST
DEVELOPMENT
DTE

• PROTOTYPE
• LAB TEST
• QUALIFICATION
• SUBSYSTEM TEST

OTE

• FULL SCALE DEVELOPMENT
• FLIGHT TEST
• LOGISTICS
• SYSTEMS
• MEET SPEC

• PRODUCTION
• FLIGHT TEST
• SUITABILITY
• UTILITY

MODELS AND SIMULATION USED THROUGHOUT
14
MAJOR TE PLANNING TASKS

• EVALUATION PLAN
• REQUIREMENTS
• OBJECTIVES AMENABLE TO EVALUATION
• DATA COLLECTION
• DATA REDUCTION/ANALYSIS
• TEST PLAN
• TEST REQUIRED TO COLLECT DATA
• INSTRUMENTATION REQUIREMENTS
• LOGICAL/INCREMENTAL APPROACH
• TEST RESOURCES REQUIRED
• MODELING AND SIMULATION
• PLAN FOR VVA
• THREAT SYSTEMS AND TARGETS
• PLAN FOR VVA

15
MAJOR TE TASKS(EVALUATE/REPORT)

• TEST AND EVALUATE
• CONDUCT TESTS AND COLLECT DATA
• PERFORM EVALUATION TO PRODUCE RESULTS,
  CONCLUSIONS, AND RECOMMENDATIONS
• REPORT
• TEST REPORT TELLS HOW TEST WAS CONDUCTED AND
  PROVIDES TEST RESULTS
• EVALUATION REPORT DESCRIBES ANALYSIS PROCESS AN D
  PROVIDES RESULTS, CONCLUSIONS, AND
  RECOMMENDATIONS
• TEST AND EVALUATION REPORT MAY BE COMBINED INTO
  ONE REPORT

16
MANAGEMENT PRINCIPLES - TEST TEAM

• IT TAKES A TEAM TO TEST
• BUILD A GOOD TEAM PRIOR TO TEST
• GET PEOPLE DEDICATED TO TEST TEAM
• GET TEST TEAM WORKING TOGETHER
• PERFORM NEEDED TRAINING/PRACTICE
• DE-CONFLICT OTHER TASKS DURING TEST
• USE TEST IPT CONCEPT

17
PLANNING PRINCIPLES - GENERAL

• DEFINE DATA REQUIREMENTS FOR EACH
  ISSUE/OBJECTIVE/MOE/MOS/CTP
• DEFINE ACCURACY REQUIREMENTS FOR DATA
• DESIGN TEST EVENTS TO GET REQUIRED DATA
• USE MODELING SIMULATION TO PREDICT REFINE
  TEST POINTS
• VALIDATE PRIORITIZE TEST EVENTS
• IDENTIFY RESOURCE REQUIREMENTS
• FACTOR IN CONSTRAINTS (RESOURCES, TIME)
• WRITE DETAILED TEST PLANS

18
...of course, you must punch airholes in the
hamsters box. However, you must do this before
placing a hamster inside!
SAFETY REVIEW BOARDHAMSTER TEST
19
TEST METRICS

• NEED A GOOD TEST PROGRESS METRIC
• HOURS OPERATED NOT SUFFICIENT
• NUMBER OF EVENTS COMPLETED NOT SUFFICIENT
• NEED TO MEASURE
• HOW MUCH HAS BEEN SUCCESSFULLY COMPLETED
• HOW MUCH REMAINS TO BE DONE
• HOW IMPORTANT/DIFFICULT/TIME CONSUMING THESE ARE

20
Hey, a schedule is aschedule! Dontcome to
me nowwith technicalproblems!
21
Someday, the Test Engineer Will Try to Kill You!!!
Here, Fifi! Cmon!... Faster, Fifi!
22
SUMMARY

• TEST PLANNING MUST START IN PHASE 0
• EARLY PLANNING IS KEY TO SUCCESSFUL TEST
• USE TEST IPTS TO GET EVERYONE INFORMED INVOLVED
• TEST PROCESSES AS WELL AS PRODUCTS
• MAKE CONTINGENCY PLANS
• USE APPROPRIATE TEST METRICS
• CONDUCT TEST IN DISCIPLINED CONTROLLED MANNER