The Wrong and the Right Way to Integrate Hardware, Software, and People

1
The Wrong and the Right Way to Integrate
Hardware, Software, and People

• Arthur Pyster
• Senior Vice President and
• Director of Systems Engineering and Integration
• March 15, 2006

2
Topics

• SAIC Profile
• Illustrative Examples of Wrong and Right Way to
  Approach Building and Integrating Systems with
  Significant Hardware, Software, and People
  Requirements
• Wrong
• Standard Terminal Automation Replacement System
  (FAA)
• Delphi Enterprise Resource Management System
  (DOT)
• Right
• Common Driver Trainer (Army)
• Mobile Inshore Undersea Warfare (Navy)
• Four Strategies and Their Rationale
• Conclusions

3
SAIC Profile

• Homeland Security Co. of the Year Frost
  Sullivan May 2005
• 1 IT Contractor at OSDINPUT January 2005
• 8 Largest DoD ContractorDoD January 2005
• 2 Federal IT Contractor INPUT April 2005
• 2 Top Systems Integrator Federal Computer Week
  September 2005
• 3 Professional Services Firm FederalTimes.com
  June 2005
• 3 Technology Contractor Government Executive
  August 2005
• Leadership Top IT Outsourcing Vendor In North
  AmericaMeta Group METAspectrum January 2004
• Leadership Desktop and Help Desk Outsourcing in
  North AmericaGartner Magic Quadrant December
  2004
• 3 Top Systems Integrator in North America
  Gartner Dataquest August 2005
• Americas Most Admired Companies in Computer and
  Data Services FORTUNE 2001-2005

O u r V i s i o n Be a leading systems and
solutions company, solving our customers most
important business and mission-critical problems
through innovative applications of technology and
domain knowledge From Science to Solutions

• O u r S u c c e s s e s
• More than 3 decades of continuous revenue growth
• 7.2 billion in annual revenues for FY05
• FORTUNE 500 company - 276
• 15.5 revenue CAGR over last 5 years
• Superb staff of qualified professionals
• More than 43,000 personnel worldwide
• 11,000 employees with advanced degrees
• 20,000 with security clearances
• Key positions on programs of national importance
• Including DoD transformation, border security,
  intelligence analysis, cancer research and other
  national priorities
• More than 10,000 active contracts
• Leading provider of contracted RD services
• 3 billion in cash to support growth

4
STRATEGIES for Effective Hardware,Software, and
People Integration
Generally, nothing flashy is required. Follow
modern systems engineering practices and you will
succeed.
Strategy 1. Keep users involved
throughout. Strategy 2. Embrace and manage the
business process reengineering that is inevitable
when large critical systems are
replaced. Strategy 3. Anticipate the next system
you will build and incrementally create a product
line architecture so that the next system (and
ones beyond it) can be more rapidly constructed
and fielded. Strategy 4. Build a little, test a
little, field a little. Do it often.
5
First, the wrongs System developments
that used poor strategies
6
STARS Overview
The primary automation to control an airplane
when it is beyond an airport and lower than
18,000 feet
1
CUSTOMER
A perfect example of how not to succeed
FACT 2,000,000 passengers are handled by TRACONS
every day
7
Picture of Early ARTS, the STARS Predecessor
8
Context in Which STARS was Created

• From early 1980s through early 1990s, FAA spent
  4B to modernize air traffic control Advanced
  Automation System
• One of the most studied examples of disastrous
  large-scale acquisition, systems engineering, and
  software engineering
• Big bang replacement planned for most legacy
  systems controlling air traffic failed because
  problem was much too complex, too poorly
  understood, and too volatile
• In 1996, new FAA acquisition management system
  put in place separate from Federal Acquisition
  Regulations
• New strategy was set to modernize the National
  Airspace System (NAS) incrementally with multiple
  vendors, multiple acquisitions, driven by
  enterprise architecture

9
STARS Shone Dimly at First

• ARTS system, designed more than 20 years earlier,
  was being used for low altitude air traffic
  control across U.S.
• In 1996, Raytheon was awarded 960M contract to
  replace ARTS.
• STARS was billed as a quick delivery system
  because it would mostly be COTS with rollout
  complete in 2004 or 2005.
• Raytheon was to largely reuse system they had
  deployed for Norway.
• First rollout was to be in Boston in 1998.
• Air traffic controllers would not accept
  capabilities and demanded extensive
  customization. FAA management accepted demands
  without understanding the implications.
• STARS became major development effort, needing
  hundreds of thousands of lines of new code. Cost
  escalated to 1.4B. (Now up to 1.7B)

10
User Interface Morass

• FAA requirements team, systems engineers, and
  acquisition team were from different
  organizations and didnt come together into an
  effective integrated team.
• Controllers had no effective way to articulate
  their requirements.
• FAA did not have a good way to elicit, negotiate,
  or manage requirements.
• There was little human factors expertise on the
  FAA STARS team or, more broadly, within the FAA.
• Controllers made demand after demand for features
  without a good process for either side to
  understand the consequences.
• Engineers specified a controller interface that
  was radically different from what controllers
  wanted and were used to.

11
User Interface Resolution

• Unlike ARTS, new display had no shortcut
  switches and knobs.
• Everything in STARS was to be controlled through
  a trackball and keyboard it felt more like a
  Windows PC interface.
• Drop-down menus cluttered screen, blocking
  controllers view of air traffic.
• Controllers union went to Congress and the press
  for changes.
• After much finger-pointing between controllers
  union and FAA, the FAA adopted new approach led
  by their new Chief Scientist for Human Factors.
• New process included prototyping, impact studies,
  method to resolve conflicts between controllers
  and program, .
• FAA Administrator stayed personally involved in
  regular STARS meetings for next several years.

12
Picture of Early STARS
13
STARS Backend Morass

• In addition to radically revising the frontend
  user interface, the FAA faced numerous new
  requirements from controllers and technicians for
  new backend features around improved robustness,
  maintenance, and performance.
• Major rework of the backend processing system
  would take years to complete.
• Pressure mounted to get something in the field
  quickly, but the STARS lifecycle was waterfall
  development and incremental deployment.

14
Backend Recovery

• New lifecycle was defined, enabling incremental
  capability delivered to field.
• FAA Administrator began to talk about build a
  little, test a little, deploy a little.
• STARS Early Display Configuration (EDC) defined
  to have old ARTS backend with the new display
  front-end.
• STARS Full Scale-1 (FS-1) and Full Scale2 (FS-2)
  defined incremental backend capability.
• By 2001, STARS EDC was deployed at two low-volume
  TRACONS El Paso and Syracuse. Recall original
  roll out for STARS had first deployment at Boston
  one of the busiest TRACONS in the country.

15
Meanwhile

• Legacy system ARTS, that STARS was to replace,
  kept advancing under a contract with Lockheed
  Martin.
• Common ARTS began to offer many of the advantages
  that STARS was to offer, including large color
  screens.
• Critics begin to question whether the FAA should
  abandon or cut back on STARS and field more
  Common ARTS systems.
• Eventually, FAA agreed to a hybrid with some
  Common ARTS systems and some STARS systems.
• Today, STARS deployment is proceeding, but wont
  be completed for several more years.
• Bostons STARS system was commissioned in May
  2004.

16
Picture of Common ARTS
17
Delphi Enterprise Resource Planning

• Legacy system DAFIS built in 1970s as custom
  application for Department of Transportation
  (DOT)
• Dozens of bolt-on applications developed over
  the years to compensate for its shortfalls
• Was recently replaced by new Delphi system that
  was based on Oracle Federal Financials.
• FAA is by far the dominant part of DOT, having
  nearly all the complexity because it acquires
  billions of dollars worth of systems annually and
  operates a real-time safety-critical system worth
  more than 25B.
• DOT managed the acquisition of Delphi without a
  good understanding of FAA complexities. The
  primary acquisition leadership largely reflected
  the other parts of DOT that managed grants,
  investigated accidents, kept statistics, etc. The
  gulf in perspectives was large.

18
Business Process Reengineering Misunderstood

• DOT insisted FAA reengineer their business
  processes in order to make effective use of
  Delphi.
• FAA leadership insisted that DOT didnt
  understand the magnitude of the reengineering and
  that DOT wasnt willing to fund it.
• FAA line organizations had operated in a fairly
  decentralized way using DAFIS with lots of
  significant variation between organizational
  elements in how financial and contractual
  functions were performed. FAA line organizations
  did not want to change their ways.
• DOT was under great pressure from OMB to get
  Delphi operational quickly.
• FAA opted to minimize business process change in
  the interests of speedy implementation and
  because reaching consensus about new business
  processes was very hard.
• FAA didnt appreciate they couldnt avoid much of
  the business process reengineering anyhow,
  leading to gross underestimation of the amount of
  time to convert to Delphi, including data
  conversion and user training. Symptoms included
  being behind in paying hundreds of millions of
  dollars in bills.

19
Next, the rights System developments that
followed smart strategies
20
Common Driver Trainer Architecture
Advanced virtual simulators built from a common
architecture used to train Army personnel to
drive many types of vehicles
A model of how to succeed
FACT 272 casualties in operations Iraqi and
Enduring Freedom (through 12/3/05) were caused by
private automobile accidents and vehicle
crashesthe majority in non-hostile conditions.
21
Modular Reconfiguration for Multiple Vehicles,
Scenarios, and Training Objectives
Product Line Architecture describes combined
software/hardware system
Combination of words and pictorials (Depiction of
the project in operation how it satisfies the
shortfall)
Product line includes instructor operator
station, training management system, automated
data collection and after action review
22
Mobile Inshore Underwater Warfare
Provide surface and subsurface surveillance in
shore areas and provide supporting command and
control
FACT MIUW improve-ment and deployment has been
rapid since the attack on the USS Cole in 2000.
23
Mobile Inshore UnderwaterWarfare Installation
24
STRATEGIES for Effective Hardware,Software, and
People Integration
Generally, nothing flashy is required. Follow
modern systems engineering practices and you will
succeed.
Strategy 1. Keep users involved
throughout. Strategy 2. Embrace and manage the
business process reengineering that is inevitable
when large critical systems are
replaced. Strategy 3. Anticipate the next system
you will build and incrementally create a product
line architecture so that the next system (and
ones beyond it) can be more rapidly constructed
and fielded. Strategy 4. Build a little, test a
little, field a little. Do it often.
25
Strategy 1. Keep Users Involved

• STARS failed to systematically involve
  controllers from the beginning in a way that
  built up their confidence and really analyzed
  their needs, how they really operated their
  equipment. This built up animosity and created
  an atmosphere that took years to correct.
• STARS failed to involve human factors experts
  early in the lifecycle and had to recreate the
  user interface later at great expense and time.
• STARS failed to say no to users who wanted
  features but didnt pay for them.
• CDT recognized the gap between SEs who specify a
  system and the actual users. SAIC specifiers are
  usually twice the age of military system users.
  Young soldiers have different mental models and
  handle many tasks simultaneously. Use SEs with
  characteristics of real users where possible.
• MIUW recognized that their users are often
  reservists, who dont get the same level of
  training as regular forces. They needed to make
  it possible to both use and maintain MIUW
  equipment with minimal training.
• MIUW used some of their own technicians in the
  design e.g., technicians suggested that each
  case has storage room for the cables that go with
  that case in order to make setup and tear down
  easier for the user.
• MIUW observed that the resource loaded network is
  a great communications device with the customer.

26
Strategy 2. Embrace and Manage Change

• Delphi demonstrated that putting in place a major
  new system will be disruptive to existing
  business processes.
• COTS systems bring constraints on business
  processes that may clash with old business
  processes.
• Delphi demonstrated that senior leadership must
  embrace the change in business process and then
  manage it.
• Delphi demonstrated that failure to embrace
  change will lead to underestimating the costs of
  training, the time for conversion, and the
  disruption to current business operations.

27
Strategy 3. Anticipate the Next System

• Product line architectures (PLAs) enable speedy
  creation of individual systems.
• There is little time or to create fully
  generic PLAs outside real projects. Product
  line is more a mindset than a toolset any
  standard architectural tool can support this.
• Begin with the order for a single product, but
  anticipate a second product and design an
  architecture for both.
• Since first product is delivered, focus more
  attention on ensuring PLA supports it. Support
  for second product is more notional. Second
  product proves architectural robustness.
• Keep asking how to build next product while
  building current one improve product line
  architecture incrementally.
• Keep line between HW and SW as fluid as possible.

28
Strategy 4. Build quickly. Field often.
Impatience is a virtue. Dave Pratt

• STARS initially tried to get all requirements
  correct and define a big bang delivery. They
  backed into a spiral design with incremental
  delivery. By backing into that strategy, the
  program looked weak. Had they embraced it from
  the beginning, they would have looked smart and
  been smart.
• CDT assumes that they wont get things right at
  the beginning and fields trainers anticipating
  rapid cycles of improvement. Trial and error and
  rapid release is especially important for such
  highly interactive systems as trainers.
• MIUW fielded early prototypes to ensure they
  really understood the requirements and that the
  requirements were really right. They also
  encountered the downside of this approach because
  they had to keep the prototype in the field
  longer than expected and it wasnt ruggedized.
• MIUW used modeling, simulation, quality
  assurance, and testing early and often to keep
  quality high. QA and testing are especially
  important for integration of components from
  others.

29
In Conclusion
Because
The big bang is dead. The waterfall is
dead. Ignored users kill systems. Impatience is a
virtue. Failure to anticipate leads to failure.
You must

• Keep your users close.
• Embrace change.
• Anticipate before you are asked.
• Build quickly, get more right than wrong, fix it,
  repeat.