Using Observational Study as a Tool for Discovery:

1
Using Observational Study as a Tool for
Discovery Uncovering Cognitive and
Collaborative Demands and Adaptive
Strategies Emilie M. Roth Roth Cognitive
Engineering
2
Objectives of Cognitive Task Analysis (CTA)

• Uncover the cognitive and collaborative
  activities entailed by the field of practice
• Identify opportunities for providing more
  effective support
• new forms of training
• new displays and visualizations
• new decision aids

3
Motivating Questions

• What are the goals of the domain?
• What are the cognitive activities that need to be
  performed and what makes them difficult?
• Monitoring, planning, choosing among alternatives
• What are the sources of complexity (that any
  agent would confront)?
• Time pressure, lack of information, uncertainty,
  goal conflict, risk
• What support can be provided to facilitate
  performance?

4
Two Aspects of CTA

• Uncovering cognitive and collaborative activities
  (Knowledge Acquisition)
• Modeling the results to drive design
  (Knowledge Representation)
• Cognitive Work Analysis (e.g., Vicente, 1999)
• Applied Cognitive Work Analysis (e.g., Elm et
  al., in press)
• ...

The focus of this talk is on knowledge
acquisition
5
CTA Methods Vary in Knowledge Acquisition
Approach

• Observational techniques
• Interview techniques
• Analytic techniques
• Psychometric techniques

The focus of this talk is on field observations
6
Field Observations

• Goal Uncover the actual cognitive and
  collaborative demands of the domain and the
  strategies that domain practitioners have
  developed to deal with those demands.
• Main elements
• Observe domain experts in actual work environment
• Opportunistic interviews -- Ask questions during
  low workload periods
• Use Check List to guide observation and
  interviews

7
Field Observations Serve as Tools for Discovery

• Provide illustrative cases/critical incidents
  that certain situations can arise (A type of
  Existence Proof)
• Stimulate conjectures about general relationships
  that can then be tested under more controlled
  conditions.
• Insure that hypotheses are firmly grounded in the
  field of practice.

8
Illustrative Field Studies

• A field study to understand the existing field
  of practice as a basis for guiding deployment of
  advanced technologies (Railroad Dispatch Center)
• A field study to uncover and document
  unanticipated changes in cognitive and
  collaborative demands that resulted from the
  introduction of new technology into a field of
  practice (Power Plant Control Room)

9
Using Observational Studies to Guide Technology
Deployment

• Uncover and document
• Cognitive and collaborative demands imposed by
  domain of practice
• Adaptive strategies that practitioners have
  developed in response to those demands
• Identify
• Cognitive activities that can be supported more
  effectively
• Features of the existing environment that are
  important to effective performance and should be
  preserved.

10
Changing Cognitive Landscape of U. S. Railroad
Industry

• New display and communication technology (e.g.,
  data link)
• Increases in level automation
• Introduction of high-speed trains

11
Examining Train Dispatching in Todays
Environment

• Examine how experienced dispatchers schedule
  trains and manage track use
• Identify cognitive activities that could be more
  effectively supported
• Identify features of existing environment that
  contribute to effective performance and should be
  preserved when transitioning to new technologies

12
CTA Approach

• Iterative bootstrap approach
• Preliminary field observations in a dispatch
  center
• multiple dispatchers
• multiple shifts (including shift turn-overs)
• Structured Interviews
• Follow-up field observations
• observations at a second dispatch center
• additional observations at original center

13
Guiding Framework

• Identify and document
• Sources of task complexity
• Strategies experienced practitioners have
  developed to cope with task demands
• Opportunities for performance improvement
• Illustrative incidents
• Deviations from canonical

14
(No Transcript)
15
The Train Dispatching Environment

• Train dispatchers are responsible for
• managing track use
• insuring that trains are routed safely and
  efficiently
• insuring the safety of personnel working on and
  around railroad track
• An example of a distributed team planning task
• Multiple train dispatchers handling adjoining
  territories
• Train engineers
• Maintenance of way workers

16
What Makes Train Dispatching Difficult?

• Need to dynamically re-compute train routes and
  meets
• train delays and track outages
• unplanned demands on track usage
• Need to satisfy multiple demands placed on track
  usage
• Heavy attention and communication demands
  (particularly over the radio)

17
Expert Strategies for Coping with Task Demands

• Off-load memory requirements
• Anticipate and plan ahead
• Act proactively
• Level workload

18
Anticipating and Planning Ahead

• Maintain Big Picture/Monitor Activity Beyond
  Own Territory
• Take advantage of the radio party line feature
• Anticipate train delays
• Identify equipment problems
• Listen for/head off potential conflicts
• Listen for mistakes
• Consider what can go wrong and plan for
  contingencies

19
Insights from Field Observations

• Dispatchers have developed a variety of
  strategies that
• smooth the way for trains to pass safely and
  efficiently
• satisfy the multiple demands placed on track
  use
• These strategies depend heavily on communication
  and coordination among individuals distributed
  across time and space.

20
Implications for Introduction of Data Link
Technology

• Confirmed the need to off-load some radio channel
  communication and suggested opportunities for
  data link technology
• electronic transmission of movement authorization
  forms
• Revealed the importance of preserving the
  broadcast/party-line characteristic for some
  types of communication.
• Exploring ways to implement data link systems
  that have broadcast capabilities.

21
Points Ilustrated by Case Study

• Can reveal the sources of task complexity
• Can reveal the knowledge and skills that underlie
  expert performance
• Can reveal opportunities to improve performance
• Can reveal features of the current environment
  that facilitate effective performance and should
  be preserved as new technologies are introduced.

22
Questions?
23
Using Observational Studies to Explore the
Envisioned World

• Introduction of new technology inevitably changes
  the nature of cognitive and collaborative work.
• Some changes explicitly engineered
• Some unanticipated effects
• Observational studies provide a powerful tool for
  exploring the envisioned world
• evaluate validity of designer assumptions
• drive further discovery and innovation

24
Prototype as
Implemented System
CTA
Hypotheses
User in Loop Tests
25
Case Study Introduction of Advanced Human
System Interfaces (HSI) in a Power Plant Control
Room

• Plant recently implemented advanced HSI
• Computer-Based Procedure System
• Advanced Alarm System
• Graphic-based Plant Information Display System
• HSI were in final phases of implementation and
  operators were undergoing training on a
  high-fidelity simulator
• Created opportunity to capture Lessons
  Learned.

26
(No Transcript)
27
(No Transcript)
28
(No Transcript)
29
Objectives of Study

• Capture lessons learned as input to Designers,
  Trainers, and Evaluators of new HSI
• Aspects that were clear improvements over
  traditional control boards
• Aspects that introduced new challenges or raised
  new issues.

30
Issues of Particular Concern

• Impacts on team communication patterns, and
  situation awareness of individual crew members
• Impact on the ability of crews to
• monitor effectiveness of procedures
• detect and respond to situations where the
  actions specified by procedural steps were not
  fully appropriate to the situation..

31
Methodology

• Five crews observed and interviewed during their
  one-week training on a high-fidelity simulator
• Crews included 4 - 6 members
• Each crew observed in four simulated emergency
  events
• Crews interviewed at the end of the two days of
  observations

32
Observations of Simulated Emergencies

• Provided an opportunity to see how crews
  interacted with the new HSI and each other
• Provided an opportunity to observe and document
  illustrative cases that revealed
• the kinds of complexities that can arise
• how the new HSIs impacted the ability of the
  crews to identify and respond to these
  complexities.

33
Crew Interviews

• Provided the operators perspective
• Questions probed impact of new HSI on
• operator workload
• situation awareness
• distribution of tasks and responsibilities among
  team members
• communication and coordination among team members

34
Results

• Observations and interviews provided
  complimentary and reinforcing perspectives
• Overall the new HSI had a positive effect
• expanded the range of data available
• reduced workload associated with gathering and
  integrating plant parameter information
• freed attention resources to more broadly monitor
  plant state.
• Some HSI issues were identified that require
  further exploration.

35
Impact on Team Structure and Dynamics

• The new HSI affected
• The scope and responsibility of different crew
  members
• The communication pattern among crew members
• The situation awareness of different crew members
• Study highlighted the need to train and practice
  communication skills explicitly targeted at
  building and maintaining shared situation
  awareness.

36
Impact on Ability to Monitor and Redirect
Procedures

• Operators were able to utilize the computer-based
  procedures to work through the procedures in pace
  with the events.
• As with paper-based procedures, instances (3
  cases) were observed where the computer-based
  procedure provided misleading information or
  directed the operators down the wrong path.
• In all cases the operators were able to correctly
  detect that the computer-based procedure was
  off-track and redirect it.

37
Implications for Design of Computerized Procedure

• Importance of including features that allow
  operators to
• follow the logic behind the procedure,
• form an independent assessment of plant state,
  and
• over-ride and redirect the procedural path when
  necessary
• Findings highlight the importance of having
• multiple diverse sources of information available
  to operators
• effective communication among the operators

38
Studies Illustrate Multiple Roles of
Observational Studies

• Uncover new cognitive and collaborative demands
  that were previously unanticipated
• Document illustrative cases that provide an
  existence proof that certain situations can
  arise that need to be explicitly considered by
  system designers, trainers, and evaluators
• Provide suggestive evidence that constitute
  hypotheses that can then be examined under more
  controlled conditions.

39
Questions?
40
Field Observation Methods Some Guiding
Principles and Pragmatic Advice
41
Field Observations

• Most effective for uncovering
• Actual demands of the work domain
• (Undocumented) strategies for coping with those
  demands
• Work tempo
• Communication and coordination demands/strategies
• Features of the existing environment that are
  important to effective performance and should be
  preserved.
• Less effective when
• Goal is to get a detailed description of the
  cognitive processes

42
Focus is on Discovery

• Objective is to broaden the set of observations
  and perspectives brought to bear in order to
  maximize the opportunity to uncover interesting
  findings and draw productive conclusions
• broadly sample domain of practice (multiple
  shifts, multiple practitioners, multiple levels
  of experience, multiple sites)
• use multiple converging techniques (field
  observations, structured interviews,
  questionnaires).
• Use multiple observers who are likely to bring
  different perspectives (conceptual frameworks).

43
Test Conjectures to Guard Against Sampling
Error and Bias

• Treat each new observation as an opportunity to
  generate new conjectures as well as test
  conjectures based on prior observations
• Compare insights and perspectives of multiple
  observers
• Solicit feedback from domain practitioners on the
  accuracy of observations and interpretations.

44
Pragmatic Considerations

• Prepare a list of questions/topics to cover ahead
  of time
• Be prepared to adapt on the fly
• Use the list of questions/topics as a checklist
• Be prepared to change questions based on what you
  learn
• Be prepared to change the whole approach if
  necessary (e.g., from observation to interview
  and vice versa)

45
Sample Check List for Field Observations

• Tasks/Standard Practice/Human-Machine interfaces
• Illustrative Cases/ Revealing incidents
• Sources of complexity
• Strategies that domain practitioners have
  developed to cope with complexity
• User-Created Artifacts
• Deviations/Exceptions from standard practice
• Opportunities for error/Contributors to error

46
Pragmatic Considerations

• Be Open to identify and pursue new findings you
  hadnt anticipated.
• Always end with Is there anything I forgot to
  ask? Anything you came in wanting to say that I
  didnt ask about?
• Be minimally intrusive
• only ask questions during low workload periods.
• leave if user becomes uncomfortable (e.g., if
  workload is too high emergencies occur)

47
Related References

• Roth, E. M., Malsch, N. and Multer, J. (2001).
  Understanding how train dispatchers manage and
  control trains Results of a cognitive task
  analysis. Washington, C. C. U. S. Department of
  Transportation/Federal Railroad Administration.
  (DOT/FRA/ORD-01/02) Available online at
  http//www.fra.dot.gov/pdf/cta.pdf
• Roth, E. M. and OHara, J. (2002) Integrating
  Digital and Conventional Human System Interfaces
  Lessons Learned from a Control Room
  Modernization Program. Washington, D. C. U. S.
  Nuclear Regulatory Commission. (NUREG/CR-6749
  also BNL-NUREG-52638) Available online at
  http//www.nrc.gov/reading-rm/doc-collections/nure
  gs/contract/cr6749/6749-021104.pdf
• Mumaw, R. J., Roth, E. M., Vicente, K. J.
  Burns, C. M. (2000). There is more to monitoring
  a nuclear power plant than meets the eye. Human
  Factors, vol 42, 1, 36-55.
• Roth, E. M. and Patterson, E. S. (in press).
  Using observational study as a tool for
  discovery Uncovering cognitive and collaborative
  demands and adaptive strategies. In Brehmer, B.,
  Lipshitz, R., Montgomery, H. (Eds.). How do
  professionals make decisions? Mahaw, NJ
  Lawrence Erlabaum.

48
Questions?