Cognitive Task Analysis

1
Cognitive Task Analysis

• Different Approaches
• By Jeff Budau

2
Articles Reviewed

• OHare, D., Wiggins, M., Williams A., Wong, W.
  1998, Cognitive task analysis for decision
  centres design and training, Ergonomics, 41,
  1698-1718.
• Militello, L.G., Hutton, R.J.B., 1998, Applied
  cognitive task analysis (ACTA) a practitioners
  toolkit for understanding cognitive task demands,
  Ergonomics, 41, 1618-1641.

3
Purpose

• Both papers
• Introduce variations on methods for conductive
  Cognitive Task Analysis (CTA)
• Try to support or validate the method
• Offer practical examples and applications for the
  methods and outcomes of their processes.

4
Cognitive Task Analysis (CTA)What is it?

• A way of identifying mental processes and skills
  necessary to perform a task proficiently.

5
How are CTAs done?

• There seems to be a number of already established
  ways of performing CTAs.
• These ways fall under two major headings
  (according to OHare et al)
• Constraint Identification
• Activity Analysis in Functioning System

6
OHares View

• OHare subscribes to Kleins (1989) Recognition
  Primed Decision (RPD) model
• Looks at the Critical Decision Method (CDM) which
  is one of the techniques used to analyse
  activities in a functioning system

7
Recognition Primed Decision (RPD) model

• In simple situations the decision maker
  recognizes the broad situation (goals, cues,
  expectancies and courses of action)
• Will seek more information in complex situations
  to create a more complete situational assessment.
• Decision maker may create mental simulations to
  verify the suitability of their best guess
  course of action.

8
Critical Decision Method (CDM) of CTA

• CDM is a set of cognitive probes (read
  questions) used in retrospective interviews
  about non-routine decisions.
• OHare revised the probes to delve into expert
  level cognitive skills by focussing specifically
  on
• Situational awareness
• Planning

9
OHares Studies

• All focussed on training implications and
  applications. The experts studied were
• White water rafters (experts, novices and a group
  in between)
• General aviation pilots
• Ambulance dispatchers in a large urban center

10
OHares Thoughts and Opinions!

• Analytical decision making is more indicative of
  novice/intermediate performance modes while the
  RPD model better explains expert decision making
• Pre-priming is used to mentally rehearse
  possible upcoming situations
• They found their revised probes useful in
  generating info. About expert performance
• Liked the post-it notes

11
What OHare et al would like you to take away
from OHare et al

• CDM can be used as the practical tool to help
  build decision centered training and user
  interfaces in line with Kleins RPD model

12
Concerns about OHare et al

• A tool (CDM) built to support a theory will not
  likely ask the kinds of questions that would
  refute the theory. If all you have is a
  hammer.
• There may be reasonable alternate theories if
  they looked beyond the RPD model.

13
Some Thoughts about OHare et al

• The CGA graph looked chaotic
• The practical training products developed seemed
  excellent
• The design thoughts for the ambulance dispatchers
  seemed reasonable
• There is little in this paper that leads me to
  believe the success met by the authors is more
  dependant on their tools than their skills.
• Is their version of CDM the right tool or just
  the right tool for them?

14
Militello Huttons View Applied Cognitive Task
Analysis (ACTA)

• ACTA can be conceptualized as CTA-lite
• The streamlined methods are designed for used
  by instructional and systems designers rather
  than knowledge engineers, cognitive psychologists
  and human factors/ergonomics professionals

15
ACTA Steps

• Task Diagram Overview
• A broad and shallow look at the task that shows
  where to focus resources on
• Knowledge Audit
• Uses probes (read questions) to capture the
  important points of expertise in this situation
• Simulation Interview
• The interviewer presents a challenging situation
  and the SME then talks their way through it and
  answers a series of probes again.

16
ACTA Output

• Cognitive Demands Table
• Fairly flexible format
• Helps in identifying common themes

Table 1. Example of a cognitive demands table.
17
The Militello Hutton Study

• Tried to compare the ACTA techniques with
  unstructured techniques.
• Used grad students with no CTA background
• Gave everyone general CTA info and info on the
  cognitive demands table, gave the ACTA folks
  extra training in ACTA techniques
• Studied Firefighters and EW specialists with the
  goal of improving a training program for the
  field in question.

18
Judging the Data Output

• Data gathered in the Cognitive Demands table was
  coded by CTA experts.
• Coding system developed by Authors
• SMEs used to determine if info in the table was
  relevant to a highly skilled specialist versus a
  lesser experienced person in the field.
• Finally a usability questionnaire was filled out
  by the subjects performing the interviews.

19
Results?

• ACTA trained students were more comfortable
  conducting the interview.
• Keep in mind they had an additional 6 hours of
  training in the ACTA methods!
• No difference in what the trained versus less
  trained subjects output in their tables, but
  there was lots of variability.
• Low power and a ton of possible confounds
• Students rated the ACTA tools as useful.
• When it is your first time doing something almost
  ANY tool would be better than none

20
What Militello and Hutton would like you to take
away from this

• That the ACTA captures experienced based
  knowledge and is a useful tool.
• The instructional material produced by the
  students was relevant.
• More relevant in the firefighting domain, where
  knowledge of the domain may have played a role in
  the students ability to create good training
  material
• The cognitive Demands table is a good tool
• Exposure to the CTA concepts plays a large role
  in conducting good CTAs

21
Concerns about Militello and Hutton

• This was a fairly shaky study.
• Variability in the students, the subjects they
  interviewed, the topics they interviewed on, the
  way the data was coded and the interpretation of
  the data by the SMEs make any conclusion that
  could have been found dubious.
• The authors have (for sale) a multimedia training
  tool and promote it as a way to increase the
  reliability of performing ACTAs.
• Militello used to work for Klein and Kleins
  company was selling the software (the company was
  bought out in 2005)
• Hardly an unbiased promotion of the product.

22
Some Thoughts about Militello and Hutton

• The ACTA seemed quite useable and applicable to
  situations that require a less in-depth approach
  than that necessitated by the full blown CTA
  techniques
• Its hard not to see this study, which is of
  questionable quality, as much more than a
  promotion tool for the software specifically and
  the ACTA method in general.

23
Final thoughts on the two papers

• The studies within them are entirely
  forgettable and only helpful in illustrating the
  authors points.
• Both papers offer excellent background into CTA
  and sound reasoning for making alterations
• Both papers provide good initial training and
  overview for people conducting CTAs using either
  of the techniques.