IMPRINT Pro Usability Assessment

1
IMPRINT Pro Usability Assessment

• Pratik Jha, Julie Naga and Dr. Raja Parasuraman
• George Mason University

2
Introduction

• A professional (Pro) version of IMPRINT is
  currently under development
• More Modular
• Added Functionality
• Greater Flexibility
• This has also resulted in series of User
  Interface changes
• Multiple windows to simultaneously display
  information
• Tree structure for navigation
• Pallete to show common used functionality

3
Project Goals

• To Uncover Usability Issues of IMPRINT Pro
• Does the new interface allow users to conduct
  their analyses as effectively as with older
  versions of IMPRINT?
• Is the added functionality and flexibility
  understandable and usable?
• Is the new system easy to learn for a user
  trained on earlier versions?
• Does the IMPRINT Pro interface allow users to
  navigate through the space of modeling options in
  an effective and timely manner?

4
Method

• 2 step approach
• Heuristic evaluation (first cut)
• Usability walkthrough (confirmatory)

5
Heuristic Evaluation Methodology

• Familiarization phase
• Team read product manuals and other background
  material
• Conducted high level task analysis
• Gathered information on user characteristics
• Attended IMPRINT workshop and tutorial
• Review Phase
• 3 reviewers conducted independent usability
  evaluation
• 7 well established heuristics were used for
  evaluation
• Several group meetings were held to discuss
  reviews and consolidate evaluation

6
Heuristics
7
IMPRINT Pro Areas Evaluated

1. Overall look and feel (windows, menu structure,
   legibility of text and icons etc.)
2. Analysis tree
3. Navigational tab
4. Properties window
5. Adding a mission from library
6. Adding warfighter
7. Adding mission
8. Adding equipment

8
Tasks

• Add new analysis
• Add name of analysis
• Add brief description and purpose of the
  analysis
• Add a warfighter
• Add name
• Select MOS
• Select workload strategy

• Add a mission
• Add mission name and description
• Add task network (functions and task)
• Add task parameters or change properties
• Add a equipment
• Add equipment name and description
• Add or change parameters
• Save an analysis
• Add analysis from the library

9
Results

• 37 usability issues were uncovered from heuristic
  analysis
• Results suggest some major usability problems in
  IMPRINT PRO software that could lead to
• Poor user experience
• Increased training time and poor learnability
• Negative training transfer effect because the
  software is inconsistent with the previous
  versions
• Majority of the usability issues are severe or
  moderate in nature

10
Results Usability Issues by Severity
11
Results Usability Issues by Heuristic Criteria
12
Results

• Ease of Navigation
• Software does not lead the user to the next step
• User options are not visible
• Number of user clicks are not optimized
• Analysis tree and navigational bread crump gets
  cluttered fairly soon
• Look and Feel
• Display clutter due to multiple windows
• Poor contrast between background and text

13
Results

• Functionality
• Pane seems to be of limited use and adds to
  display clutter
• Windows functionality in the view menu is not
  enabled
• Parameters cannot be changed in the properties
  window
• Enabling user options
• A new version is added instead of new analysis
• Cant drag and drop missions from other analysis

14
Results

• Maximize Recognition
• Excessive use of Acronyms
• MOS categories and workload strategies are not
  defined
• Preventing User Error
• Incorrect user input sort (for example user is
  asked to input probability in percentage)

15
Results

• Providing User Feedback
• Navigational tab and analysis tree are not
  synchronized
• No feedback provided to the user if the analysis
  is saved or not
• Property windows are not labeled

16
Usability Walkthrough

• We conducted a usability walkthrough to validate
  results of heuristic analysis
• Methodology
• Tested 4 participants on Pro (2 experts and 2
  novice)
• Participants performed a series of task
• Participants were asked to think aloud
• Session was video taped for future analysis

17
Task

• Add new analysis
• Name it NCops
• Add brief description about the purpose of the
  analysis for reference
• Add new Warfighter
• Name the warfighter as soldier 1
• Select appropriate MOS category
• Select appropriate workload management strategy
• Add new mission
• Name the mission Network Centric Warfighter
• Add a brief description

• Add task network
• Add function and name it Gather Information
• Add a task to above function and name it
  Situation Monitoring (task 1)
• Add another task to above function and name it
  Scan Head Mounted Display (task 2)
• Change the priority of task 1 to 1 (Priority can
  be changed from the Effects screen of task
  properties)
• Change the priority of task 2 to 2
• Review the task properties in snapshot view
• Save your analysis and exit

18
Data Collected

• Background questionnaire
• IMPRINT usage
• Experience
• Others
• Subjective ratings
• Overall reaction to the application
• Screen
• Terminology and system operations
• Learning
• System
• Think aloud data
• Comments

19
Results Overall Reaction to Application
20
Results
21
Results User Comments

• Feedback
• " It is very different than 7 and will require
  old IMPRINT users to have new training because it
  is vastly different
• There is no feedback on saving an analysis."

22
Recommendations

• A detailed task analysis should be conducted to
  improve the navigation and refine the
  functionality of IMPRINT Pro
• An analysis should be performed to investigate if
  the radically different interface of IMPRINT Pro
  from the previous versions could cause a negative
  transfer training effect
• Functionality aimed at minimizing training time
  or to provide users help in conducting the
  analysis should be explored. For example, the
  efficacy of wizards that help users in building
  an analysis should be evaluated.

23
(No Transcript)