Evaluating Information Systems

1
Evaluating Information Systems

• November 16, 1999

2
Credits

• Material in these slides are adapted from
  materials developed by Professors Roxanne Hiltz
  and Murray Turoff.

3
Functionality

• Functionality - Refers to the features of a
  system, i.e. what is can do!
• Task of specifying functional requirements is
  generally straightforward.
• However, comparing a list of functions requested
  by the users to a list of features in a system is
  not necessarily a measure of how good the
  system functions.
• Peoples reasons for using a system differ (it
  may be the only way to get a job done!)

4
Functionality - 2

• The system may help a person do a job quicker or
  better (more efficiently).
• Whether or not a user feels a computer is
  necessary for a task depends on how well the
  computer meets the users needs.
• For example, if you have to send someone a large
  detailed table of information (spreadsheet) and
  the e-mail system does not handle this well,
  perhaps the system does not meet the users
  needs.

5
Functionality - 3

• There is often the perception that the more
  functions are provided, and the more flexibility
  and more complexity in the system, the better.
• FALSE
• For discretionary and non-discretionary users,
  how the functions are implemented will have a
  significant impact on system usability.

6
Usability

• A more difficult construct to define.
• An abstract concept, it relates to ease of use in
  which functionality can be accessed.
• Another way to understand usability is the ease
  of use in which a user communicates with a
  system.
• However, if the functionality provided is easy to
  use, yet the functionality does not address the
  task at hand, then the system is not usable.
• Summary Usability depends on characteristics of
  the user and characteristics of the task.

7
User Types Modes

• NOVICE, CASUAL, INTERMEDIARIES
• EXPERIENCED
• ROUTINE
• FREQUENT
• OPERATORS
• PROBLEM SOLVERS
• POWER
• RESULTS
• DIFFERENT ROLES IN ONE SYSTEM
• MULTIPLE INTERFACE METHODS

8
Why Systems Are not Used?

• 1. Limited functionality if the functions
  provided do not match task requirements, a system
  will not be usable (ex. Email system with no
  editor!)
• 2. Poor interface design (e.g., inadequate
  flexibility (won't take abbreviations, provide
  defaults, commands as well as menus, etc.)
• Poor consistency and integration within the
  system (Different parts look and work in
  different ways, very confusing!)

9
Why Systems Are not Used? - 2

• 3. ACCESSIBILITY Access or Availability
  problems But this interacts with motivation.
• 4. Start-stop hassles (losing some of the work
  you have done if you have to stop in the middle
  and do something else)
• 5. Poor response time
• 6. Poor systems dynamics
• 7. Inadequate training and user aid

10
Why Systems Are not Used? - 3

• 8. Poor or non-existent documentation.
• 9. Poor integration with other systems
  (non-transferability of data negative
  interference with learned interface conventions)
• 10. COSTS - including learning time, exceed
  expected benefits

11
Nancy Goodwin

• THESIS There is no functionality without
  usability.
• Corollary It is not true that the more
  functionality, the better! "Richer" but less
  usable systems provide less "effective" (actual)
  functionality.
• to be truly usable a system must be compatible
  not only with the characteristics of human
  perception and action, but, and most critically,
  with users cognitive skills in communication,
  understanding, memory, and problem solving...

12
Usability Does Matter

• Providing extensive functionality is not enough.
  People must understand what the functions do and
  how to use them.
• Designing for different types of users compounds
  the problem (e.g., novice vs. expert).
• Failure to consider usability can lead to system
  failure.
• Summary Point Usability contributes to overall
  system functionality by making it accessible to
  users and facilitating effective use of
  functional capabilities.

13
Ways to Help in Designing Usable Systems

• The Prototype Methodology
• Protocol Analysis

14
Prototyping
15
Prototyping

• As a systems development methodology, it is not
  in and of itself an evaluation methodology.
• Rapid prototyping facilitates extensive use of
  protocol analysis or other methods for
  systematically obtaining user feedback.
• Prototyping is a development methodology based on
  building, testing, and iteratively improving a
  model of a system.
• Eventually the model becomes the system.

16
Benefits of Prototype

• 1. A prototype can provide a user with tangible
  means of comprehending and evaluating the
  proposed system, and eliciting meaningful
  feedback.
• 2. It can provide a common baseline and frame of
  reference so designers and users can communicate
  better.
• 3. It provides a way for users to participate and
  commit to a project (reference Joint Application
  Design). It can generate user enthusiasm for the
  project.

17
Benefits - 2

• 4. The prototype can get things right,
  i.e.,help ensure that the system performs
  adequately before widespread use.
• 5. Prototyping can reduce both the development
  life cycle time to installation and overall
  cost. Some estimates have been as high as a 70
  reduction in cost.

18
Possible Disadvantages

• 1. Some formal prototyping software can be
  expensive. However, new PC tools (e.g. Visual
  basic, and even PowerPoint provide inexpensive
  means to do screen mock-ups).
• 2. If a formal prototyping software is used, the
  actual result may have to be re-written for a
  final production version.
• 3. Early versions of a prototype can provide
  early disappointment for the user and possibly
  cancel the project.
• By definition, this early version has only
  essential features. However, some users may
  expect to see a finished system.

19
Possible Disadvantages - 2

• 4. The Prototyping process may be difficult to
  manage and control. Its nature is iterative and
  may repeat itself multiple times.
• It does not have the established phases of the
  traditional systems development life cycle
  (waterfall model). Establishing phases,
  milestones, deliverables is more difficult since
  user requirements are always evolving.
• 5. It is difficult to prototype large
  information systems that must efficiently handle
  large amounts of data and many users.

20
PROTOCOL ANALYSISadapted from Murray Turoff
21
Protocol Analysis

• The Thinking Out Loud Method
• A protocol is a record of a step by step
  procedure. In this method, one records the step
  by step procedures of a user thinking out loud
  while trying to use an information system.
• A qualitative, direct observation method for
  determining usability.

22
WHY DO IT?

• Objective To discover the process a person goes
  through in solving a problem.
• Uses Learning Cognitive Processes, Developing
  Expert System material, evaluating interfaces.

23
KEY ASSUMPTION

• Cognitive processes that generate verbalization
  are subset of those that generate behavior

24
CONCERNS ABOUT THE METHOD

• Subjects may have incomplete knowledge of their
  thinking processes therefore the record may be
  incomplete.
• Subjects may not have an accurate understanding
  of the processes of which they are aware.
• The thinking process may be distorted by the
  thinking out loud process.
• Ambiguity in language may lead to
  miscommunications.

25
NECESSARY ATMOSPHERE

• Honesty
• No evaluation of subject
• No pressure for performance
• No introduction of bias
• No contamination of mental process (e.g. help)
• Reciprocity and Respect

26
STIMULUS RESPONSE METHOD

• Different stimulus may produce different mental
  behaviors
• Do you know the capital of Delaware?
• Which of the three Newark, Wilmington, or Dover
  is the capital?
• Name the capital of Delaware.

27
MODES OF PROBING

• Talk Aloud, Think Aloud While information is
  attended.
• Concurrent Probing While in short term memory.
• Retrospective Probing After completion of the
  task.
• Note you want to concentrate on the first type,
  but not to interrupt the subjects thinking.

28
PROTOCOL ANALYSIS CONDITIONS

• Subject asked to verbalize what they are thinking
• Subject is not being evaluated
• Observer must not participate in process
• Observer must not aid the subject
• Subject providing knowledge of how they solve or
  a problem (or learn a system)

29
PROTOCOL ANALYSIS PROCESS I

• Present the subject a written explanation of what
  is taking place.
• Explain that this is to evaluate the system and
  not them
• Explain you are there only to observe and can not
  help them.
• Present in writing a task written the user
  terminology and the way the user would think
  about it.

30
PROTOCOL ANALYSIS PROCESS II

• Total time should take between thirty minutes to
  an hour if no major problem encountered.
• Observe and record (video, voice tape, PC
  interactions, notes and coding)
• Only interrupt user for further verbalization if
  it is unclear what they are doing.

31
PROTOCOL ANALYSIS PROCESS III

• Ask the user to describe what he or she is doing
  out loud.
• Ask the user to go through all the terms on the
  screen and explain what they think they mean
• Ask the user to forecast what they think an
  action will do
• Carry out the task on the system

32
PROTOCOL ANALYSIS PROCESS IV

• Give help only if user is at a dead end
• Questions you can ask during if necessary
• Why do you do/say that?
• What is troubling you?
• How do you know that ...?
• Why do you do it this way?
• Save retrospective questions for end of session.

33
MAJOR LIMITATION

• Can not use it on a task that the user is not
  familiar with.
• Requires training on the task first.

34
TYPICAL WRITTEN INSTRUCTIONS

• PLEASE EXPLAIN
• What you think something means.
• What you are trying to do.
• What confusion or concerns you have.
• What you expect to happen next.
• What you dont know the meaning of.

35
CODING SCHEMA FOR VERBALIZATION I

• EXAMPLE I - Thinking Out Loud includes relating
• Intentions goals, shall, will, must, have to
• Cognitions current attention situation, define,
  understand
• Planning If x than y
• Evaluation Yes, No, Damit, Fine
• Changing conditions \ view of the problem

36
CODING SCHEMA FOR VERBALIZATION II

• EXAMPLE II
• Surveying given information
• Generating new information
• Developing a hypothesis
• Unsuccessful solutions
• Self reference or criticism
• Silence

37
PROTOCOL ANALYSIS OBSERVATIONS I

• Verbalization occurs only 30 to 50 of the time.
• Subjects cannot verbalize when
• Reading text
• Doing intense cognitive activity
• Making choices
• Subjects have to slow down to verbalize
• Subjects will tend to be more careful as a result
  of verbalization and observation

38
PROTOCOL ANALYSIS OBSERVATIONS II

• Experts on a problem verbalize a lot more than
  non experts (double)
• Experts have more difficulty verbalizing at a
  very detailed level with respect to the problem
  with which they are dealing with.
• Experts usually spend more time in planning and
  in the restructuring of the problem.

39
INCREASING VERBALIZATION

• 1. Hold back stimulus or encourage slowness.
• 2. Segment stimulus (subtasks)
• 3. Interrupt with pre-arranged signal or set
  point to pause
• e.g. when you are ready to indicate an action,
  first explain what you think everything means on
  the screen.

40
OBJECTIVES FOR INTERACTIVE SYSTEMS I

• Determine their understanding of terms in the
  interface
• Understand the cause of errors or
  misinterpretations
• Determining missing functionality or user
  requirements

41
OBJECTIVES FOR INTERACTIVE SYSTEMS II

• Determining reactions to and utility of features
• Determine the utility of a metaphor for learning
• Determining the utility of help and guidance

42
PROTOCOL ANALYSIS ADVANTAGES

• A lot less effort than other approaches.
• Can be done with prototype or mockup before any
  coding.
• Learning how user approaches task
• Finding major mistakes in design
• Can learn attitude
• Rapid feedback from small samples
• Also useful for understanding user task

43
PROTOCOL ANALYSIS REQUIREMENTS

• Subjects must be representative
• Three subjects for each distinctive type of user
  and for each different set of tasks
• Instructions simple
• Observe only

44
PROTOCOL ANALYSIS QUESTIONS

• Can ask/say
• Please explain your choice.
• What are you thinking?
• What does that term mean?
• Should not ask
• Why did you do that?
• What does append do?
• Have physical signal for interrupt

45
PROTOCOL ANALYSIS HOW TO I

• A one page explanation to the subject
• Set of written tasks in user terms
• Subject should only spend 40-60 minutes.
• Categorization scheme for recording
• Tape record their verbalizations

46
PROTOCOL ANALYSIS HOW TO II

• Retrospective questionnaire for end
• Retention of major concepts
• Perceived utility of features/functionality
• Do not try to test everything
• At least three subjects on same task
• Be specific about user explaining choice they are
  about to make

47
UTILITY OF MOCK UP

• Ease of understanding (current knowledge)
• Ambiguity in terms
• Confusion generation
• Loss of continuity
• Developing on line help
• Marketing and acceptance

48
UTILITY OF WORKING SYSTEM

• Ease of learning (new knowledge)
• Error impact Utility of feedback
• Ease of exploration
• Realistic task execution
• Developing final user documentation
• Utility of new features (beyond current system)

49
BASIC QUESTIONS I

• Do the terms used on the screen mean to the
  subject what the designer thought they would
  mean?
• Do the alternatives presented at that point in
  the interaction include what the subject wishes
  to do?
• Is the help material or the system messages
  understandable or relevant to the needs of the
  user?

50
BASIC QUESTIONS II

• Does the subject have difficulty locating or
  perceiving things on the particular screen?
• Does the subject utilize the sequences of
  operations that the designer expected to be
  utilized in accomplishing a given task?
• Can the user utilize the interface metaphor for
  learning the system?
• What type of errors is the user making and why?

51
WHEN TO USE

• Protocol analysis should be used
• Before system requirements are finalized
• After every major change to requirements or
  interface
• Before installation of system for users
• Before introduction to new user population type

52
Some Basic Principles ofUser Interface Design
53
Principle 1 Recognize the Diversity of Users

• Know the User - Review User Types and Modes
  discussed earlier.
• Know the Tasks
• Consider these Five Human Factors which are
  central to evaluation
• Time to Learn
• Speed of Performance
• Rate of Errors
• Retention over Time
• Subjective Satisfaction

54
Interaction Styles

• Direct Manipulation
• Menu Selection
• Form Fill-In
• Command Language
• Natural Language

55
Direct Manipulation

• Advantages
• visually presents task concepts
• allows easy learning
• allows easy retention
• allows errors to be avoided
• encourages exploration
• affords high subjective satisfaction
• Disadvantages
• may be hard to program
• may require graphics display and pointing devices

56
Menu Selection

• Advantages
• Shortens learning
• reduces keystrokes
• structures decision-making
• permits use of dialog management tools
• allows easy support of error handling
• Disadvantages
• presents danger of many menus
• may slow frequent users
• consumes screen space
• requires rapid display rate

57
Form Fill-In

• Advantages
• simplifies data entry
• requires modest training
• gives convenient assistance
• permits use of form-management tools
• Disadvantages
• consumes screen space

58
Command Language

• Advantages
• is flexible
• appeals to power users
• supports user initiative
• allows creation of user-defined macros
• Disadvantages
• poor error handling
• requires substantial training and memorization

59
Natural Language

• Advantages
• Relieves burden of learning the system
• Disadvantages
• requires clarification dialog
• may requires more keystrokes
• may not show context
• is unpredictable

60
Principle 2 - Use the Eight Golden Rules of
Interface Design

• 1. Strive for Consistency.
• 2. Enable Frequent Users to Use Short-Cuts.
• 3. Offer Informative Feedback
• 4. Design dialogs to Yield Closure
• 5. Offer Error Prevention and simple error
  handling.
• 6. Permit easy reversal of actions.
• 7. Support internal locus of control.
• 8. Reduce short-term memory load.

61
Principle 3 - Prevent Errors

• Improve error messages provided by the system so
  that users learn from mistakes.
• Understand the nature of errors.
• You may be able to avoid some errors people make
  - slips by
• organizing screens and functions functionally.
• Design commands or menu choices to be
  distinctive.
• Make it difficult to make irreversible actions.