The Technical and Pedagogical Usability Criteria for Digital Learning Material

1
The Technical and Pedagogical Usability Criteria
for Digital Learning Material

• Petri Nokelainen
• University of Tampere
• petri.nokelainen_at_uta.fi

Paper presented at the ED-MEDIA 2005 conference,
29.6.2005, Montreal, Canada Salon Joyce
2
Introduction

• Computers have replaced some of the prerequisites
  set for students skills, bringing about their
  own requirements that are often related to the
  use of computer applications.
• The usability analysis of learning material aims
  to influence the process where the correct
  applications and target groups find one another.

3
Introduction

• This enables a student to focus his/her energy on
  the substance itself while working on the
  learning material, instead of irrelevant things
  caused by the insufficient usability of the
  material.
• This paper presents the criteria for evaluating
  the usability of digital learning material.

4
Introduction

• The purpose of the criteria is not to brand any
  user interface or learning material as good or
  bad, but to offer a learner a chance to choose
  the most suitable material for the learning
  situation.
• The criteria have two parts technical and
  pedagogical usability.

5
Introduction

• When evaluating the technical usability, the
  basic assumption is that learning to use the
  central functions of the system is fast and that
  it is easy to use the system.
• Another assumption is that error situations teach
  the user to use the system so that the situation
  that caused the error will not be repeated.

6
Introduction

• When evaluating the pedagogical usability, the
  assumption is that the actions of people who made
  the learning platform or the learning unit has
  been directed by either a conscious or
  unconscious idea of how the functions of the
  system facilitate the learning of the material
  brought to it.
• For example, design paradigms such as objectivism
  (instructivism, behaviorism) and constructivism
  (focus on a learner, learners active role in
  learning and learning from experience).

7
Introduction

• The criteria have been operationalized into an
  on-line Likert scale self-rating Pedagogically
  Meaningful Learning Questionnaire (PMLQ) that has
  97 items (see http//evaluator.hamk.fi).
• In the PMLQ, separate items have been developed
  to measure the usability of the learning platform
  and the learning unit.

8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
Introduction

• When evaluating the usability of a specific
  learning platform (i.e., a learning management
  system), we can evaluate how easy it is to use
  the platform itself (technical usability), or
  what kind of learning material it allows the
  users to input (pedagogical usability).

14
Introduction

• When evaluating the usability of a learning unit
  or a learning object, we assume that each
  learning unit has its own interface relating to
  the content, and learning content based on a
  certain learning goal.
• Evaluation of the technical usability of a
  learning unit can be directly compared with the
  evaluation of the usability of the learning
  platform itself, as described earlier in order
  to be able to evaluate the pedagogical usability
  of a learning unit, we must try to control the
  effect of the pedagogical solutions of a learning
  platform.

15

• In this study, operationalization of usability
  attributes is based on Nielsens (1990, 148)
  model, where the top-level concept is system
  acceptability.

16
System acceptability has two components 1.
social acceptability, and 2. practical
acceptability.
17
Practical acceptability is composed of cost,
compatibility, reliability and usefulness
(Nielsen, 1990, 148).
18
Usefulness is further divided into utility and
usability.
19
Entertainment product has high utility for end
user if it is fun to use, and learning material
has high utility if learners learn from using it
(1993, 25).
20
Pedagogical usability is categorized in this
study into sub component of utility, as technical
usability is a sub component of usability. Thus
dialog between user and system, and learning
goals set by student and teacher are both aspects
of pedagogical usability.
21
The technical usability criteria

• The literature review showed that several
  recommendation lists for the evaluation of
  usability have been developed during the last
  twenty years (e.g., Shneiderman, 1988 Chin,
  Diehl Norman, 1988 Nielsen, 1993 Lin, Choong
  Salvendy, 1997 Preece, Rogers Sharp, 2002
  Chalmers, 2003 Tognazzini, 2003).

22
The technical usability criteria

• The usability lists have many common components,
  such as consistency, user control, error handling
  and recovery.
• A large part of the components in the
  recommendation lists are relevant in the
  evaluation of the usability of current digital
  learning materials, but the accessibility
  dimensions has been added within this research.

23
The technical usability criteria

• Accessibility
• Learnability and Memorability
• User control
• Help
• Graphical layout
• Reliability
• Consistency
• Efficiency
• Memory load
• Errors

24
The technical usability criteria

• 1. Accessibility. The most important point is
  that the learning material has no value for
  learner, if she is not able to use it in the
  first place.
• Learner should be able to use learning material
  with different browsers and devices. (W3C, 1999.)

25
The technical usability criteria

• 2. Learnability and memorability. Learnability
  concerns novice, and memorability concerns casual
  expert user (Nielsen, 1993, 31).
• System that is hard to learn is only valuable for
  those users who are able to spend time to learn
  it.
• System that is impossible to learn has no value
  for any user.

26
The technical usability criteria

• 3. User control. User has the feeling that the
  software operates for her, not the opposite way
  (Shneiderman, 1998 Nielsen, 1993 Tognazzini,
  2003 Lin, Choong Salvendy, 1997).
• Use of the system should be so intuitive, that no
  "help" is needed (Squires Preece, 1996).

27
The technical usability criteria

• 4. Help. Help should be available at any time
  Nielsen (1993, 149) and in all possible
  situations, meaning that those situations should
  be mapped before real users start using the
  system.
• Help and instruction should be presented in
  understandable form (Nielsen, 1993, 151-152).

28
The technical usability criteria

• 5. Graphical layout. Layout is structured in best
  possible way to promote users ability to use the
  system, for example according to Fitt's law
  (Tognazzini, 2003).
• User interface should be intuitive for most
  users.
• Information (subject) and structure (user
  interface) should be two different things
  (Leflore, 2000, 103).

29
The technical usability criteria

• 6. Reliability. System should be technically
  reliable.
• User should be able to trust that her work is
  safe with the software. (Nielsen, 1993
  Shneiderman, 1998 Tognazzini, 2003).

30
The technical usability criteria

• 7. Consistency. Consistent user interface gives
  the user transferable skills, that are useful in
  other systems and contexts, too.
• User interface components should be placed in
  consistent way. (Shneiderman, 1998 Nielsen,
  1993 Tognazzini, 2003 Lin, Choong Salvendy,
  1997 Chalmers, 2003.)

31
The technical usability criteria

• 8. Efficiency of use. User should be able to
  adopt conceptual structure of the system in order
  to automatize common routines, for example with
  shortcuts and recordable macros (Shneiderman,
  1998 Nielsen, 1993 Tognazzini, 2003).

32
The technical usability criteria

• 9. Memory load. User is at her best recognizing
  things, computer is much more effective in
  remembering things (Nielsen, 1993, 129).
• Less is more - more synchronous information
  available, longer it takes from the user to
  process it and make decisions (Nielsen, 1993).

33
The technical usability criteria

• 10. Errors. Error messages should tell clearly,
  what is wrong and what are users next possible
  steps (if there is any) to correct the problem
  (Nielsen, 1990 1993 Shneiderman, 1998).

34
The pedagogical usability criteria

• Pedagogical aspects of designing or using digital
  learning material are much less frequently
  studied than technical ones (Reeves, 1994
  Squires Preece, 1996 Quinn, 1996 Albion,
  1999 Squires Preece, 1999 Horila, Nokelainen,
  Syvänen Överlund, 2002).

35
The pedagogical usability criteria

• There are relatively few inventories measuring
  subjective end-user satisfaction with the
  pedagogical aspects of digital learning material.
  
• Further, not a single inventory was found that
  had undergone a process of empirical psychometric
  testing.

36
The pedagogical usability criteria

• Learner control
• Learner activity
• Cooperative learning
• Goal orientation
• Applicability
• Added value
• Motivation
• Valuation of previous knowledge
• Flexibility
• Feedback

37
The pedagogical usability criteria

• 1. Learner control. Control of the technology
  should be taken away from the teachers and
  instructional designers and given to the learners
  (Jonassen, Myers McKillop, 1996).
• Meaningful encoding (chunking), e.g. presenting
  learning material in meaningful units (Wilson
  Meyers, 2000).

38
The pedagogical usability criteria

• 2. Learner activity. Teacher role
  (facilitative/didactic) depends on underlying
  pedagogic assumptions (Reeves, 1992).
• Learning material should gain learners attention.
  
• Learners should feel that they own the goals of
  action and thus the results (Jonassen, Peck
  Wilson, 1999).

39
The pedagogical usability criteria

• 3. Cooperative learning. Constructivist view is
  based on social learning and knowledge sharing
  via collaborative tasks.
• Learners are able to discuss and negotiate about
  different approaches to the learning task
  (Jonassen, 1995).
• Tools might support asynchronous or synchronous
  social navigation (Mayes Fowler, 1999 Kurhila,
  Miettinen, Nokelainen Tirri, 2002).

40
The pedagogical usability criteria

• 4. Goal orientation. Instructivists emphasize few
  clearly defined goals, constructivist goals
  should also be clear, but set by the learners
  themselves (Wilson Meyers, 2000).

41
The pedagogical usability criteria

• 5. Applicability. Authentic activities and
  contexts examples should be taken from authentic
  situations results (Jonassen, Peck Wilson,
  1999).
• Transfer - learned knowledge or skills are useful
  in other contexts, too.
• Learning by doing (Wilson Meyers, 2000).
• Human development should be considered in a way
  that the material is relevant for target
  population's developmental stage (Wilson
  Meyers, 2000).

42
The pedagogical usability criteria

• 6. Added value for learning. Relevance of media
  elements (sound, animation, video).
• Jansen, van den Hooven, Jägers Steenbakkers
  (2002)
• Better adaptation to individual needs
• More flexible offer of content
• Student takes over learning functions
• More attractive subject matter
• Improvement of communication
• Student is more involved.

43
The pedagogical usability criteria

• 7. Motivation. Motivation affects all learning
  (Ruohotie, 2002 Chalmers, 2003).
• For example, intrinsic motivation (need for deep
  understanding) and extrinsic motivation (need for
  high grades) (see e.g., Reeves, 1992 Ruohotie
  Nokelainen, 2003).

44
The pedagogical usability criteria

• 8. Valuation of previous knowledge.
  Prerequisites, what is needed to accomplish
  learning tasks.
• Meaningful encoding (elaboration), learner is
  encouraged to make use of her previous knowledge
  (Wilson Meyers, 2000).

45
The pedagogical usability criteria

• 9. Flexibility. Pretesting and diagnostics help
  to adapt learning material for different learners
  (Hannafin Peck, 1988, 48 Wilson Meyers,
  2000).
• Task decomposition, small and flexible learning
  units (Leflore, 2000).

46
The pedagogical usability criteria

• 10. Feedback. Feedback is response sensitive and
  accurate (Hannafin Peck, 1988, 47).
• Learner has feeling that there is a real dialogue
  between her and computer (Mayes Fowler, 1999).

47
Empirical evaluation of the criteria

• A PMLQ (Pedagogically Meaningful Learning
  Questionnaire) research instrument was developed
  on the basis of aforementioned technical and
  pedagogical usability criteria.
• First version of the instrument contained 92
  multiple-choice items.
• The five-point scale ranged from 1 (totally
  disagree) to 5 (totally agree) and the sixth
  response option was not applicable.

48
Empirical evaluation of the criteria

• Preliminary empirical measurements were carried
  out in October 2003 with 5th and 6th grade
  elementary school students (n 66) and their
  teachers (n 4).
• Participants evaluated the OPIT learning system
  and four learning modules embedded into the
  system.
• Dependencies between the variables were
  investigated with Bayesian dependency modeling
  due to ordinal measurement scale and small sample
  size. The results supported the chosen
  dimensionality (Nokelainen, 2004).

49
Empirical evaluation of the criteria

• The next step in the development process was to
  revise the old items as interview with the
  children and teachers revealed some deficiencies
  in wordings.
• The second version of the PMLQ contained 97 items
  after five new items were included.
• Empirical measurements were carried out in
  February 2004 with 5th and 6th grade elementary
  school students (n 74) with age median of 12
  years.

50
Empirical evaluation of the criteria
51
Empirical evaluation of the criteria

• The technical usability Cronbachs alpha
  reliability coefficients range from .59 (1.7
  Consistency) to .95 (1.2 Learnability and
  Memorability) for the OPIT learning management
  system.
• The pedagogical usability alphas range from .72
  (2.4 Goal orientation) to .86 (2.3 Cooperative
  learning).

52
Empirical evaluation of the criteria

• We further studied the reliability coefficients
  for the dimensions measuring the technical and
  pedagogical usability of the two learning
  modules.
• 34 students used and evaluated the decimal
  numbers in a continuum math learning material.
• Reliability coefficients range from .66 (1.8
  Efficiency) to .94 (1.2 Learnability and
  Memorability).

53
Empirical evaluation of the criteria
Although high alpha value may indicate
unidimensionality, most dimensions have
subdimensions, for example 1.1 Accesibility
54
Empirical evaluation of the criteria

• The reliability coefficients for the English
  singular or plural learning material were also
  satisfactory, but the last technical usability
  dimension (1.10 Errors) had a low reliability
  score with this sample (.39).
• The explanation for this is quite obvious The
  dimension is composed of two different sub
  dimensions. The first one measures users ability
  to use undo function, and the second one
  measures if the error messages are informative.

55
Empirical evaluation of the criteria
56
Conclusion

• Two-stage empirical evaluation of the criteria
  supported the theoretical structure of both
  technical and pedagogical usability criteria.

57
Thank you!

• Technical and pedagogical usability items and
  PMLQ factorial structrure are available at

http//www.uta.fi/aktkk/papers/edmedia2005
58
(No Transcript)
59
(No Transcript)