Suzanne Kabel

1
EISS - Usability field study

• Suzanne Kabel
• Roeland Stoter

2
TOC

• Today
• Evaluation
• Context
• Demo
• Next week
• Carry out field study, 13.00 room 9.01

3
Intro

• Applied evaluation of an indexing-vocabulary
• In Kabels evaluation framework
• Keystroke
• Process
• System image
• Mental model
• System content
• Efficacy

4
Goal of the evaluation

• To determine to what extent the use of indexes
  can be standardized with ontologies, and to what
  extent people use and apply indexes in the same
  fashion.

5
Indexing

• What is it? Describing information or metadata
  about a certain domain. E.g. library of
  scientific articles writer, date, topic,
  keywords, etc.
• Of what? Of elements or fragments of content on
  the Internet or in a database.
• Why? To re-use and share information.
• How? By using standardized indexing vocabularies
  expressed in XML.

6
Markup languages

• Technical implementation of metadata
• generic coding movement (late 60s) separate
  document content from format (IBM, publishing)
• SGML (standard, around 1985)
• HTML fixed tag-set to describe layout
• XML user defined tag-set to describe content

7
XML vocabularies in practice
8
XML vocabularies in Education

• IMS Global Learning Consortiums Metadata
  specification imsproject.org (US)
• ARIADNEs Learning Object Metadata
  ariadne.unil.ch (Europe)
• ADLs SCORM adlnet.org (Defense)
• IMAT European project imat.swi.psy.uva.nl (UvA
  and others)

9
Indexing voc. and ontologies

• Merge of SCORM and IMAT
• In IMAT indexing with ontologies
• An ontology is an explicit specification of a
  conceptualization
• Here A fragment is a frame with slots that need
  to be filled. An ontology serves as a fixed
  vocabulary to fill the slots.

10
Design of the evaluation

• Domain JAVA
• Target use
• Build database of indexed JAVA fragments
• Compose lesson material from JAVA fragments
• Target group JAVA teachers
• Domain knowledge (JAVA and instruction)
• Knowledge of computers
• Experience with efficacy and process features of
  the of the tool the indexing vocabulary

11
Design of the evaluation (2)

• 25 subjects (you) conduct the indexing task
• Index JAVA fragments with the indexing tool
• Answer questionnaire
• Analyze results Compare indexes
• Define percentage correspondence
• Categorize differences
• Overlapping concepts
• Gap in the vocabulary
• Incomplete concept definitions

12
Example analysis
e.g. learning context 6 , due to incomplete
indexing vocabulary Next step adapt indexing
vocabulary
13
Example JAVA fragment
14
Example JAVA fragment

• if (Character.isUpperCase(aChar))
  System.out.println("The character " aChar "
  is upper case.")
• else
• System.out.println("The character " aChar "
  is lower case.")
  System.out.println("The value of aBoolean is "
  aBoolean)

15
Example JAVA fragment

• The direct manipulation of an object's variables
  by other objects and classes is discouraged
  because it's possible to set the variables to
  values that don't make sense. For example,
  consider the Rectangle class from the previous
  section. Using that class, you can create a
  rectangle whose width and height are negative,
  which, for some applications, doesn't make sense.
  
• Ideally, instead of allowing direct manipulation
  of variables, a class would provide methods
  through which other objects can inspect or change
  variables. These methods ensure that the values
  of the variables make sense for objects of that
  type. Thus, the Rectangle class would provide
  methods called setWidth, setHeight, getWidth, and
  getHeight for setting and getting the width and
  the height. The methods for setting the variables
  would report an error if the caller tried to set
  the width or the height to a negative number. The
  other advantage of using methods instead of
  direct variable access is that the class can
  change the type and the names of the variables it
  uses for storing the width and the height without
  affecting its clients.

16
Example index

• Learning resource type narrative text
• Learning context University
• Difficulty medium
• Instructional role explanation
• Domain topic JAVA Class

17
Indexing vocabulary

• General aspects, applicable to the complete set
  of JAVA fragments
• General category
• Lifecycle category
• Content, educational and domain aspects,
  applicable to each individual JAVA fragment
• Content category
• Educational category
• Domain classification category

18
General category

• This category groups the general information that
  describes this resource as a whole. Apply this
  category to the complete set of fragments.
• Title Name given to this resource.
• Description A textual description of the content
  of this resource being described.

19
Lifecycle category

• This category describes the history and current
  state of this resource and those who have
  affected this resource during its evolution.
  Apply this category to the complete set of
  fragments.
• Contribute This sub-category describes those
  people or organizations that have affected the
  state of this resource during its evolution
  (includes creation, edits and publication).
• Role Kind of contribution. (Predefined
  vocabulary)
• Centity The identification of and information
  about the people or organizations contributing to
  this resource, most relevant first.
• Date This sub-category defines the date of the
  contribution.

20
Content category

• This category describes the characteristics of
  the content of this resource. Apply this category
  to each fragment individually. All predefined
  vocabularies.
• Learning resource type Specific kind of resource,
  most dominant kind first.
• Semantic density This element defines a
  subjective measure of this resource's usefulness
  as compared to its size or duration.
• Description type The perspective from which the
  resource was originally written.
• Description scope The nature of the resources
  size.
• Knowledge type The kind of knowledge involved in
  the resource.

21
Educational category

• This category describes the key educational or
  pedagogic characteristics of this resource. Apply
  this category to each fragment individually. All
  predefined vocabularies.
• Interactivity type The flow of interaction
  between this resource and the intended user.
• Interactivity level This element shall define the
  degree of interactivity between the end user and
  this resource.
• Learning context The principal environment within
  which the learning and use of this resource is
  intended to take place.
• Difficulty This element defines how hard it is to
  work through this resource for the typical target
  audience.
• Instructional role The way the resource is used
  in instruction or instructional material.

22
Domain classification category

• This category describes a taxonomic path in a
  specific classification system. Each succeeding
  level is a refinement in the definition of the
  higher level.
• Is-a Leaf-node in the JAVA Is-a ontology
• Part-of Leaf-node in the JAVA Part-of ontology

23
Demo