Broken pattern in certification process

1
Corso Interazione Uomo Macchina II
(specialistica) Anno 2008-2009
Lez. 17 Il pattern interrotto. Un esempio di
valutazione basata sulla semiotica.
Doc. Andrea Marcante
Questi lucidi sono stati preparati da Andrea
Marcante, Università degli Studi di Milano, per
uso didattico. Essi contengono materiale
originale di proprietà dell'Università degli
Studi di Milano e/o figure di proprietà di altri
autori, società e organizzazioni di cui e'
riportato il riferimento. Tutto o parte del
materiale può essere fotocopiato per uso
personale o didattico ma non può essere
distribuito per uso commerciale. Qualunque altro
uso richiede una specifica autorizzazione da
parte dell'Università degli Studi di Milano e
degli altri autori coinvolti.
2

• Broken pattern in the Q certification process.A
  semiotic based evaluation

3
The Q project (1)

• QUATRO Plus Quality Assurance and Content
  Description.
• Goal certify web content quality through the
  labeling of web resources

4
The Q project (2)

• Remark the quality depends on the domain which a
  specific resource belongs or is related to

5
Labelling Management System (LME) (1)

• LME the system to support labelling
  experts/organizations through the lifecycle of
  the description process ? it is the system to be
  evaluated
• LME aims
• supporting the process of creating Content Labels
  using agreed vocabularies
• representing labels using a common and
  interoperable format
• supporting continuous web resources monitoring to
  certify the quality of the web resource(s) over
  time.

6
Labelling Management System (LME) (2)

• Some remarks
• one system for different domains
• different vocabularies for different domains
  i.e. health vocabulary (WMA),
• systems users domain experts

7
Identifying the LME users (1)

• Who are the users?
• In a semiotic perspective, who are the receivers
  of the meta-message from the designer?

8
Identifying the LME users (2)

• Domain experts mean
• Experts in a specific knowledge domain who are
  not expert in (web) labeling (i.e. doctors)
• People who are experts because their status
  (i.e. parents)
• Labeling Authorities (LA) representatives

9
Identifying the LME users (3)

• It is possible to identify common characteristics
  of the users but in any case some important
  differences remain.
• and the LME is one!
• How to perform a significant evaluation?

10
The two-steps usability evaluation (1)
LME is a framework for managing quality labelling
of web resources. Framework an integrated set
of components that collaborate to provide a
reusable architecture for a family of related
applications. (Schmidt 08) It has different
configurations according to the knowledge domain
which it refers to. Configuration an application
characterized by one (or more) domain vocabulary
used to label the resources in a specific domain
of knowledge. The semantics embedded in LME
depends on the domain.
11
The two-steps usability evaluation (2)

• Evaluation of the framework usability on
  functionalities which do not depend on semantics
  of the domain designed and performed by UNIMI.
• LME framework evaluation aims at highlight
  usability problems regarding structure,
  navigation architecture and interaction style

2. Evaluation of the health domain (WMA-COMB)
configuration on functionalities which depend on
semantics of the domain designed and performed
with the necessary COMB collaboration. LME
configuration evaluation consider also semantic
problems, related in particular to the
understanding and interaction with domain
vocabulary criteria.
12
The two-steps usability evaluation (3)

• LME framework evaluation test with 30 users FCD
  students according to Nielsen 93, they are
  potentially future teachers or parents and also
  LA representatives.

2. The health domain (WMA-COMB) LME configuration
evaluation test with 5 users 5 health experts
of COMB who are also LA representatives
according to Nielsen 07, it is enough to
highlight usability problems.
13
The LME designer meta-message (1)

• Two aspects of the problem
• What is a label? Recursively it is a web
  resource which
• a. describes a resource,
• b. contains the evaluation of the resource
  (label)
• 2. What is labeling? The process of add labels
  in the Q, the process to certify the content
  quality of a web resource

14
The LME designer meta-message (2)

• The conceptual model hypothesis
• The label is a web resource, composed by two main
  clusters of information
• The description of the web resource it refers to
  (name, URL)
• The quality criteria selected according to one or
  more vocabularies
• THIS IS ALSO THE REPRESENTATION OF THE POWDER
  FORMAT (the RDF representation of the label)

15
The LME designer meta-message (3)

• The conceptual model hypothesis
• 2. If the label is what we described before, the
  user has to add two kind of information
• a. The description of the web resource it refers
  to (name, URL)
• b. The quality criteria selected according to one
  or more vocabularies
• 3. The system has to provide a two-steps process
• Step 1. the user adds the description of the
  resource
• Step 2. the user certifies the resource selecting
  the criteria

16
The LME designer meta-message (3)

• The vocabulary of signs provided to the user

17
Step 1 Add web resource
The designer message is not add the label,
create label, create web resource description
Step 2
Step 3
Step 4
Step 5
18
A semiotic misunderstanding

• It is a problem on the thirdness level
• the designer mixes up the representamen (the
  resource description) and the referent (the
  resource itself)
• ? The link, an indexical sign, becomes the
  resource in the designer metaphor (see the space
  it fills in the next slide)

19
The first break (1)
Create label
20
The first break (2)

• The designers reason focusing on the clusters of
  information and not on the unity of label even
  if the object is one (the label contains both
  the description and the criteria), the process is
  divided in two steps.
• The designers try to unify this forcing the user
  to pass to the My web resources item.

21
Framework evaluation results

• The users are disoriented when they have to
  select the description of the resource to create
  the label because
• 1. They do not recognize the symbol to add
  criteria
• 2. It is not easy to find the description (the
  list of descriptions is long )
• 3. They can label only the resources they
  described.

22
Suggestions from the results

• Introduce a menu
• Separate the access to all the descriptions and
  the access to my own descriptions
• Highlight by a menu item the function to create
  label.
• We followed the designers metaphor and were not
  aware about the unity of the object-label and of
  the process!!!

23
Configuration evaluation the new interface
24
menu
Register web resource
25
First break in the new interface
26
Second break in the new interface
Step 1 Create label
27
Configuration evaluation results

• The users do not understand
• 1. The distinction global/my resources/labels
• 2. The steps to create the labels
• ? The broken pattern

28
The User Mental Model

• The label is a web resource, composed by two main
  clusters of information
• The description of the web resource it refers to
  (name, URL)
• The quality criteria selected according to one or
  more vocabularies
• ? the label format resumes both the information
  clusters
• 2. The certification process is a continuum

29
Suggestions

• Maintain a left menu with two access one for the
  descriptions and one for the criteria
• Use tabs in the main space to show in the
  perceptual space of the user both the description
  form and the form to select criteria

30
Suggested improvement
Resource description
Criteria selection
delete
31
A problem of perception
32
The conceptual tools

• A semiotic approach to analyze the communication
  process (the De Souza epistemic tool are used
  here)
• The star-life cycle
• A participatory-oriented analysis (elaborating
  together with a domain representative the
  suggestion to improve the system)
• ? related to the SIM or CEM method proposed by De
  Souza, it is fundamental to involve users in the
  process of evaluation (not only observing them,
  but also in the debriefing and in the improving
  suggestions)

33
Exercise

• Describe the improved system proposal using user
  and HCI patterns
• Describe the system interaction using the
  state-chart formalism
• http//tethis.iit.demokritos.gr8280/aqua/seam_log
  in.seam
• Usr/pwd unimi