Vocabularies for Description of Accessibility Issues in MMUI

1
Vocabularies for Descriptionof Accessibility
Issues in MMUI
eljko Obrenovic, Raphaël Troncy, Lynda
Hardman Semantic Media Interfaces, CWI,
Amsterdam zeljko.obrenovic_at_cwi.nl
http//www.cwi.nl/obrenovi/
2
Introduction

• What (output) modalities aremost suitable in
  which situation?
• How should different (output) modalitiesbe
  combined?
• Using rich accessibility descriptionof MM UIs to
  answer these two questions

3
Motivation

• Expressing human functionality andanatomical
  structures required by modalities
• Capturing existing knowledge
• Reasoning over modalities
• Limited scope, small non-standard vocabularies
• Card, Mackinlay, Robertson
• Morphological analysis of input device
• Modality theory, Niels Ole Bernsen
• Based on taxonomy of output modalities
• Linguistic, analog, arbitrary, static, media

4
Describing Accessibility Issuesin Multimodal
User Interfaces

• MM user interfaces - systems that communicate a
  message, an effect,
• Stimulating a particular human functionality or
  anatomical structures
• Interaction constraints
• Influence of various factors on human anatomical
  structures and functionalities.

5
Describing Accessibility Issuesin Multimodal
User Interfaces
Accessibility issues (interaction constraints)
User constraints
Social constraints
Environment constraints
Device constraints
Computer
Human
Sensing
Modality
Perception
Modality
Motor skills
Multimodal issues
Linguistic skills
Modality
Cognition
6
Interaction Modalities
Vocabularies
described in terms of
Interaction Constraints
described in terms of
Interaction Context
ReasoningFramework
query
result
Applications
define
7
WHO Resources
Vocabularies
Interaction Modalities
Bioinformatics
described in terms of
Interaction Constraints
Additional concepts
described in terms of
Interaction Context
ReasoningFramework
query
result
Applications
define
8
Vocabularies ICF1/2

• WHO International Classification ofFunctioning,
  Disability and Health (ICF)
• 2001, 9 years revision, widely used in health
  community
• Describing the person in his/her world
• Applicable to all people, whatever their health
  condition
• Carefully designed to be relevant across
  culturesas well as age groups and genders
• Uses neutral terms (function vs. disease)
• Allows for an assessment of the degree of
  disability

9
Vocabularies ICF2/2

• Around 1500 concepts
• Body functions and structure
• Activities (tasks and actions by individual)and
  participation (involvement in a life situation)
• Environmental factors
• Problems with formalization
• Not defined by knowledge engineering experts
• Formalized ICF Checklist 180 core concepts
• Possibility to reuse millions of profilesand
  statistical data expressed by ICF!

10
Vocabularies Anatomy

• Foundational Model of Anatomy (FMA)
• Detailed description of human anatomy
• Open source and available for general use
• Used in bioinformatic community
• 100 000 concepts
• Available in OWL format
• Problems size

11
Vocabularies Interaction Effects

• Some examples
• Gestalt visual grouping
• By similarity, motion, texture, symmetry,
  proximity, parallelism, closure, good
  continuation
• Gestalt visual highlighting
• By color, polarity, brightness, orientation,
  size, motion, flicker, depth, shape
• 3D cues
• Visual stereo vision, motion parallax, linear
  perspective, rel. size, shadow, familiar size,
  interposition, horizon
• Audio inter-aural time/intensity difference,
  HRTF, head movement, echo, attenuation of high
  frequencies

12
WHO Resources
Vocabularies
Interaction Modalities
Bioinformatics
described in terms of
Interaction Constraints
Additional concepts
described in terms of
Interaction Context
ReasoningFramework
query
result
Applications
define
13
Example Modality Speech
14
Example Constraint Noise
15
WHO Resources
Vocabularies
Interaction Modalities
Bioinformatics
described in terms of
Interaction Constraints
Additional concepts
described in terms of
described in terms of
Interaction Context
Userinterface
ReasoningFramework

• Device profile
• User profile
• Environment profile

query
result
Applications
define
16
Reasoning Using Descriptions

• Possible with rich and explicit descriptionsof
  (implicit) accessibility requirements
• What modalities are suitable in which situation?
• Combining descriptions of constraintsand
  modalities (speech in noisy environment?)
• How should different modalities be combined?
• Combining modality descriptions to identify
  conflicting requirements (speech short term
  memory)

17
Implementation

• In early stage
• Using Semantic Web technologies
• OWL, RDF standards
• Tools and database to support reasoning
• Sesame, Jena
• Reusing some of our and existing toolsfor data
  visualization and exploration
• Facet browsing

18
Conclusions

• Expressing human functionality andanatomical
  structures required by modalities
• Relations with other projects
• W3C Multimodal Interaction Framework - content
• W3C Accessibility Initiative (WAI) - guidelines
• Future work
• Full implementation of reasoning framework
• Solving problems with vocabularies
• No relations among concepts, overlapping
• New vocabularies are coming
• The Human Brain Project (with FMA)
• DARPA Digital Human unifying medical ontologies