Text-Free User Interfaces

1
Text-Free User Interfaces

• Kentaro Toyama
• Assistant Managing Director
• Microsoft Research India
• Based on work with Indrani Medhi
• TCS Excellence in Computer Science
• January 10, 2008 Pune, India

2
People

• Lead Researcher
• Indrani Medhi
• Collaborators
• Kentaro Toyama
• Archana Prasad
• Vibhore Goyal
• Shibili G
• Nimmi Rangaswamy
• Sean Blagsvedt
• Interns
• Aman Sagar (Adobe)
• Bharathi Pitti (CMU)
• Chandima Patabandhige (Univ. of Moratuwa, Sri
  Lanka)
• Renee Kuriyan (UC Berkeley)

Photo Indrani Medhi
Indrani (holding baby) and some interns visiting
a family in Bangalore
3
Outline Text-Free User Interfaces

• Introduction
• Initial ethnography and iterative rapid
  prototyping
• Initial evaluation
• Optimal audio-visual representation
• Full-context video
• Methodology
• Discussion

4
Outline Text-Free User Interfaces

• Introduction
• Initial ethnography and iterative rapid
  prototyping
• Initial evaluation
• Optimal audio-visual representation
• Full-context video
• Methodology
• Discussion

5
Illiteracy

• 1-2 billion illiterate population in the world.
• 98 live in developing countries.
• Indias rate of literacy estimated at 60.
• Many labeled literate are semi-literate with
  low functional literacy.

Photo Indrani Medhi
Two residents of a slum in Jayanagar, Bangalore
6
Target Users

• Women from Bangalore slums
• Informal sector domestic workers
• Income range INR 800-2500 (US20-60) per month
• Illiterate or semi-literate
• Most have never seen a PC (those who have seen
  have never touched)

Photo Indrani Medhi
initial explorations before text-free UI work
7
NGO Partner

• Stree Jagruti Samiti
• Head Geeta Menon
• Small NGO
• Works with several Bangalore slums
• Focus on childrens rights, womens empowerment,
  fair wages for domestic laborers

Photo Indrani Medhi
Geeta Menon leading a Saturday Stree Jagruti
Samiti meeting
8
The Problem
How can an application UI be converted into one
that is usable by non-literate users?
9
Outline Text-Free User Interfaces

• Introduction
• Initial ethnography and iterative rapid
  prototyping
• Initial evaluation
• Optimal audio-visual representation
• Full-context video
• Methodology
• Discussion

10
Initial Ethnography

• TVs, radios, CD players common some mobile
  phones no PCs
• Established slums some with 30-year history in
  city concrete housing
• Subjects most concerned about
• Jobs
• Healthcare
• Education

Photo Indrani Medhi
Nimmi interviewing a subject
11
Text-Free UI?

• Design principles
• Pen or touch interface
• Liberal use of imagery
• No text
• Semi-abstracted cartoons
• Voice annotation
• Aggressive use of mouse-over functionality
• Consistent help icon

Naukri.com for domestic labourers?
12
Nouns vs. Verbs
Text-Free UI Details

• Goal is to achieve exact association between
  image and word.
• Without action cues drawings tend to be
  interpreted as nouns.
• With action cues, drawings tend to be interpreted
  as verbs.
• Phenomenon well-understood by cartoonists and
  diagram designers.

Kitchen sink or washing dishes?
Pot or cooking?
13
Cultural Differences
Text-Free UI Details

• Goal is to achieve as global a representation as
  possible.
• Abstracted representations are very dependent on
  culture.
• Minor misrepresentations can result in major
  confusion.
• Testing, testing, testing!

An urban family of three?
14
Religious Differences
Text-Free UI Details

• Goal is still to achieve as global a
  representation as possible.
• Religious differences can be the strongest of
  cultural differences.
• There is no neutral culture or religion.
• Testing, testing, testing!

From when to when?
Increasingly general representations for time
indication
15
Quirks of Non-Literacy
Text-Free UI Details

• Various degrees of literacy
• Total illiteracy
• Numbers, but not alphabets
• Alphabets, but not words
• Words, but not sentences
• Semi-literate, but not fluent
• Etc.
• Many who are non-literate can read numbers
• Accustomed to currency
• Able to do basic arithmetic
• Not necessarily 100 accurate
• Testing, testing, testing!

Illiteracy doesnt necessarily mean inability to
read numbers.
16
Text-Free UI?
Original design
Revised design
17
Text-Free UI

• Design principles
• Pen or touch interface
• Liberal use of imagery
• No text (but numbers OK)
• Semi-abstracted cartoons
• Aggressive use of mouse-over functionality
• Voice feedback
• Consistent help icon

Screenshot of text-free job search
18
Other Domains

• Principles transfer to other applications

Text-free maps
Text-free healthcare information
19
Outline Text-Free User Interfaces

• Introduction
• Initial ethnography and iterative rapid
  prototyping
• Initial evaluation
• Optimal audio-visual representation
• Full-context video
• Methodology
• Discussion

20
Questions
Initial Evaluation

1. Can non-literate users use traditional text-based
   user interfaces at all?
2. Do the proposed design principles for text-free
   user interfaces allow non-literate users to use
   computers, and to what extent?
3. Which of the design principles make the most
   difference for a text-free UI?

21
Experimental Set-Up
Initial Evaluation

• Three versions
• Text-based
• Text-free without help icon
• Text-free with help icon
• Subjects
• Illiterate or semi-literate
• (no literate subjects)
• No PC experience
• Subject grouping
• Five-women group x 2
• Individual x 4
• Total time up to one hour per session

• Task
• For a friend who is unemployed, find the
  best-paying job in her neighborhood.
• Measured
• Task completion (yes/no)
• Time required (second)
• Number of prompts required (n)
• All comments recorded some trials
  video-recorded.

22
Quantitative Results
Initial Evaluation

• Text-based UIs were completely unusable.
• Obvious, but first formal such test in literature
• With text-free UI, 30 were able to complete
  task.
• (However, not enough participants for statistical
  significance.)

Tabulation of initial results
23
Qualitative Results
Initial Evaluation

• Strong preference for text-free principles, in
  particular
• Voice annotation
• Help icon
• Collaborative use more successful
• Subjects eager to engage, once comfortable with
  set up
• Various barriers to technology fear, lack of
  awareness, lack of cognitive model

Indrani and a subject during initial evaluation
24
Questions
Initial Evaluation

1. Can non-literate users use traditional text-based
   user interfaces at all?
2. Do the proposed design principles for text-free
   user interfaces allow non-literate users to use
   computers, and to what extent?
3. Which of the design principles make the most
   difference for a text-free UI?

NO
Yes, but only partially more work required

• Hypothesis
• Voice feedback
• - Imagery
• Help

25
Outline Text-Free User Interfaces

• Introduction
• Initial ethnography and iterative rapid
  prototyping
• Initial evaluation
• Optimal audio-visual representation
• Full-context video
• Methodology
• Discussion

26
Text-Free UI
From initial round of research

• Design principles
• Pen or touch interface
• Liberal use of imagery
• No text (but numbers OK)
• Semi-abstracted cartoons
• Voice feedback
• Aggressive use of mouse-over functionality
• Consistent help icon

Screenshot of text-free job search
27
Goal of Study

• Determine what types of icon representations are
  best for UIs for non-literate users.
• Audio
• With voice annotation
• Without voice annotation
• Visual
• Photorealistic video
• Hand-drawn animation
• Photograph
• Hand-drawn image
• Text

28
Domain of Testing

• Desired characteristics of domain
• Admits to visual representations,
• Consists of information which subjects are not
  exposed to on a daily basis,
• Has universally recognized meaning,
• Offers a range of cognitive complexity,
• Offers a range of visual complexity, and
• Can be applied to a practical purpose.
• Healthcare / medical imagery
• 13 medical symptoms multimodal in nature,
  involving elements of color, temperature,
  proprioception, and some requiring motion
  spanning some temporal duration

29
Creation of Representations
Ensure consistency across representations

Photograph
Animation
Video
Drawing
30
Tested Representations
Photograph
Animation
Video
Drawing
Text
Each with and without voice annotation (total 10
representations), for 13 medical symptoms
31
Experimental Set-Up

• Representations
• One of 10 types
• Evenly distributed among subjects
• All 13 symptoms per subject
• Subjects
• Illiterate or semi-literate
• (no literate subjects)
• No PC experience
• Size
• Individual n 200
• 20 for each representation
• Total time up to one hour per session

• Threefold task
• Speak phrase that first comes to mind when shown
  a representation.
• Explain symptom representation is intended to
  convey
• Enact or depict the intended symptom (as
  explained by tester), for analysis
• Measured
• Accuracy (correct, close, wrong)
• Response time (seconds)
• All comments recorded

32
Quantitative Results

• Voice annotation critical, with dominant effect
• 30 quicker response time with audio
• Among cases with audio, hand-drawn
  representations (both animation and static
  imagery) most accurately recognized
• But, effect not dramatic
• Among cases without audio, animation most
  accurately identified
• 23 more accurate than static drawing

Number of accurate responses (out of 20) for
various representations
33
Qualitative Results

• Test / text anxiety of subjects
• Subjects uncomfortable with test-like environment
  of studies
• Short-term category conditioning
• Priming for aches over other types of symptoms
  when they appeared in sequence
• Richer information is not necessarily better
  understood overall
• Photorealism contains clutter associated with
  actual instance
• Context-laying activity provides additional in
  dynamic imagery
• Socio-economic conditions correlated with
  cognition
• Those with less formal education had difficulty
  processing visuals and audio at the same time
  confused by multi-modal interface

34
Outline Text-Free User Interfaces

• Introduction
• Initial ethnography and iterative rapid
  prototyping
• Initial evaluation
• Optimal audio-visual representation
• Full-context video
• Methodology
• Discussion

35
Quantitative Results
From initial evaluation

• Text-based UIs were completely unusable.
• Obvious, but first formal such test
• With text-free UI, 30 were able to complete
  task.
• However, not enough participants for statistical
  significance.

Help icon
only!!!
Tabulation of initial results
UI instruction video
Insufficient for end goal!
36
Qualitative Results
From initial evaluation

• Strong preference for text-free principles, in
  particular
• Voice annotation
• Help icon
• Collaborative use more successful
• Subjects eager to engage, once comfortable with
  set up
• Various barriers to technology fear, lack of
  awareness, lack of cognitive model

• From post-experiment interviews
• Afraid to break PC
• Intimidated by use of complex technology
• Confusion as to value of PC for job search
• Lack of understanding about how PC contains
  relevant information
• Task better accomplished by asking people
• Verbal explanations by tester not convincing

Indrani and a subject during initial evaluation
37
New Problem!
Original question
How can an application UI be converted into one
that is usable by non-literate users?
ILLITERACY
FEAR OF TECHNOLOGY
LACK OF TRUST IN TECHNOLOGY
LACK OF AWARENESS OF WHAT TECHNOLOGY CAN DELIVER
New question
Can a UI be developed to allow an illiterate,
first-time PC user to access information he/she
needs without any assistance or prompting?
38
Whats the Underlying Problem?
If you were told that, if you put anything in
this box and spelled the objects name in reverse
100 times out loud, it would come to life
Any sufficiently advanced technology is
indistinguishable from magic. Arthur C. Clarke
would you bother to do it?
39
Solution?

• Observations about TV
• Many households have TVs
• Soap operas a common topic of discussion
• Even households without TVs are familiar with TV
• Observations about subjects cognitive styles
• Subjects engage in long linear narratives
• General ideas conveyed through specific instances
• Stories included real people, specific objects,
  or actual instances of events
• (Based on intuition developed through prolonged
  engagement with subjects!)

Televisions common in households
40
Full-Context Video
Solution Full-Context Video
A full-context video explains the broader context
of the application and how it works, in addition
to instructional material about how to use the
application.
41
Experimental Set-Up
Full-Context Video

• Two versions
• Original text-free (without full-context video)
• Text-free with full-context video
• Subjects
• Illiterate or semi-literate
• (no literate subjects)
• No PC experience
• Subject grouping
• Individuals x 35
• Total time up to one hour per session

• Task (the same)
• For a friend who is unemployed, find the
  best-paying job in her neighborhood.
• Measured
• Task completion (yes/no)
• Time required (second)
• Number of prompts required (n)
• All comments recorded some trials
  video-recorded.

42
User Studies
  Without Video (A) With Video (B)
Total Task Completed (out of 35) 8 35
Prompts reqd for completion 9.8 5.2
Avg completion time (min) 9.01 4.59
   
Sequence - AB (17 total) Sequence - AB (17 total) Sequence - AB (17 total)
Task Completed - AB 1 17
Prompts reqd for completion 11 5.9
Avg completion time (min) 8.2 8.6
   
Sequence - BA (18 total) Sequence - BA (18 total) Sequence - BA (18 total)
Task completed BA 7 18
Prompts reqd for completion 6 4.7
Avg completion time (min) 10.8 6.5

• Full-context video has clear value
• Without video first, only one out of 17 (6) was
  able to complete the task at all, taking 11
  prompts and 8.2 minutes
• With video first, 18 out of 18 completed the
  task, with an average of 4.7 prompts and 6.5
  minutes

43
Qualitative Results

• Other observations
• Round-two subjects were incredulous that
  round-one subjects didnt understand the
  application.
• Impact of video not permanent for most subjects.
  Many wanted to see the full-context video each
  time, even after seeing it before.
• Full-context video appears to increase
  motivation, as well as performance.
• Those who saw full-context video were interested
  in providing feedback on the specifics of the UI.

Photo Indrani Medhi
44
Text-Free UI

• Design principles
• Pen or touch interface
• Liberal use of imagery
• No text (but numbers OK)
• Semi-abstracted cartoons
• Aggressive use of mouse-over functionality
• Voice feedback
• Consistent help icon
• Full-context video

Full-context video playing at start of application
45
Outline Text-Free User Interfaces

• Introduction
• Initial ethnography and iterative rapid
  prototyping
• Initial evaluation
• Optimal audio-visual representation
• Full-context video
• Methodology
• Discussion

46
Initial Ethnography
Methodology

• Open-ended interviews
• Various combinations focus groups and
  individuals
• Topic everything and anything
• Follow leads around hardships
• Structured interviews
• Focus on themes from open-ended interviews
• Prepare questions
• Be willing to deviate
• Goal Information, intuition, rapport

Photo Indrani Medhi
An open-ended interview in a subjects home
Time and attention critical!
47
Iterative Prototyping
Methodology

• Build initial prototype, based on intuition from
  interviews.
• Iterate prototype and test.
• Maintain informality
• Work with small groups
• 2-4 people
• Prompt whenever necessary
• Record every comment
• Video-record, if possible (but, high procedural
  overhead)
• Listen to every comment
• Modify prototype
• Refine until steady state.

Photo Indrani Medhi
Testing a prototype
48
Participative Design
Methodology

• Modify standard methods for non-literate
  partners.
• Encourage storytelling.
• Use aids that can be physically manipulated.
• Be prepared for long, counseling-like sessions.

Photo Indrani Medhi
Eliciting descriptions of medical symptoms with
physical design aids
Subject enacting medical symptoms
49
Formal Evaluation
Methodology

• Base on techniques from courses, textbooks
• Plan according to standard practices
• Avoid direct counterfactuals
• Place task in a story
• Use friend or neighbor as protagonist, but not
  subject
• Chavans Bollywood method
• Be flexible on
• Location (avoid labs)
• Formal instructions
• Etc.

Video Archana Prasad
Scene from a full-context video hypothetical
user and NGO worker
50
Outline Text-Free User Interfaces

• Introduction
• Initial ethnography and iterative rapid
  prototyping
• Initial evaluation
• Optimal audio-visual representation
• Full-context video
• Methodology
• Discussion

51
Related Work

• Interfaces for non-literate users
• Huenerfauth 2002
• Chand 2002
• Parikh et al. 2003
• Visual representations
• Modley et al. 1976
• Horton 1996
• Tversky et al. 2002

• Video tutorials
• DeBloois et al. 1984
• Rickman et al. 1986
• Bransford et al. 1992
• Much more related work in these categories (see
  papers)
• Text-Free UI novelty
• Focus on non-literate users
• Rigorous evaluation
• Full-context video

52
Publications

• Medhi, I., Pitti B. and Toyama K. Text-Free UI
  for Employment Search. Asian Applied Computing
  Conference (AACC2005). Nepal, December 2005.
• Medhi, I., Sagar A., and Toyama K. Text-Free User
  Interfaces for Illiterate and Semi-Literate
  Users. International Conference on Information
  and Communication Technologies and Development
  (ICTD2006). Berkeley, USA, May 2006.
• Medhi, I., Prasad, A. and Toyama K. Optimal
  Audio-Visual Representations for Illiterate
  Users. International World Wide Web Conference
  (WWW2007). Banff, Canada, May 2007
• Medhi, I. and Kuriyan R. Text-Free UI Prospects
  for Social Inclusion. International Conference on
  Social Implications of Computers in Developing
  Countries (IFIP 2007). Brazil, May 2007
• Medhi, I. and K. Toyama. CHI Workshop position
  paper on User-Centered Design and International
  Development, San Jose, USA, April 2007
• Medhi, I., K. Toyama. Full-Context Videos for
  First-Time, Illiterate Users. alt.chi forum at
  ACM CHI '07, San Jose, USA, April 2007
• Medhi, I. User-Centered Design for Development.
  ACM Interactions. COLUMN Forum under
  development Volume 14 , Issue 4 (2007), pp. 12
  14.
• Medhi, I., C. R. Patabandhige, K. Toyama. Cartoon
  Generation for Text-Free User Interfaces. 1st
  International Workshop on Computer Vision
  Applications for Developing Regions in
  Conjunction with Eleventh IEEE International
  Conference on Computer Vision (ICCV 2007
  Workshop), Rio de Janeiro, Brazil, October 2007
• Medhi, I., Toyama, K. Full-context video for.
  International Conference on Information and
  Communication Technologies and Development
  (ICTD2007). Bangalore, India, December 2007.

53
Text-Free UI History
Text-Free healthcare information
Online site (babajob.com)
Practical Systems
UI authoring tool
Optimal AV representa- tion
Initial principles
Text-Free UI for mobiles
UI Research
Full-context video
Semi- automated cartooning
Psychology of non-literate users
Other Research
Key
Completed
Ongoing
Future
54
Summary Text-Free User Interfaces

• Introduction
• Initial ethnography and iterative rapid
  prototyping
• Initial evaluation
• Optimal audio-visual representation
• Full-context video
• Methodology
• Discussion

55
Thank you!
Photo Indrani Medhi

• http//research.microsoft.com/research/tem
• indranim,kentoy_at_microsoft.com