Automatically Personalizing User Interfaces

1
Automatically Personalizing User Interfaces

• Daniel Weld
• University of Washington

2
Acknowledgements

• Corin Anderson
• Oren Etzioni
• Pedro Domingos
• Krzysztof Gajos
• Keith Golden
• Cody Kwok
• Tessa Lau
• UW AI Group
• NSF, ONR, NASA, DARPA, WRF

3
Two Interface Trends
Usage
4
Steelcase-Inspired Software
-- David Gelernter

• One Size Fits All

• Need Increased Adaptivity
• Beyond inconsistent defaulting
• Adapt to available devices, interaction modes
• Adapt to user location, tasks goals
• Adapt to user calendar current time
• Adapt to network connectivity,

5
Adaptivity Customization

• Consistent Across Applications
• Adaptation in every dialog
• Bridging Applications
• Data gathering
• Transformations
• Deep Deployment OS layer

6
Outline

• Motivation
• Deliberative Software Agents
• Programming by Demonstration
• Adapting User Interfaces to
• Screen Size
• User Behavior

7
Genesis Internet Softbots
Etzioni Weld CACM 94

• Interface Principles
• Goal-Oriented
• Integrated
• Balanced
• Safe

User says what she wants Agent determines how
when to achieve it ? planning-based softbots
Single, expressive uniform interface E.g.,
printing files vs. FTPing them
DT-lite Softbot must balance the cost of
finding information on its own with the nuisance
of asking the user.
Asimovs first law Safety, tidiness
thriftiness constraints in planner
8
UW Softbot Family Tree

• Specifying goals is bottleneck
• Logical spec. language ?!_at_!
• Forms interface
• Limited to anticipated use
• Not scalable
• Often easier to just do task!
• Except data integration tasks

• Natural language interfaces
• Popescu et al Yates et al 03
• Programming by demonstration

9
Outline

• Motivation
• Deliberative Software Agents
• Programming by Demonstration
• Adapting User Interfaces to
• Screen Size
• User Behavior

10
Programming by Demonstration
Lau Weld IUI-99
Wolfman et al. IUI-01
Lau et al. ICML-01

• If its too hard for users to specify goals
• Lets watch them
• And try to help
• Like plan recognition
• Initial domains
• Email
• Text editing
• Cross domain

11
Objectives
Demand on user
Expressiveness
Domain engr
Robustness
Macro Record
None
On / Off
Brittle
Straight Ln
12
SmartEdit 1
Lau et al. ICML-01
13
2
14
3
15
4
16
5
17
6
18
PBD as a learning problem

• Action is function input state output state
• Editor state text buffer, cursor position, etc.
• Actions move, select, delete, insert, cut,
  paste,
• Given a state sequence, infer actions
• Many actions may be consistent with one example
• Challenge Weak bias low sample complexity

19
Version Space Algebra

• Version space set of complex functions
• Define version space hierarchically
• Algebraic operators for combing VSs
• Union È analogous to set union
• Join cross-product with consistency
  predicate
• Transform convert functions to different types
• Can factor a version space

20
SMARTedit's version space
21
Fast Consistency

• Test consistency of example against entire
  version space
• Quickly prune subtrees

• Innovations
• Independent join allows BSR representation

22
Preliminary User Study

• 6 undergrad CS majors
• 7 repetitive tasks with later w/out SMARTedit
• Tasks 4 to 27 iterations, 1-5 min to complete
• Evaluation metrics
• Time saved completing task with SMARTedit's help
• user actions (keyboard mouse) saved
• User feedback

23
Time saved using SMARTedit
Time (sec) Cost Saved
Six Users
X X
Task Number
24
Action Savings with SMARTedit
Six Users
Percent of Actions Cost Saved
XXXX
X X
Task Number
25
Observations from PBD

• Overhead of macro recorder UI is high
• Most repetitive tasks short
• How many shell scripts do you write / day?
• Should focus on pure adaptivity
• E.g., automatic segmentation

26
Outline

• Motivation
• Deliberative Software Agents
• Programming by Demonstration
• Adapting User Interfaces to
• Screen Size
• User Behavior

27
Early Adaptation Mitchell,Maes

• Predict Email message priorities
• Meeting locations, durations

• Principle 1 Defaults minimize cost of errors
• Principle 2 Allow users to adjust thresholds

28
Adaptation in Lookout Horvitz
Adapted from Horvitz
29
Resulting Principles
Horvitz CHI-99

• Decision-Theoretic Framework
• Graceful degradation of service precision
• Use dialogs to disambiguate
• (Considering cost of user time, attention)

Adapted from Horvitz
30
Principles About Invocation

• Allow efficient invocation dismissal

• Timeouts minimize cost of prediction errors

31
Adapting to Device Characteristics
Custom Interface Rendering
Func Interface Spec
Device Model

• Approaches
• Templates
• Expert System
• Optimization

Hierarchy of State vars Methods
Screen size Available widgets Interaction modes
32
Optimize Widgets Layout

• Available widgets F(type(state-var))
• E.g. for an integer
• For hierarchy
• Tabs, adjacent panes,

33
SUPPLE
Gajos Weld, draft 2003

• Optimization methods
• Branch bound vs. local search
• CSP MRV, FC
• Cost function minimize user effort
• Navigation manipulation
• Component importance (? trace data)

34
SUPPLE Output
35
SUPPLE Output
36
SUPPLE Output

• 260,000,000 possibilities
• Simulated annealing fastest
• Can improve with bin-packing methods

37
Web Pages on Small Screens
38
Web Site Adaptation in Proteus
Anderson et al. WWW-01

• Personalizing in two steps
• 1. Learn model of visitor from access logs
• Transform content per learned model
• Hill-climbing thru space of websites
• Transforms shortcuts elision
• Expected utility of candidate site
• Sum of utility of each screen of each page
• Discounted by access difficulty (links,
  scrolling)

39
Analysis of Proteus

• Why Proteus worked well
• Suggested useful shortcuts
• Elided mostly unnecessary content
• Why Proteus worked poorly
• Sometimes elided useful content
• Users didnt find shortcut, tho it existed
• Flaws with implementation, not concept

Links followed
40
Principles

• Eliminating features is dangerous
• Accurate prediction is crucial
• Saliency of new UI operations is crucial
• How name shortcuts?
• Must partition dynamicity
• Maintain separate dynamic static areas
• Always allow previous navigational methods
• Duplicate functionality if necessary

41
Partitioned Dynamism
42
Partitioned Dynamism
43
Partitioning Failure
44
Principles

• Must partition dynamicity
• Accurate prediction also crucial

45
Predicting User Behavior
Anderson et al. IJCAI-01

• Model as Sequential Process
• Markov Models
• Mixtures of Markov Models
• Second-Order
• Conditioning on Position in Trace
• Etc.

times s?d was followed
P(s?d)
Total visits to s
46
Weakness of Markov Models

• Each state is trained independently
• Abundant training data at one state cannot
  improve prediction at another state
• Large state models ? vast training data

• Problem Web trace data is sparse
• A single visitor views 0 of any site
• New dynamic content not in training data

47
Reasoning about Uncertainty
DPRM
PRM
Sanghai et al. Thurs 1210 Cholula 3
Bayes Net
DBN
Structure
Relational
RMM
Sequence
Markov Model
48
Relational Markov Models
Anderson et al. KDD02

• Domains often contain relational structure
• Each state is a tuple in relational DB sense
• Structure enables state generalization
• Which allows learning from sparse data

ProductPage
StockLevel
ProductName
in_stock
Apple_iMac
backorder
Palm_m505
49
Defn Markov Model
Relational

• Q set of states
• Pages in a web site

• Each state a relation
• ProductPage(Apple_iMac, in_stock)

• p init prob distribution

• A transition probability matrix

• R a set of relations

50
Domain Hierarchies

• Instance of relation
• with leaf values is a
• state, e.g.
• ProductPage(iMac, in_stock)

ProductName
51
Domain Hierarchies

• Instance of relation with
• non-leaf values is a set of
• states an abstraction, eg
• ProductPage(AllComputers, in_stock)

ProductName
52
E-commerce Site Markov Ver.
m505_backorder.html
main.html
checkout.html
iMac_instock.html
dell4100_instock.html
53
RMM generalization

• Want to estimate P(s ? d) but no data!
• Use shrinkage

d
?
54
RMM generalization

• Want to estimate P(s ? d) but no data!
• Use shrinkage

• Can do this with abstractions of d and s

• Let ? be an abstraction of s and ? of d

?
55
RMM generalization

• Want to estimate P(s ? d) but no data!
• Use shrinkage

• Can do this with abstractions of d and s
• Let ? be an abstraction of s and ? of d

?
?
?
56
RMM generalization

• Want to estimate P(s ? d) but no data!
• Use shrinkage

• Can do this with abstractions of d and s
• Let ? be an abstraction of s and ? of d

57
Calculating Shrinkage Weights

• Intuitively, the lab should be large when
• Abstractions are more specific
• Training data is abundant
• Three methods for assigning weights
• Uniform
• Heuristic (Based on lattice depth, of examples)
• EM (Data intensive)

58
Gazelle
59
The Google Generation

• Most WWW traces very short
• Cant beat trace 2
• Not true in desktop / mobile apps
• Adapt to user behavior
• Adapt to device characteristics

60
Striving for Duplex
61
Still Striving for Duplex
62
Finally!
63
Confirm (Twice!)
64
State machine (partial)
Six clicks required!
65
Remember Partitioned Dynamicity

• Proteus Users didnt find shortcuts!
• Saliency of new UI operations is crucial
• Must partition dynamicity
• Maintain separate dynamic static areas
• Duplicate functionality

66
With Controlled Adaptation
Maintain Stable Navigation
Optimize For User Behavior
67
Or Rather
Curry to Boolean
68
Challenges for AI

• Formalizing interface descriptions
• Pebbles, iCrafter, XIML,
• Customization languages
• Rec start/stop shortcut make default
• Datamining the clickstream
• Improved prediction ? dim ? data
• Interface transformation algorithms
• Complex constraint satisfaction

69
Conclusion
High-level Customization

• Goal-oriented softbots
• Programming by demonstration
• Adaptive interfaces / websites

Pure Adaptation

• Principles
• Partitioned Dynamicity

• Techniques
• Version-Space Algebra
• Relational Markov Models

70
The End
71
Abstract

• Todays computer interfaces are one size fits
  all. Users with little programming experience
  have very limited opportunities to customize an
  interface to their task and work habits.
  Furthermore, the overhead induced by generic
  interfaces will be proportionately greater on
  small form-factor PDAs, embedded applications and
  wearable devices. Searching for a solution,
  researchers argue that productivity can be
  greatly enhanced if interfaces anticipated their
  users, adapted to their preferences, and reacted
  to high-level customization requests. But
  realizing these benefits is tricky, because there
  is an inherent tension between the dynamism
  implied by automatic interface adaptation and the
  stability required in order for the user to
  predict the computers behavior and maintain
  control. This talk will list challenges for the
  field, describe principles governing effective
  adaptation, and present new algorithms for data
  mining user action traces and dynamically
  transforming interfaces.

72
Gentle-Slope Systems
C
ClicknCreate (Multimedia Fusion)
VBasic
Hypercard
MFC
C
C Plugins
kCmds
Difficulty of use
Hypertalk
Ideal
Basic
Sophistication of what can be created
Adapted from Myers et al.
73
Learning programs from traces

• Illustrated VS algebra with SMARTedit
• Grand vision cross-domain PBD
• Learn an abstract programming language
• SMARTpython learn Python programs
• Lau PhD thesis
• Loops, conditionals,

74
Reqs for Interface Representation

• Same as Pebbles Project?
• State variables
• Typing
• Commands
• Hierarchical organization

75
Adaptation in Pebbles
See next talk
76
State abstractions form lattice
ProductPage(AllProducts, AllStockLevels)
ProductPage(AllProducts, in_stock)
ProductPage(AllComputers, AllStockLevels)
ProductPage(AllComputers, in_stock)
ProductPage(AllDesktops, AllStockLevels)
ProductPage(AllDesktops, in_stock)
ProductPage(AppleDesktops, AllStockLevels)
ProductPage(AppleDesktops, in_stock)
ProductPage(iMac, AllStockLevels)
ProductPage(iMac, in_stock)
77
Guiding the Search

• Expected utility based on model of visitor
• Model learned by mining server access logs
• Sum value of each screen of each page
• Discount by difficulty of reaching screen from p
• Depends on how manylinks followed and howmuch
  scrolling required

p
78
Specifics of Screen Evaluation
Expected utility of sij
79
Proteus Empirical Study

• Observe real users on the desktop
• Info-seeking goals drawn from random distribution
• Personalize based on observations
• Measure performance on mobile device
• Number of links and scrolls, amount of time
• Compare unmodified and personalized sites
• Half users did unmodified first, others vice versa

80
Average number links followed
81
RMM generalization

• Want to estimate P(s ? d)
• Use shrinkage

• Can do this with abstractions of d and s
• Let ? be an abstraction of s and ? of d

?
?
?
82
Backup

• Want to estimate P(s ? d) but no data!
• Use shrinkage

• Can do this with abstractions of d and s
• Let ? be an abstraction of s and ? of d

?
?
?