Huddle and Information Triggers: Helping Users Automate Complex and Temporally-Distant Tasks

1
Huddle and Information Triggers Helping Users
Automate Complex and Temporally-Distant Tasks

• Jeffrey Nichols
• IBM Almaden Research Center
• UC Berkeley
• June 14, 2007

2
Today Two Mini Talks

• Huddle
• Automatic generation of interfaces for systems of
  connected appliances
• Part of my dissertation work on the Personal
  Universal Controller
• Information Triggers
• Helps users recall important information or tasks
  at the moment they are important
• New project at IBM Almaden Research Center
• In collaboration with Jake Wobbrock at the
  University of Washington

3
The Multi-Appliance Problem
4
The Multi-Appliance Problem, cont.
How can we provide improved interfaces for
systems of connected appliances?
5
Control of Multiple Appliances
Universal Remote Controls
Built-in Interfaces
Custom Installations
6
Huddle

• Improved multi-appliance interfaces based on
  content flow

Flow-Based Interface
Aggregate User Interfaces
7
Personal Universal Controller (PUC)

• Automatically generates interfaces for remotely
  controlling appliances Nichols, UIST 2002
• Specification language for describing appliance
  functionality
• Dependency information that describes the
  availability of functions based on appliance
  state
• Automatically generate new interfaces to be
  consistent with previous interfaces Nichols,
  CHI 2006
• Knowledge base that identifies and stores
  similarities between appliances

8
Related Work

• Roadie Lieberman, IUI 2006
• Goal-oriented natural language interface for
  appliances
• Uses planner to configure appliances based on
  common sense database (EventNet)
• ICrafter Ponnekanti, Ubicomp 2002
• Builds aggregate user interfaces based on
  service interfaces
• Hand-coded interface aggregators
• Limited automatic generation capabilities
• Omojokun, et al. Omojokun 2006
• Examined how remote control interfaces might be
  generated from recordings of actual usage
• Recorded remote control usage with IR receiver
• General Work on Interface Generation
• Task-based work TERESA Mori 2004, XIML
  Puerta, IUI 2002, others
• SUPPLE Gajos, IUI 2004

9
Huddle Overview

• Introduction
• Content Flow Modeling
• Flow-Based Interface
• Aggregate User Interfaces
• Discussion and Future Work

10
Content Flow Model

• Home Theater

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
11
Modeling Tasks

• Home Theater Watching Television

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
12
Modeling Tasks

• Home Theater Watching Television

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
13
Modeling Tasks

• Home Theater Watching Television

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
14
Modeling Tasks

• Home Theater Watch DVD

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
15
Modeling Tasks

• Home Theater Watch DVD Copy Tape

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
16
Modeling Tasks

• Presentation Room

VCR
PowerPoint
Projector
Laptop Task Manager
External Video Control
Media Player
17
Modeling Tasks

• Presentation Room Presenting PowerPoint

VCR
PowerPoint
Projector
Laptop Task Manager
External Video Control
Media Player
18
Where does the model come from?
DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
19
Where does the model come from?
20
Where does the model come from?
21
Flow-Based Interface (FBI)

• Allows user to manage high-level tasks
• Quick specification of flows
• AI Planner automatically configures appliances
• Question/Answer interface to address conflicts

22
Flow-Based Interface
23
Flow-Based Interface
24
Flow-Based Interface
25
Flow-Based Interface
26
Flow-Based Interface
27
Flow-Based Interface
28
Flow-Based Interface
29
Flow-Based Interface
30
Flow-Based Interface
31
Aggregate User Interfaces (AUIs)

• Allows user to manage low-level details
• Control or setup of active flows
• General setup of all appliances
• Merged functions

32
Aggregate User Interfaces

• Three kinds
• Control of Active Flows

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
33
Aggregate User Interfaces

• Three kinds
• Control of Active Flows

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
34
Aggregate User Interfaces

• Three kinds
• Control of Active Flows

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
35
Aggregate User Interfaces

• Three kinds
• Control of Active Flows

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
36
Aggregate User Interfaces

• Three kinds
• Control of Active Flows

DVD
Receiver
BroadcastTelevision
TV
VCR 1
VCR 2
37
Aggregate User Interfaces

• Three kinds
• Control of Active Flows

Copy Tape
Watch TV using TV speakers
Watch TV using Receiver speakers
38
Aggregate User Interfaces

• Three kinds
• Control of Active Flows
• Setup of Active Flows

Two screens of Watch DVD setup
39
Aggregate User Interfaces

• Three kinds
• Control of Active Flows
• Setup of Active Flows
• General Setup of all appliances

Two screens of General Setup for a home theater
config.
40
Aggregate User Interfaces

• Three kinds
• Control of Active Flows
• Setup of Active Flows
• General Setup of all appliances
• Merged Functions
• Most functions shouldnt be merged
• Some should
• clock, language, sleep timer
• Future work
• Baudischs flat volume control
• General merging

Some merged functions
41
Discussion

• How can our use of content flow be extended?
• Lighting in a presentation room
• Video-conferencing rooms
• Outside the home factories offices
• Kitchen?

42
Future Work

• Extending to more detailed content types
• Home theater support for HD signals, encoded
  audio, etc.
• Setup and Troubleshooting
• Tool for specifying initial wiring
• Helping with Problems
• More aggregate interfaces
• Usage-based aggregate?
• Usability evaluations

43
Information Triggers
44
Information Triggers

• Motivation

Its easy to find stuff on the web
Its even easy to re-find something youve
previously found...
...if you remember to re-find it when you need it
Participants in a re-finding study complained
that they often forgot to look for an information
item until the period of its usefulness had
passed. (Jones et al. 2005)
45
Information Triggers

• What weve built so far
• A Firefox extension that allows people to
  associate web pages with future events at which
  that page will be particularly relevant
• When the event occurs, the web page is
  automatically opened for the user
• Goal is to make triggers as easy and lightweight
  to configure as possible
• Supports three different types of triggers
• Date time
• Arrival of an e-mail
• Visit to a web page

46
Beyond Current Reminder Systems

• Existing reminder systems provide a short message
  about their topic, but info triggers provides the
  information itself
• Context-based reminder systems (e.g. Cybreminder)
  have mainly used location for context

47
Scenarios

• Waiting to login to a website to configure
  account information (date time trigger)

48
Scenarios

• Resuming a task that cant be completed in one
  sitting, e.g. waiting for manager approval (new
  e-mail trigger)

49
Scenarios

• Opening a note-taking site when visiting a page
  with specific keywords (page visit trigger)

information retrieval
web pages
50
Scenarios

• Delaying a task until a leisure period (page
  visit trigger)

51
Scenarios

• Opening a set of pages all at once, e.g. opening
  one bill payment site opens others (page visit
  trigger)

52
Scenarios

• Automating routine browsing tasks (date time
  trigger)

53
Demo
54
User Feedback

• Distributed the extension within IBM, currently
  have several regular users
• Needed features
• Means for configuring mail triggers within mail
  reader
• Ability to share triggers with others
• Uses include
• Reminders of information related to a current
  event
• Automatically display the web page/blog of a
  person you are about to meet with
• Regular reminders of a task prior to a deadline
• Make the conference registration page appear
  every morning on the week before the early
  registration deadline
• Automating phone call forwarding
• One user has a set of triggers that are able to
  automatically start and stop call forwarding from
  the office on days that he regularly works from
  home
• Automating regular browsing tasks
• Weekly entry of status reports

55
Discussion

• Programming triggers can be hard or require being
  clever
• Cant always predict when some information will
  be useful
• Current set of triggers is powerful, but making
  them do what you want may not be straight-forward
• Delaying until leisure time trigger
• Triggering the next time you open your browser
• Triggers that open pages that trigger other
  things
• Where should triggers be set?
• Triggered information vs. triggering information
• Need usability studies to improve authoring of
  triggers
• Deferring Triggered Information
• Moving the window to the background can be
  sufficient
• May need an additional explicit mechanism

56
Future Work

• Other Trigger Events?
• What other context information might be useful to
  trigger on?
• Sharing Triggers
• Can some of the overhead of sharing triggers be
  offset using social mechanisms?
• Distributed Triggers
• No one is always at their browser
• Trigger on mobile devices or across a users
  machines
• Trigger Queues
• A trigger that fires regularly associated with a
  queue of pages
• Each time the trigger fires the next item from
  the queue is displayed
• Interesting new articles from today, Quick
  online to-do items, Papers to read, etc.

57
Future Work, cont.

• Usability and Usage Studies
• Improve the interaction
• Is it useful? When?
• Deeper Questions
• How well can people anticipate when they will
  need information?
• How well can people anticipate when they will
  next be interruptable?

58
IBM Almaden Research Center

• 400 research employees 100 students and
  postdocs
• Research in Computer Science, Storage Systems,
  Science and Technology, Services Science
• HCI-related Departments
• User Focused Systems group in CS
• Services Sciences research

59
USER Group at Almaden

• Four main research thrusts
• Activities
• Unified Activity Management
• Koala
• Web 2.0 and Social Computing
• BlueMail
• Fringe
• Content Management
• AACS, etc.
• Interaction in Healthcare
• Data entry for EMR
• Communication management (awareness context)
• Patient-centered healthcare

60
Thanks for listening!

• For more information
• http//www.pebbles.hcii.cmu.edu/puc/
• http//www.almaden.ibm.com/
• http//www.jeffreynichols.com/

61
Huddle and Information Triggers Helping Users
Automate Complex and Temporally-Distant Tasks

• Jeffrey Nichols
• IBM Almaden Research Center
• UC Berkeley
• June 14, 2007

62
(No Transcript)
63
(No Transcript)
64
Info Triggers Demo
65
Info Triggers Demo
66
Info Triggers Demo
67
Info Triggers Demo
68
Info Triggers Demo
69
Info Triggers Demo
70
Info Triggers Demo
71
Info Triggers Demo
72
Info Triggers Demo