An Overview of ContextAware Computing

1
An Overview of Context-Aware Computing
Jason I. Hong Group for User Interface
Research University of California at Berkeley
2
Motivation

• Modern computers are divorced from reality
• Unaware of who, where, and what around them
• Leads to mismatch
• Computers have extremely limited input
• Aware of explicit input only
• Can take a lot of effort to do simple things
• Context-Aware Computing
• Making computers more aware of the physical and
  social worlds we live in
• Breaking computers out of the box

3
Why Context-Aware Computing?
Context Types
Existing Examples
Human Concern
4
Technology Trends

• Sensors
• GPS, Active Badges, Active Bats
• Smart Dust
• Cameras and microphones
• Recognition algorithms
• MSR Radar location from 802.11
• Smart Floor footstep force
• Wireless technologies
• Bluetooth, 802.11, cell phones

5
Why Context-Aware Computing?
Existing Examples
Context Types
Potential Examples
Human Concern
Activity
Convenience
Identity Time Location Proximity
Activity History
Auto Lights On / Off
Auto Cell Phone Off In Meetings
Activity
Finding Info
File Systems
Tag Photos
Identity
Memory
Calendar Reminders
Proximal Reminders
Identity Time
Safety
Smoke Alarm
Health Alert
Time
Efficiency
Barcode Scanners
Service Fleet Dispatching
6
Outline

• Background and Motivation
• Sample Context-Aware Apps
• Building Context-Aware Apps
• Open Research Challenges
• Summary

7
Sample Context-Aware Apps
1
2
3
4
8
Sample Context-Aware AppsActive Badges

• Badges emit infrared signals
• Gives rough location ID
• Teleport
• Redirect screen output from "home" computer to
  nearby computer
• Phone forwarding
• Automatically forward phone calls to nearest phone

Active Badge Olivetti / ATT Hopper, Harter, et
al
9
Sample Context-Aware Apps ParcTabs

• Active badge wireless
• Rough location ID
• Proximate selection
• Interfaces for nearby objects
• Auto-diaries
• People, places, and time
• Triggers
• Alerts on preset events
• Reconfiguration
• Bind device to room

ParcTabs Xerox PARC Want, Schilit, et al
10
Sample Context-Aware Apps ParcTabs

• Active badge wireless
• Rough location ID
• Proximate selection
• Interfaces for nearby objects
• Auto-diaries
• People, places, and time
• Triggers
• Alerts on preset events
• Reconfiguration
• Bind device to room

ParcTabs Xerox PARC Want, Schilit, et al
11
Sample Context-Aware Apps ParcTabs

• Active badge wireless
• Rough location ID
• Proximate selection
• Interfaces for nearby objects
• Auto-diaries
• People, places, and time
• Triggers
• Alerts on preset events
• Reconfiguration
• Bind device to room

ParcTabs Xerox PARC Want, Schilit, et al
12
Sample Context-Aware Apps ParcTabs

• Active badge wireless
• Rough location ID
• Proximate selection
• Interfaces for nearby objects
• Auto-diaries
• People, places, and time
• Triggers
• Alerts on preset events
• Reconfiguration
• Bind device to room

ParcTabs Xerox PARC Want, Schilit, et al
13
(No Transcript)
14
Sample Context-Aware Apps ParcTabs

• Active badge wireless
• Rough location ID
• Proximate selection
• Interfaces for nearby objects
• Auto-diaries
• People, places, and time
• Triggers
• Alerts on preset events
• Reconfiguration
• Bind device to room

ParcTabs Xerox PARC Want, Schilit, et al
15
(No Transcript)
16
Sample Context-Aware Apps ParcTabs

• Active badge wireless
• Rough location ID
• Proximate selection
• Interfaces for nearby objects
• Auto-diaries
• People, places, and time
• Triggers
• Alerts on preset events
• Reconfiguration
• Bind device to room

ParcTabs Xerox PARC Want, Schilit, et al
17
Sample Context-Aware Apps Tour Guides

• GPS or infrared tracking
• Fairly precise location
• Display location on screen
• Predefined points of interest
• Automatically pop up if nearby
• Travel journal
• Keep log of places seen and photographs taken

Cyberguide Georgia Tech Abowd et al
18
Sample Context-Aware Apps Tour Guides
Cyberguide Georgia Tech Abowd et al
19
Sample Context-Aware Apps Tour Guides
Cyberguide Georgia Tech Abowd et al
20
Sample Context-Aware Apps Enhanced PDAs
Microsoft Research Hinckley et al
21
Sample Context-Aware Apps Enhanced PDAs

• Voice memo recording
• Hold like phone near mouth to start recording
• Portrait / Landscape mode
• Just physically rotate screen
• Tilt scrolling
• Tilt instead of scrollbars
• Power management
• Turn on if being held and tilted

Microsoft Research Hinckley et al
22
Some Issues in Context-Aware Computing

• Sensor ambiguity
• Sensors not 100 reliable
• Precision / Accuracy / Granularity
• Sensor Fusion
• Merging different sensor inputs together
• Self-contained vs. distributed systems
• PDA doesn't need location sensors if it can ask
  nearby sensors to approximate
• Requires lots of knowledge and effort to build
• Sensors, recognition algorithms, devices,
  application
• Few kinds of context beyond location ID used
• There haven't been many rigorous evaluations of
  utility

23
Some Issues in Context-Aware Computing

• Sensor ambiguity
• Sensors not 100 reliable
• Precision / Accuracy / Granularity
• Sensor Fusion
• Merging different sensor inputs together
• Self-contained vs. distributed systems
• PDA doesn't need location sensors if it can ask
  nearby sensors to approximate
• Requires lots of knowledge and effort to build
• Sensors, recognition algorithms, devices,
  application
• Few kinds of context beyond location ID used
• There haven't been many rigorous evaluations of
  utility

24
Outline

• Background and Motivation
• Sample Context-Aware Apps
• Building Context-Aware Apps
• Open Research Challenges
• Summary

25
Building Context-Aware Apps

• Describe support at app-level
• ParcTab System
• Context Toolkit
• MUSE
• Cooltown
• Mobisaic
• Stick-E Notes
• Context Fabric
• Try to make it easier to build a certain class of
  context-aware apps

26
A Rough Taxonomy of Context-Aware Apps

• Triggers
• Metadata Tagging
• Reconfiguration and Streamlining
• Input specification
• Presentation

27
A Rough Taxonomy of Context-Aware Apps

• Triggers
• On X do Y
• "Notify doctor and nearby ambulances if serious
  health problem detected"
• "Remind me to talk to Chris about user studies
  next time I see him"
• Metadata Tagging
• "Where was this picture taken?"
• "Find all notes taken while Mae was talking"
• Memory prosthesis

28
A Rough Taxonomy of Context-Aware Apps

• Reconfiguration and Streamlining
• Telephone forwarding and Teleport
• Turn off cell phone in theaters
• Automatically adjust brightness / volume
• Automatic file pre-caching
• Select modes in multimodal interaction
• Multimedia / Bandwidth adaptation

29
A Rough Taxonomy of Context-Aware Apps

• Input specification
• Send mail only to people in building now
• Print to nearest printer
• "Find gas stations nearest me"
• Presentation
• Current location
• Idle?
• Currently in?
• Contextual info about objects
• Proximate selection

30
ParcTab System(Schilit 1995)

• Three key abstractions
• Device Agents, User Agents, Active Maps

31
ParcTab System(Schilit 1995)

• Queries
• Check context state in Agents or Active Maps
• Room LID35-2-
• Printer model lw
• Events
• Standing queries
• Notified on changes in context state

32
Context Toolkit(Dey, Salber, Abowd 2001)

• Toolkit for distributed context-aware apps
• Framework for acquiring and handling context
• Standard components
• Three key abstractions
• Widgets, Interpreters, and Aggregators

33
Context Toolkit(Dey, Salber, Abowd 2001)

• Widgets abstract out sensors

App
App
Location Widget
GPS
Active Badge
Cell Phone Location
Active Badge
34
Context Toolkit(Dey, Salber, Abowd 2001)

• Interpreters transform context data

App
Location to Street Interpreter
Location to Room Interpreter
Location Widget
35
Context Toolkit(Dey, Salber, Abowd 2001)

• Aggregators group related widgets together

App
Location to Room Interpreter
App
Person Aggregator
Activity Widget
Affect Widget
Location Widget
36
Context Toolkit(Dey, Salber, Abowd 2001)
In / Out Board
37
Context Toolkit(Dey, Salber, Abowd 2001)
Dummbo
38
MUSE(Castro and Muntz 2000)

• Sensor management issues
• Sensor discovery (currently via Sun's Jini)
• Specify performance goals and resource
  constraints
• "Find a printer but minimize power consumption"
• Probabilistically modeling sensor data
• Can measure service cost
• Sensor fusion (via Bayesian nets and HMMs)

39
CoolTown(Hewlett-Packard 2000)

• Since web is universal, link context awareness to
  web
• Literally everything has a URL
• People, places, things
• Infrared beacons, bar codes, etc
• Literally everything has a web page
• Current status, contact info, services offered,
  etc

40
CoolTown(Hewlett-Packard 2000)

• SF Exploratorium
• Get extra info
• Create bookmarks for later use

41
CoolTown(Hewlett-Packard 2000)

• Buses equipped with GPS and net
• In bus
• Show location
• Show nearby points of interest
• Waiting for bus
• Show location
• Show wait time

42
Mobisaic(Voelker and Bershad 1994)

• Context-awareness web page authoring
• Dynamic URLs
• Variables processed by modified web browser
• http//www.offices/(Location).html
• http//www.offices/525.html
• Active Documents
• Extensions to HTML
• lt!- subscribe to (Location), reload -gt

43
Stick-E Notes(Pascoe 1997)

• Authoring tool for using context

ltnotegt ltrequiredgt ltatgt (1,4) .. (3,5) ltfacinggt
150 .. 210 ltduringgt December ltbodygt The large
floodlit building at the bottom of the hill is
the cathedral.
44
Context Fabric(Hong and Landay 2001)

• Distributed sensors, services, and devices
• Won't be designed a priori for interoperability
• Shift burden of context-awareness from individual
  devices onto infrastructure
• Provide basic services usable by any device
• Provide a high-level way of specifying context
  needs
• Dynamically coordinate sensors and services

45
Context Fabric(Hong and Landay 2001)

• Push burden of context-awareness from individual
  devices onto network
• Share computational power, data, sensors
• Support incremental evolution
• Support richer set of devices
• Provide core set of context services
• Basic services that exist everywhere
• Uniform level of abstraction hiding nasty details

46
Automatic Path Creation

• A way of assembling small services into larger,
  more interesting ones
• Operators
• Paths

47
Automatic Path Creation

• Suppose we had a way to specify context needs
• Suppose we had a way to specify service semantics
• Automatically assemble paths based on resources
• Don't have to know specific sensors and services

48
Context Fabric Architecture
App
Application Layer
Context Specification Language
Services
Context Event Service
Context Query Service
Context Layer
Automatic Path Creation
Logical Sensor Layer
Sensor Management Service
Physical Sensor Layer
49
Outline

• Background and Motivation
• Sample Context-Aware Apps
• Building Context-Aware Apps
• Open Research Challenges
• Summary

50
Open Research ChallengesSystems Issues

• Programming model
• Programming the physical world
• Unreliable sensors, recognition algorithms, plus
  standard distributed computing issues
• Interoperability
• Sensors, services, and devices
• Useless if everyone has proprietary / custom
  systems
• Need standard data formats, protocols, and
  frameworks
• Varying capabilities of sensors, services, and
  devices

51
Open Research ChallengesSystems Issues

• Need clearer definitions of context
• What is and isn't context?
• Temperature? The monitor I'm looking at? Personal
  history?
• Have to avoid the AI tarpit (ie does it matter?)

52
Open Research ChallengesPeople Issues

• Avoiding embarrassing situations
• Active Badges bathrooms
• Inconvenient phone forwarding
• Avoiding dangerous situations
• Need to take into consideration cost of mistake
• Smoke alarms when cooking
• Lights that turn off when you're still there
• Woman locked in "smart toilet stall"
• Will adding more context really help here?

53
Open Research ChallengesPeople Issues

• Making it predictable and understandable
• Setting preferences
• "I want my cell phone to ring except in theaters
  and when I'm in a meeting unless"
• Why the heck did it do that?
• Privacy
• What does the computer know about me?
• What do others know about me?
• What do I gain? What do I lose?

54
Take Home Ideas

• Strides in sensors, recognition, and wireless are
  enabling new context apps
• Still need to lower barriers to entry
• Lots of systems issues, even more people issues
• Lots of potential here for new kinds of
  interaction and applications

55
The End

• Group for User Interface Research
• University of California at Berkeley
• Context Fabric Link
• http//guir.berkeley.edu/cfabric

56
Backup Slides
57
Proposed Solution
App
Agents
Application Layer
Device Agents
Context Specification Language
User Agents
Services
Context Event Service
Context Query Service
Context Layer
Automatic Path Creation
Logical Sensor Layer
Sensor Management Service
Physical Sensor Layer
58
Context Specification Language

• Need to express context about and relationships
  between People, Places, Things
• Predicates
• Identity (Who is? What is? Is with?)
• Location (Near? Nearest? Distance? Path?)
• Activity (Is busy? Is in meeting? Current task?)
• Time (In past? In present? In future? On date?)
• Some of this vocabulary done by Schilit
• Implicitly encoded in his APIs
• One goal is to extend his work in spec language
• Another is to make it extensible for future
  context types

59
Context Specification Language

• Common parameters
• Max number of results wanted
• Return name
• Return data type (e.g. String, List, Table)
• Minimal probability of correctness desired
• Relevant sensor input requestor has
• Event parameters
• Event rate (e.g. at most 1 event per second)
• Event callback (e.g. RPC, socket port)
• Max number of events desired
• Granularity of change (e.g. 1 meter)

60
What is Context?

• Situated Action (Suchman)
• Responsiveness, improvisation to situation
• Actions are moment-by-moment, highly fluid on
  situation
• Activity Theory (Nardi)
• Subject, Objective, Operations, Artifacts, Env
• Activity defines context but it changes as list
  above changes
• Context depends on sensing and inferring, but
  difficult to know people or goals

61
What is Context?

• Provides fair criticisms on existing
  context-aware apps
• Response Focus on the more routine and
  predictable actions
• Response Focus on simple first-order approx
• Examples Smoke alarms, Calendar reminders
• Response Is occasionally wrong, but won't get
  better unless we do more research here
• Response Infrastructure enables more research in
  this area

62
What is Context?

• Merging of virtual with physical and social
• Mostly implicit input, often secondary to
  task at hand
• People / Places / Things / Virtual / Time
• People Location, Identity, Task, Affect
• Places Activity, People, Temperature, Audio
• Things Location, Identity
• Virtual Services, Bandwidth
• Time All of these Past, Present, and Future

63
What is Context?

• But it's a rathole to try to define it
• Workflows? (Greenberg 2001)
• Intent and Desire?
• Natural Language Processing?
• What isn't context?
• Instead, operationalize it
• Distributed, multi-device, sensor-based
• Fairly well-defined and computable concepts (e.g.
  location and identity, but not intent or workflow)

64
Why Context-Awareness?
Skeptics toward the ubiquitous computing movement
quickly point out that those involved in
promoting Mark Weiser's initial vision are mainly
interested in pushing technology As a reaction
against this fair criticism, many researchers
spend countless hours trying to dream up the
"killer application" that will cause everyone to
adopt ubiquitous computing technology. I think
this is the wrong goal to have in mind as a
researcher for two reasons. First, we are not
schooled in the practices to judge a market
Second, my interpretation of Weiser's vision is
not that there would be a single, most compelling
use of ubicomp technology, but rather that the
totality of the experience would cause a
significant paradigm shift in the way we as
humans perceive our relationship to interactive
computing. In short, Weiser argued for what I
would like to call a "killer existence," in which
many disparate devices would provide an array of
services in ways that seamlessly complement our
lifestyles. -- Gregory Abowd, 2001
65
Why Infrastructure Approach?

• Sharing of data
• Collaborative filtering, large data sets
• Sharing of processing power
• Heavy-duty processing on dedicated machines
• Sharing of sensors
• Sensors in the environment
• In-place evolution of services
• New algorithms, new features
• Supports heterogeneity
• Sensors, Devices, OS, Programming Langs

66
Why Infrastructure Approach?

• Building and evaluating now can
• Provide more design space for interaction
• Let us rapidly prototype and evaluate these apps
• Find problems now
• Develop the right mechanisms
• Evolve the right policies

67
Some Sensors

• Temperature
• Pressure
• Humidity
• Soil makeup
• Acceleration
• Noise
• Material stress

• Biometric
• Motion
• Distance
• Touch
• Location
• Orientation
• Light

68
Context-Aware Mechanisms

• Events
• Something interesting has just happened
• e.g. "Larry has entered room 306"
• Queries
• Introspect current context state (discrete)
• e.g. "Where is Larry now?"
• Filters
• Modify events and queries based on preferences
  and privacy
• e.g. "Only show others if in office or not"

69
Are Queries and Events Enough?
Query
Event
Triggers
On Event do Action
Metadata Tagging
Query for relevant context state
Reconfiguration
Query current context state before and while
running
On Event do Reconfigure
Input Specification
Query current context state to fill in the blanks
Presentation
Query current state for more information
On Event do Update Presentation