Title: User Interfaces for Post-Desktop Environments
1User Interfaces for Post-Desktop Environments
- Jan BorchersStanford Interactivity
Labhttp//www.stanford.edu/borchers/ - Advanced Systems Development LabTalk Series,
PARC, 6/3/2002
2Outline
- The Stanford iRoom A Ubicomp Environment
- User Interface Challenges for Ubicomp
environments - iStuff Supporting Research in Ubicomp UIs
3The Shift to Ubiquitous Computing
- Desktop-centric
- Generic machines
- Client-server
- Monolithic apps
- One person, one application, one set of hardware
- Invisible/implicit interaction
- Dedicated appliances
- Universal glue-like connectivity
- Infrastructure-centric services
- Multi-person, multi-application, multi-device
integration
4What Is an Interactive Room?
- An augmented dedicated space
- e.g. conference room
- can be used with teleconnected spaces
- Task-oriented scenarios
- e.g. brainstorming meeting, design review
- Multi-device, multi-user, multi-application
- Easy to (re)configure
- Movement of information across multiple devices,
and from personal to/from multi-user devices - Integration augmentation of legacy apps
5The Stanford iRoom
6The Stanford iRoom
7Whats in the iRoom Today
- 3 video cameras
- Projector Control System
- Speakers for each Smart Board
- X-10 Lighting Control
- iPaq PDA
- Printer, scanner
- Anoto pen, Wacom tablets
- Other custom hardware
- Behind the Scenes
- 5 Windows 2000 Boxes
- 5 Windows NT 4.0 Boxes
- 2 Windows 98 Boxes
- 32 processor Linux Cluster
- 1 Mac OS X Box
- Lightning 2 Digital Video Switch
- IEEE 802.11 Wireless LAN
- 1000s of feet of cable!
- more as time goes on...
- Accessible Infrastructure
- 3 4x5 Touch Sensitive Smart Boards
- Back Projected 4x6 Interactive Mural with 12
tiled projectors (9Mpix) - Interactive Table with 3x4 Screen and Pen Input
- Control Panel Machine with Wacom LCD Tablet
Screen - PointRight Mouse and Keyboard Redirection System
- Laptop video drop switched to all displays
- Cluster Control Machine with display to Smart
Boards
8iRoom Video
- http//mediastream.stanford.edu8080/video/iwork_s
cenario/iwork_scenario_dv.ram
9Application Domains
3-D Medical Imaging (Med School/Radiology)
Design Brainstorming (Center for Design Resarch,
IDEO)
Construction project management (CivEng Center
for Integrated Facilities Engineering)
10Ubicomp UI Challenges
- Multiple users, multiple applications, multiple
displays - No simple 2-D "desktop"
- Window? Focus? Pointing? Selection?
- Post-Desktop equivalent of mouse keyboard?
11Some iRoom Technology Solutions
- Mural cluster High-resolution interactive wall
("Chromium") - Event Heap "Event queue for the room"
- MultiBrowse Sending URLs around the room
- Data Heap Storing ( converting) information
in the room - PointRight One mouse keyboard for the room
- Room Controller Web pad to control room lights,
projectors, .... - Interface Crafter Generating interfaces to room
services - Barehands Gesture control for Smart Boards
12iStuff
- Prototyping desktop GUIs has become easy
- VisualBasic, Interface Builder, ...
- Lesson learned Rapid ubicomp UI prototyping is
hard - Example "Add a feedback light to a tablet-based
UI" - -gt Run wires, solder adapters, write serial
drivers,... - Idea A toolkit for building ubicomp UIs?
- Simplify experimentation with ubicomp UI
scenarios designs - Basic input/output "phidgets" Greenberg01
- But Wireless, design for augmented environment
(iRoom)
13iStuff We've Built
- Input devices
- iButton
- iSlider
- Output devices
- iLight
- iBuzzer
- iSpeaker
- Others in the works (sensors, MIDI controllers,
microphones, servos, ...)
14Taking Advantage of Infrastructure
iSpeaker
iSlider
Radio Transmit
Radio Receiver
Application
PC daemon
PC proxy
eheap.jar
eheap.jar
eheap.jar
Event Heap
- Any Event Heap based application can talk to
iStuff - iStuff device wireless device PC proxy/daemon
- Lightweight, cheap, low-power, ... devices
15Event Heap app to speak text
import iwork.eheap2. class speaktext
// Connects to
event heap in static void main(String args)
// arg0, and sends
AudioEvent
// with the text in arg1.
try EventHeap theHeapnew
EventHeap(args0) // Connect to the Event
Heap Event myEventnew
Event("AudioEvent") // Create an event
myEvent.setPostValue("AudioCommand",
"Read") // Set its fields
myEvent.setPostValue("Text", args1)
theHeap.putEvent(myEvent) //
Put event into Event Heap
catch(Exception e)
e.printStackTrace()
gtspeaktext iw-room2.stanford.edu "Hello World"
16Initial Evaluation
- Successful sample uses
- Start The Room
- iPen
- iPong
- Findings and next steps
- Allows rapid integration of ubicomp UI components
into apps - Spend time designing the software and interface,
not soldering - Advantage of collaboration with Product Design
ME students - Abstraction layers device - normalized -
application specific - Device Classes and IDs, PatchPanel
- Applications need to be written differently
17Great Unified Input Theory
- iStuff wireless hardware layer can be RF, X-10,
Bluetooth, 802.11b, Infrared, Cellular,... - Offer iStuff as abstraction layer of room devices
above this level - Adopters can
- Use our hardware and software designs and
replicate devices - Use our software and use off-the-shelf components
as iStuff - Implement the "plugins" for new hardware
technologies - Make applications listen only to Event Heap Events
18Summary
- The Stanford iRoom
- Ubicomp requires re-thinking user interface
metaphors and architecture - iStuff as a tool to facilitate experimentation in
ubicomp interfaces - Not the answer to "post-desktop mouse
keyboard", but may help finding it... - http//iwork.stanford.edu/
- http//www.stanford.edu/borchers/iStuff/
19Bonus Material
20PointRight
- A system for pointer/keyboard redirection among
multiple displays and machines (Brad Johanson,
Greg Hutchins)
video
21BareHands
- Gesture-based interaction with a wall-mounted
display (Merrie Ringel)