CS376 Introduction

1
Authoring Sensor-Based Interactions by
Demonstration with Direct Manipulation and
Pattern Recognition
Björn Hartmann, Leith Abdulla Manas Mittal, Scott
R. Klemmer
2
How would you prototype
a workout monitoring system?
Apple, Nike
3
How would you explore
motion-based game controllers?
Nintendo
4
How would you build
near
middle
far
out of range
a proximity-aware whiteboard?
Ju, Lee, Akers
5

• Sensor-based Interaction Design

6
d.tools
Hartmann et al., UIST 06
7
d.Tools Student Project
d.tools
8
Designing Sensor-based Interactions
PROTOTYPE APPLICATION LOGIC
SPECIFY RELATIONSHIP BETWEEN SENSOR DATA AND
APPLICATION LOGIC
SPECIFY RELATIONSHIP BETWEEN SENSOR DATA AND
APPLICATION LOGIC
PROVIDE SOFTWARE ABSTRACTION FOR HARDWARE
9
Types of Sensors
Force Sensitive Resistors
Accelerometers
RFID
IR/Ultrasonic Rangers
Phototransistors
Continuous
Categorical
Discrete
10
Current Tools Oscilloscopes
CC image by Timo Arnall
11
Current Tools LabView
LabView
12
Representation Matters
Accelerometer X axis
Accelerometer Y axis
//detect accelerometer peaks //read data
sample xValtreadA2DValue(xPin) //look for
changes in derivative if(((xValt-xValt-1) gt
0 (xValt-1-xValt-2) lt 0)
(((xValt-xValt-1) lt 0 (xValt-1-xValt-2
) gt 0)) //peak detected //send message
oscSendMessageInt("/x/peak",1) else //no
peak
13
Idea Programming by Demonstration
14
Idea Programming by Demonstration
15
Tacit Knowledge
16
Crux Generalization
17
Crux Generalization
18
Crux Generalization
19
Exemplar
20
Exemplar
21
Exemplar
22
Exemplar
23
(No Transcript)
24
Exemplar UI
25
Exemplar UI
26
Exemplar UI
27
Exemplar UI
28
Exemplar UI
29
Exemplar UI
30
Exemplar UI
31
(No Transcript)
32
Dynamic Time Warping
Demonstration Signal
Matching Input Signal
Sakoe, H. Chiba, S. 78
33
Filters
34
Extensible Filter Set
35
Understanding the Exemplar Approach
What?
Why?

• Cognitive Dimensions of Notation (CDN) Inspection
• First-Use Laboratory Study
• Class Deployment
• Used as Design Tools for Building Upon Everyday
  PlayCHI Interactivity Exhibit

Analysis of Exemplar as a visual authoring
environment Threshold and usability Real-world
stress test Complexity ceiling for knowledgeable
users
36
Lab Study
sketches by Wendy Ju
37
Lab Study
26 min
27 min
18 min
47 min
22 min
31 min
sketches by Wendy Ju
38
(No Transcript)
39
Mini Joystick
Bend Sensor taped to table
Navigation with Accelerometer
Navigation with XY Joystick
Test Play
40
Mini Joystick
Bend Sensor taped to table
Test Play
Trigger with Bend sensor
Refine trigger detection
Test Play
41
Post-Test Questionnaire
42
Post-Test Questionnaire
43
CHI Interactivity
Control Freaks by Haiyan Zhang
44
CHI Interactivity
Compulsory latency of pattern matching
precludes use for fast-paced actions
Thresholdcrossed here
Pattern matched here
time
Control Freaks by Haiyan Zhang
45
Related Work
PBD Tools for Ubicomp
Musical controller Design tools
Commercial Dsp software
a Capella Dey et al., CHI 04 CrayonsFails
Olsen, CHI 03 MonetLi, Landay, UIST
05 Papier-MâchéKlemmer et al., CHI 04
HID ToolkitSteiner, NIME 05 MnMBevilacqua et
al., NIME 05 FlexiGestureMerrill Paradiso,
CHI 05
LabView/Lego Mindstorms NXT
46
Related Work
A CAPella (Dey et al., CHI 04)
47
Related Work
FlexiGesture (Merrill CHI05)
48
Related Work
LabView
49
Related Work
Lego Mindstorms NXT
50
Future Directions

• Export Generate code specifications
• Programming sensor networks by demonstration
• d.mix Programming by A Sample for Web Service
  APIs
• Integration of direct manipulation environments
  with textual programming

51
Acknowledgments

• We thank MediaX/DNP for funding, Intel for
  equipment donation, Wendy Ju for
  illustrations,David Merrill Timo Arnall for
  photos.

52
Download Exemplar http//hci.stanford.edu/exempla
r
CHI Interactivity Building Upon Everyday Play
Talk tomorrow, 1130am, Room C2
53
Dynamic Time Warping
Example
Input Signal
Sakoe, H. Chiba, S. 78
54
Student Projects
d.Tools board
55
Student Projects
56
Student Projects
57
Cognitive Dimensions of Notation

• Examples
• Visibility and Juxtaposability
• Viscosity
• Hard Mental Operations

Blackwell, Green
58
Verplank
59
Evaluation

• Cognitive Dimensions of Notation (CDN) Inspection
• First-Use Laboratory Study (n12)
• Class Deployment
• Used as Design Tools for Extending Everyday
  PlayCHI Interactivity Exhibit

60
Exemplar d.tools