OnBody Sensing

1
On-Body Sensing

• Daniel Ashbrook
• Tracy Westeyn

2
Introduction

• Who we are why we are qualified

3
Sensors are great!

• input from world rather than input from human

Good!
Bad!
(This guy is too busy - leave him alone!)
4
Sensors, sensors, everywhere

• sensors are ubiquitous

5
Making things easier

• Not so many circuits to build

aa162171c7202 aa1721f1bf200 aa1821a1c11fd aa1921f1
bd1fc aa1a2191c3201 aa1b21d1bb1fd aa1c2191c0200 aa
1d21b1bb1fb aa1e21b1bd1fd aa1f21d1bb1ff aa202191c5
201 aa2121c1bd200
Before
6
Making things easier

• Not so many circuits to build

After
7
Making things easier

• Pre-built sensors
• sparkfun.com
• Accelerometers, Camera / Imaging, Capacitive, IMU
  / Gyros, IR / Ultrasonic, Magneto, Temperature

8
Making things easier

• Pre-built sensors
• sparkfun.com
• Accelerometers, Camera / Imaging, Capacitive, IMU
  / Gyros, IR / Ultrasonic, Magneto, Temperature
• xsens.com
• 3-DOF inertial sensors

9
Making things easier

• Pre-built sensors
• sparkfun.com
• Accelerometers, Camera / Imaging, Capacitive, IMU
  / Gyros, IR / Ultrasonic, Magneto, Temperature
• xsens.com
• 3-DOF inertial sensors
• qprox.com
• Capacitive sensor chips evaluation boards

Touch-based slider
Touch buttons
10
Making things easier

• Pre-built sensors
• sparkfun.com
• Accelerometers, Camera / Imaging, Capacitive, IMU
  / Gyros, IR / Ultrasonic, Magneto, Temperature
• xsens.com
• 3-DOF inertial sensors
• qprox.com
• Capacitive sensor chips evaluation boards
• xbow.com
• Wireless sensor networks (Acceleration, Magnetic,
  Light, Temperature, Acoustic)

Mote professional kit
11
Making things easier

• Pre-built wireless solutions
• Bluetooth serial modules (btdesigner.com)
• ZigBee (IEEE 802.15.4) (maxstream.net)
• Nordic radio (nvlsi.no)

12
Making things easier

• Mounting

Polar heart rate monitor strap
Bodymedia BodyBugg
13
Making things easier

• Abstraction

No!
Good!
Yes!
Hard!
14
Why on-body sensing?

• Often easier to sense personal contextual data
• e.g., activity, location
• Unique data available
• heart rate, GSR
• Less infrastructure needed
• pressure-sensitive floor vs. accelerometer for
  activity level
• Privacy benefits
• control access to your own data

15
Types of sensors

• altimeter
• compass
• GPS (Global Positioning System)
• capacitive (touch)
• pressure
• temperature

• accelerometers
• GSR (Galvanic Skin Reponse)
• gyroscopes
• bend flex
• light
• cameras
• sound (mics)

16
Thats a lot of types what next?

• What the sensor measures
• What are some of the errors
• When to use / not use

17
Accelerometers

• Acceleration dynamic characteristic of object
  requiring application of a force (a F/m)
• 2nd derivative of position
• 1st derivative of velocity
• Taking derivatives in high frequency noisy
  signals result in large errors
• Sensing method is dependent on application
  frequency

position
velocity
acceleration
1Hz
lt 1kHz
gt1Hz
18
Characteristics of an accelerometer

• single-degree-of-freedom
• Vibration periodic oscillatory motion around a
  reference position.
• seismic mass
• a spring-like supporting system
• frame structure w/ damping properties

19
How an accelerometer works

• measure displacement of an object with respect to
  some reference object
• displacement transducer generates an electrical
  signal as a function of acceleration.

20
Accelerometers everywhere!
any displacement transducer capable of measuring
microscopic movements under strong vibrations or
linear acceleration
21
Accelerometers anyone?Things to consider

• Anticipated magnitude of acceleration
• How fast the ambient temperature changes
• Anticipated frequency range
• Accuracy range required
• Maximum tolerable size vs power
• Amount of moisture present
• Anticipated over-shock
• Surrounding acoustic, electromagnetic, and
  electrostatic interference

22
GSR sensor

• Galvanic skin response change in skins
  conductivity resulting from change in arousal
  levels
• Fluctuates quickly during mental, physical and
  emotional arousal
• Eccrine glands produce ionic sweat lowering
  resistance and increasing conductivity

23
Characteristics of a GSR sensor

• Two electrodes in close proximity
• Typically attached to areas of high eccrine gland
  concentration
• palms of hands
• soles of feet
• arm pit
• simple DC amplifier

24
How the simple GSR works

• Measures changes in electrical resistance across
  two regions of the skin
• Low voltage applied to one electrode
• Current flows to second electrode
• transistor amplifies signal
• potentiometer can be used to adjust for
  individual conductivity differences

25
When can I use GSR sensors
26
GSR anyone?Things to consider

• Placement of electrodes
• Relative position of electrodes
• Pressure applied to electrodes
• Sensitivity of reading
• Duration of data collection (hours vs days)
• Activity level during data collection

27
Gyroscopes (Gyros)

• Orientation sensors
• 1 or 2 degrees-of-freedom possible

• Traditional gyros
• massive disk
• framework rotates about 1 or 2 axes

28
How the Traditional Gyro Works

• Based on principle of Conservation of Angular
  Momentum
• Spin axis will remain fixed with respect to
  space (assuming no external forces act on it)
• Delivers a torque proportional to the angular
  velocity about an axis perpendicular to the spin
  axis

29
MEMS Gyros

• rotation is replaced by vibration
• suspended mass moves linearly in simple harmonic
  motion.

30
Gyros anyone?Things to consider

• Size and cost
• Durability of device

31
Applied Force Sensors (strain gauges)

• Measuring the strain produced in a elastic member
  by an external force
• piezoresistive effect
• Long longitudinal and short transverse segments
  provide good sensitivity

32
Characteristics of Strain Gagues

• Resistor bonded with an elastic carrier
• Elastic carrier (backing), is applied to object
• Must be reliably coupled to the gauge wire
• Wire electrically isolated from the object
• Coefficient of thermal expansion of the backing
  should be matched to that of the wire

33
Strain Gauges Anyone?Things to consider

• semiconductive strain gauges are quite sensitive
  to temperature variations
• Sensor should be well bonded to device
• withstand force strain (dont snap in half)

34
Light Sensors

• Detect electromagnetic radiation in spectral
  range (ultraviolet to far infrared)
• Absorption of photons (2 response types)
• quantum detectors (ultravi to mid-infrared)
• thermal detectors (mid and far-infrared)

35
How Light Sensors Work

• photoelectric effect
• photon transfers energy to surface electron
• high energy, the electron may become mobile
• results in an electric current

36
Light Sensors Anyone?Things to Consider

• Temperature of operations (thermal gtefficiency
  quantum at room temp)
• amount of light vs intrinsic noise level of
  detector
• surface area of detector lens
• sensitivity / response range of detector

37
Cameras

• lens focuses light behind rear element
• film
• silicon chip
• CCD charged coupled device
• CMOS complimentary medal oxide sensor

38
How Camera CCDs work

• grid of photosites
• typically monochrome
• color filters laid over photosites
• RGB
• CMYK

39
Cameras Anyone?Things to Consider

• ambient light conditions
• camera settings
• type of light to sense
• resolution

40
Microphones

• detects audible range acoustics
• pressure transducer
• adapted for the transduction of sound waves over
  a broad spectral range
• generally excludes very low frequencies (below a
  few hertz)
• cannot measure constant or very slow-changing
  pressures

41
How Microphones Work

• moving diaphragm
• displacement transducer converts diaphragms
  deflections into electrical signal

42
Microphones Anyone?Things to Consider

• sensitivity
• directional characteristics
• frequency bandwidth (acoustic sensors)
• dynamic range
• size of device
• sound transmitting media (liquid, gas, or
  solids)

43
Altimeter

• indirectly measures altitude
• barometer (measuring atmospheric pressure)
• uses changes in atmospheric pressure to determine
  altitude
• inverse relation (high altitude, low pressure)
• airplanes use laser range finders

44
How an Altimeter Works

• changes its resistance in proportion with the
  height of mercury in each column

45
Altimeters Anyone?Things to Consider

• calibration for altitude of use
• weather conditions
• air pressure changes by 1 mbar during day
• skew readings by 26 feet (8 meters)
• bad weather low pressure

46
GPS

• Geo Positioning System measures position
• Receives radio signals from multiple satellites
• Computes time delay between signals received from
  one satellite to the others
• When position of a vehicle is determined with a
  periodic rate, computation of velocity is no
  problem.
• For smaller objects and shorter distances, GPS is
  not a velocity solution.

47
GPS Anyone?Things to Consider

• capture time (signal delay)
• signal indoors is typically poor
• city canyon effect
• precision (2 meter accuracy)

48
Tactile Sensors

• special force or pressure transducers
• two basic types
• two leaves of foil and a spacer
• thin layer material responsive to strain

49
How Tactile Sensors Work

• Active
• active ultrasonic coupling touch sensor
• piezoelectric films (polyvinylidene fluoride
  --PVDF)
• Passive
• signal generated by piezoelectric film without
  excitation signal
• response proportional to the rate of stress

50
Tactile Sensors Anyone?Things to Consider

• Type of senor (switch, capacitive)
• Contaminates present
• moisture
• dust
• Sensitivity

51
Pressure Sensors

• measures force applied to a surface
• detector responding to applied force
• 3 types
• absolute (barometer)
• differential (flow meters)
• gauge (blood pressure)

52
How Piezoresistive Pressure Sensors Work

• thin silicon diaphragm (elastic material)
• piezoresistive gauge resistors as diffusive
  impurities into the diaphragm

53
Pressure Sensors Anyone?Things to Consider

• Type of sensor
• Range of operation temperatures

54
Temperature sensors

• thermal expansion (liquid in glass)
• electrical transduction
• resistive
• optical
• acoustic
• piezoelectric
• thermoelectric
• semiconductive

55
How Temperature Sensors Work

• heat conduction occurs at interface between the
  object and the probe
• sensor converts thermal energy into an
  electrical signal

56
Characteristics of a Temperature Sensor

• equilibrium
• no significant thermal gradient exists between
  measured surface and the sensing element (thermal
  equilibrium)
• predictive
• equilibrium point determined by rate of the
  sensors temperature change

57
On-body challenges

• Hardware
• cables
• batteries
• wireless
• power
• placement orientation
• recordkeeping

• Software
• data collection
• synchronization
• feedback (heartbeat)
• moving data around
• processing data

58
Hardware
59
Hardware
60
Sensor placement

• What kind of data is needed?
• Sensor limitations
• Later processing
• Mounting

61
Sensor placement

• What kind of data is needed?
• Or, whats the application?
• Examples with accelerometer
• To get steps/minute, place on foot
• To track activity, place on hip

62
Sensor placement

• Sensor limitations
• Accelerometer 2G, 5G, 10G?
• Proximity sensor ambient light influence?
• Capacitive touch user wearing gloves?
• Dynamic range mic recording right frequencies?

63
Sensor placement

• Later processing
• Cheat with good placement orientation

64
Sensor placement

• Mounting
• At desired measuring points
• Slack for bending limbs
• Comfortable(-ish) mounting
• Anti-slip
• Sensors all oriented same way

65
Hardware
66
Recordkeeping

• Write down sensor position orientation!
• Which sensors where?
• Take pictures!
• Label your sensors!
• Do it the same way every time!

67
Hardware
Wireless
68
Cables vs Wireless

• Cables
• No batteries
• Reliable connection
• Secure
• Hard to mount
• Cable tangles catching
• Cable breaking

• Wireless
• Power
• Packet loss
• Insecure
• Easy to mount
• Freedom of movement

69
Hardware
70
Cable Management

• Connection to sensor
• Flexible cable stiff board breaking
• Movement of cables along body
• Allow full range of body movement
• Consider special clothing
• Keeping cables from catching
• Danger doorknobs!

71
Cable management

• Anti-slip
• Central data collection

72
Cable management
73
Hardware
74
Wireless

• Options
• Bluetooth, ZigBee, Nordic, etc...
• Transmission

75
Hardware
76
Power

• How to power your sensors?
• Cables
• Only 1 battery to charge
• Cables power distribution
• Batteries
• Multiple charging necessary
• No cables
• Make batteries replacable!

77
Hardware
78
Data collection hardware
79
Data collection hardware

• Good mounting important!
• comfort
• cables
• wireless
• access
• feedback

80
Software
Synchronization
Data Collection
Feedback
Moving data
Processing
81
Software
Synchronization
Data Collection
Feedback
Moving data
Processing
82
Data collection

• More automatic better!
• (Sensors in same place every time)
• Saving data
• Human readable is useful
• Timestamp everything!
• Packet numbers too, if you can
• Meaningful names (arm-left.dat rathern than
  sensor5.dat)

83
Data collection

• Storing your data
• Think about structure for future use!
• Put a short note with each data run
• context/data/2006-10-14_1335/arm_left.txt
• context/data/2006-10-14_1335/arm_right.txt
• context/data/2006-10-14_1335/leg_left.txt
• context/data/2006-10-14_1335/leg_right.txt
• context/data/2006-10-14_1335/NOTES.txt

84
Software
Synchronization
Data Collection
Feedback
Moving data
Processing
85
Synchronization

• Why synchronize?
• Multi-sensor learning
• Concurrent actions
• Ground truth

86
Synchronization

• How?
• Timestamp each chunk of data
• 1132085944.59504 54078 475 465 535
• 1132085944.59565 54079 476 466 534
• 1132085944.75205 54080 476 470 536
• 1132085944.75222 54081 476 470 534
• 1132085944.75231 54082 478 470 533
• 1132085944.75238 54083 479 469 533

87
Synchronization

• How?
• Multi-sensor spike

88
Software
Synchronization
Data Collection
Feedback
Moving data
Processing
89
Feedback

• Are sensors still working?
• Visual feedback
• HMD or external monitor
• Show waveform or data stream
• Uniquely ID each sensor
• Auditory feedback
• Heart beat beeping
• Or beep when expected data does not arrive

90
Software
Synchronization
Data Collection
Feedback
Moving data
Processing
91
Moving data around

• Think about how to get data off
• Dont bury your USB ports!
• Keep backups!

92
Software
Synchronization
Data Collection
Feedback
Moving data
Processing
93
gt2k is

• A user interface toolkit designed to enable the
  development of gesture-based applications.
• Written in Java for cross-platform use and easy
  integration into graphical user interface
  development tools.

94
When to use gt2k

• Applications using symbolic/iconic gesture
• Sign language
• handwriting
• Applicable to research in many fields
• Human-Computer Interaction
• Assistive Technologies
• Robotics
• Brain Research
• etc.

95
Outline

• Example applications
• WritingPad
• Accelerometer
• Camera
• gt2k architecture
• Overview
• Sensor
• Library
• Machine Learning
• Recognition Process
• Data collection
• Training
• Recognition
• In-depth sample example
• How to make your own application

96
Application WritingPad

• Sensor mouse sensor
• Allows a user to draw a gesture with a mouse

97
Application Accelerometer

• Sensor Accelerometer
• Collects information about users physical
  movement

98
Application Vision Tracker

• Sensor Image Sensor
• Tracks the movement of objects

99
Architecture overview
Application
Machine Learning
Result
Wrapper
Request
CU-HTK
Library
Segment
Data
Library
Utils/Tools
Sensor
Sample
Sample
gt2k api
user/3rd party app.
100
Sensor

• Interfaces with the hardware and collect data.
• Provides parsing or post-processing of the data.
• Designed around an event-based architecture.
• Allows for both synchronous or asynchronous
  reading of sensors.

101
Library

• Responsible for storing and organizing data.
• Composed of a collection of samples.
• lt?xml version"1.0" encoding"UTF-8"?gt
• ltlibrary name"WritingPadLib"gt
• ltsamplesgt
• ltsample id"1"gt
• lttag end"144" label"star" start"0"/gt
• ltfvectorsgt
• ltvgt0.0 0.0lt/vgt
• ltvgt-0.32175055 3.1622777lt/vgt
• ltvgt-1.815775 4.1231055lt/vgt
• lt/fvectorsgt
• lt/samplegt
• lt/samplesgt
• lt/librarygt

102
Machine Learning

• Provides the toolkits abstraction for the
  machine learning algorithms.
• Used for modeling data samples (training) and
  recognizing gesture samples.
• Utilizes Cambridge Universitys Hidden Markov
  Model Toolkit (CU-HTK).

103
Gesture Recognition Process

• Data Collection
• Segmentlabel data
• Store data as feature vector

• Training
• Train model with collected data

• Recognition
• Recognize new raw data using trained
  model

104
Data Collection
Application
Machine Learning
Result
Request
?
?
Library
Segment
Data
Library
?
Utils/Tools
Sensor
Sample
Sample
105
Training
?
Application
Machine Learning
Result
?
Request
Library
Segment
Data
?
Library
?
Utils/Tools
Sensor
Sample
Sample
106
Recognition
?
Application
Machine Learning
Result
Request
?
?
Library
Segment
Data
Library
?
Utils/Tools
?
Sensor
Sample
Sample
107
How to make your own application?

• Considerations
• What kind of sensor will you use?
• Is it already in the gt2k package or should I
  build new sensor?
• How are you going to segment data?
• Mouse Sensor segments data by clicking (start)
  and releasing (stop) mouse button
• How big is my feature vector?
• Vector size of mouse sensor 2 (X and Y)
• Vector size of accelerometer 3 (X, Y and Z)
• What if I have two accelerometer sensors?

108
How to make your own application?

• Create a new set of options
• GT2kOptions myOpts GT2kOptions()
• myOpts.setVectorSize(2)
• Create or load a library object
• try libraryLibrary.load(MyLibrary.xml")
• catch (Exception e)
• librarynew Library(MyLibrary")
• Initializes the machine learning component
• HTK htk new HTK(myOpts)
• Create a new sensor
• MouseDragVectorSensor mySensor new
  MouseDragVectorSensor()

109
How to make your own application?

• For data collection
• Connect the sensor to the library so it can save
  the samples
• mySensor.addSensorSampleListener(library)
• For recognition
• Connect the sensor to the HTK object
• mySensor.addSensorSampleListener(htk)
• Connect HTK object to application to get a result
• htk.addResultListener(myApplication)

110
How to make your own application?

• WritingPad application structure

• WritingPadPanel
• creates all of the components
• of UI
• -defines all buttons actions

• WritingPad
• Initializes the program
• -creates the main frame
• Instantiates the needed gt2k objects
• (MouseSensor, Library
• , Options and HTK)
• -connects a sensor, library, HTK,
• Result objects

• CoordinateArea
• Displays the on-screen grid for user
• feedback
• -listens to the MouseSensor and
• plots points

111
How to make your own sensor?

• Considerations
• Make sensor Runnable?
• Mouse sensor implements mouse listeners
• Accelerometer sensor implements Runnable
• Start or stop from application
• Drivers and Java library for your sensor
• May not work under Windows but Linux

112
How to make your own sensor?

• Things to do/write
• Communication with physical sensor
• Parses output (float type)
• Connects to the gt2k sensor infrastructure

113
How to make your own sensor?

• Structure

• Sensor
• Send out data or sample to the listeners
• add/remove data listener
• - add/remove sample listener

extends

• YourSensor
• - Provide Post-processing (parsing output)
• Whenever data gets generated,
• sends data.
• Whenever one gesture gets created,
• sends a sample.

• YourSensorController
• Drivers or communication tools with
• your physical sensor

114
The End

• Thank you!