The Microsoft SenseCam and Other Lifelogging Devices

1
The Microsoft SenseCam and Other Lifelogging
Devices

• Alan F. Smeaton Noel E. OConnnor
• Dublin City University

2
Overview

• The task - lifelogging
• The technologies
• The SenseCam - the device
• Our work on Event-based SenseCam image browsing
• What were doing next

3
Lifelogging

• Lifelogging is about recording daily life,
  digitally
• Sometimes its for a reason,
• work e.g. security personnel, medical staff,
• personal e.g. diaries, etc.
• Sometimes its for posterity, recording vacations,
  family gatherings, social occasions
• Sometimes its because we can, and were not yet
  sure what well do with lifelogs, e.g. MyLifeBits

4
Technologies

• Technologies for lifelogging broadly divide into
• Logging cyberspace activities thats obvious
• Recording biometrics
• Logging our environment
• Lets see our work on biometrics and environment

5
Recording Biometrics

• Recording biometrics
• Polar Heart Rate Monitor
• BodyMedia SenseWear Armband
• Galvanic Skin Response (GSR), heat flux, skin
  temperature, accelerometer
• Foster Miller vests
• Respiration, body temperature, heart rate, GPS
• Posture monitoring vest
• 18 wearable plastic optical fiber sensor outside
  the garment, on the spine, coated in paint and
  scratched along one side, used for measuring
  bending on structural beams

6
Posture Monitoring
7
Recording Biometrics

• Recording biometrics
• Polar Heart Rate Monitor
• BodyMedia SenseWear Armband
• Galvanic Skin Response (GSR), heat flux, skin
  temperature, accelerometer
• Foster Miller vests
• Respiration, body temperature, heart rate, GPS
• Posture monitoring vest
• 18 wearable plastic optical fiber sensor outside
  the garment, on the spine, coated in paint and
  scratched along one side, used for measuring
  bending on structural beams
• Smart Textiles
• Move from discrete sensors with electronic
  components attached to fabric to functionalised
  fabrics which sense stretching, bending,
  pressure, movements
• How ?

8
Some Basic Chemistry

• Polymers are macromolecules, and usually they are
  insulators but some, such as polypyrrole, conduct
  electricity (c.1970)
• Known as conducting polymers or synthetic
  metals
• We can now coat onto substrates including
  textiles like foam or lycra or anything that
  moves, twists, bends
• These conducting textiles can be used as wearable
  sensors, responding to stress or strains by
  changing their electrical conductivity
• They are
• Easily produced
• Show rapid response times
• Can be comfortable to wear

9
Some Basic Chemistry
10
Recording Biometrics

• Recording biometrics
• Polar Heart Rate Monitor
• BodyMedia SenseWear Armband
• Galvanic Skin Response (GSR), heat flux, skin
  temperature, accelerometer
• Foster Miller vests
• Respiration, body temperature, heart rate, GPS
• Posture monitoring vest
• 18 wearable plastic optical fiber sensor outside
  the garment, on the spine, coated in paint and
  scratched along one side, used for measuring
  bending on structural beams
• Smart Textiles
• Move from discrete sensors with electronic
  components attached to fabric to functionalised
  fabrics which sense stretching, bending,
  pressure, movements
• Smart shirt uses band of polypyrrole-coated
  elastic around the chest

11
Smart shirt monitoring breathing
Vmax 229 machine
Exercise Shirt
Logging Laptop
Base Station
Breathing rate can be measured, validated using
standard metabolic system.
12
Lifelogging your Environment

• Possibilities are recording audio, visual, and
  sensor values
• Rest of this talk is about SenseCam, which
  records visual and some sensor values

13
SenseCam

• SenseCam is a Microsoft Research Prototype
• Multi-sensor device
• colour camera
• 3 accelerometers
• light meter
• Passive infrared sensor
• 1GB flash memory storage of a few days
• Smart image capture 3 images/min
• Since April 2006 weve had two SenseCams

14
SenseCam

• Captured images
• Fisheye lens
• No variable aperture
• Low resolution

15
Quality Analysis

• Randomly Selected 1000 Images from 1 million
• Manually annotated for quality

16
Quality Analysis

• Lots of low quality images, but even poor ones
  are useful
• Very few really excellent photos
• Estimate approx 70 per day (out of 3,000)
• Going forward we will filter the collection using
  automatic methods
• Blur Determination
• Depth of Colour
• Focus
• Image Noise

17
Our SenseCam Use

• SCs used by others to record events we record
  the whole day
• One user using SC constantly, other one passed
  around from person to person as needed
• Over last 18 months weve developed techniques
  for SenseCam data management, without having user
  input or direction
• so our work is technologically-driven rather
  than based on user pull
• How good were our guesses ?

18
1,000,000 SenseCam Images
Millionth Image
Millionth Image

• One user wearing SC for 15 months
• Over 1 million SenseCam images
• Each with GPS position !
• Experiences
• Most people dont notice the camera
• Those that do always remember!
• Most people dont mind the camera
• Have been spotted/greeted by people who have
  heard about the guy with the camera
• About 40 of photos captured are low quality,
  even more are stop-photos (banal photos of
  typical scenes like driving or working at desk).
• Need an extremely understanding girlfriend!

Most Important Image
Most Important Image
19
Users thoughts after 15 months

• Event browsing is key
• Too many photos to browse, need event summary and
  then drill down to view event in detail
• Stop events, (like work desk and driving) can be
  hidden.
• Total Recall, little sign of Event Decay
• I remember nearly every (non stop-) event when I
  see it
• I did not expect this!
• Important axes for event search are
• Location of the event
• People in the event
• Time based organisation less important
• I will probably not remember time/day/date, but I
  will remember location and people there.

20
Types of People Encountered

• Not Bothered/Not Notice just dont care, this
  accounts for most people.
• Cautious ask if they are being captured, then
  ask if it captures audio too, always remember
  the sensecam and comment still wearing it when
  meet you again. Usually these people become Not
  Bothered types.
• Sensitive dont like it on at all, will try to
  get you to take it off, often are people with
  cameras themselves!
• Avoider avoid contact because of camera, or at
  least avoid sitting in front of you.
• Argumentative point out they dont give
  permission to take their photo, argue a lot,
  dont accept.

Most
Least
21
What have we done

• Event segmentation (v1) and event importance
• Event-based browser
• Event segmentation (v2) using multi-feature
  clustering
• Biometrics-influenced landmark image detection
• Setting detection using SIFT features
• Automatic event augmentation with images
• Concept detection for events
• Bluetooth logging and SenseCam images
• Location mapping of SenseCam images

22
A days SenseCam images (3,000 4,000)
23
Event Segmentation - V1
One Days Images

• Raw data
• Similarity matching
• Normalisation Data fusion
• Thresholding
• Events

Shot Boundary Detection
OR TextTiling
... adjacent blocks of 10 images/sensor vals
......
......
149

120
289
24
Event Segmentation Expts.

• How well does it work ?
• Work is already published at RIAO2007 conference
  (1 user and 25k images)
• Recently completed extensive experiments with 5
  different users wearing SenseCam for 1 month each
  (total 270k images)
• Each user groundtruthed their own data
• Data divided into training and test sets with
  over 3,000 different approaches evaluated

25
Event Segmentation Expts.

• From groundtruth we noticed
• Average of 1,785 images per user per day
• Average of 20 events groundtruthed per day
• 2 Approaches Recommended
• Most accurate (include MPEG-7 features)
• Quick segmentation (sensor values only)
• Performance
• RIAO (f score 0.40)
• Sensor only (f score 0.55)
• Image Sensor (f score 0.60)

26
What have we done

• Event segmentation (v1) and event importance
• Event-based browser
• Event segmentation (v2) using multi-feature
  clustering
• Biometrics-influenced landmark image detection
• Setting detection using SIFT features
• Automatic event augmentation with images
• Concept detection for events
• Bluetooth logging and SenseCam images
• Location mapping of SenseCam images

27
Daily Browser Overview
SenseCam Images of a day (about 3,000)
28
(No Transcript)
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
What have we done

• Event segmentation (v1) and event importance
• Event-based browser
• Event segmentation (v2) using multi-feature
  clustering
• Biometrics-influenced landmark image detection
• Setting detection using SIFT features
• Automatic event augmentation with images
• Concept detection for events
• Bluetooth logging and SenseCam images
• Location mapping of SenseCam images

33
3 Features Extracted

• (1) Colour Features
• Image spatiogram
• (2) Edge features
• Block-based cross-correlation
• (3) Motion features
• Accelerometer readings
• Complimentary and independent Features

34
Multi-feature Cluster Overview
One Days Images
35
Multi-feature Cluster Overview

• Classify Images into 3 classes
• Static Camera (SC)
• Static Person (SP)
• Moving Person(MP)

36
What have we done

• Event segmentation (v1) and event importance
• Event-based browser
• Event segmentation (v2) using multi-feature
  clustering
• Biometrics-influenced landmark image detection
• Setting detection using SIFT features
• Automatic event augmentation with images
• Concept detection for events
• Bluetooth logging and SenseCam images
• Location mapping of SenseCam images

37
Keyframe Detection

• Images segmented into events, but how to
  represent events ?
• Biometric sensors indicate arousal, such as
  excitement or boredom, at time of image capture,
  pointing to good keyframes, perhaps
• Biometric data for a given event is combined
  above certain threshold represents significant
  point
• Images closest to this time form keyframes for
  event
• Work being evaluated, no results yet

38
Keyframe/Landmark Detection
39
What have we done

• Event segmentation (v1) and event importance
• Event-based browser
• Event segmentation (v2) using multi-feature
  clustering
• Biometrics-influenced landmark image detection
• Setting detection using SIFT features
• Automatic event augmentation with images
• Concept detection for events
• Bluetooth logging and SenseCam images
• Location mapping of SenseCam images

40
Setting Detection

• Aim to identify events captured at the same real
  world location (e.g. in the dining room at home,
  in front of the computer in the office, in the
  park, )
• Performed using Scale Invariant Feature Transform
  (SIFT) Features
• SIFT is invariant to image rotation, scale,
  intensity change, and to moderate affine
  transformations

41
SIFT
42
Setting Detection Watching TV
43
Setting Detection In the Park
44
Setting Detection At Home
45
What have we done

• Event segmentation (v1) and event importance
• Event-based browser
• Event segmentation (v2) using multi-feature
  clustering
• Biometrics-influenced landmark image detection
• Setting detection using SIFT features
• Automatic event augmentation with images
• Concept detection for events
• Bluetooth logging and SenseCam images
• Location mapping of SenseCam images

46
Event Augmentation

• Augment low-quality SenseCam images with high
  quality images from external sources

47
Daily Browser Overview
SenseCam Images of a day (about 3,000)
Event Augmentation
48
Event augmentation Croke Park
Heres an image from a SenseCam after a big match
in Croke Park. Wed really like to see other
peoples pictures of this match.Lets search by
location
49
Event augmentation Croke Park

• Receive the following pictures
• Then filter out to just those results from the
  same day

50
Event augmentation Santa Barbara
Heres a SenseCam picture of a building that I
like from the pier in Santa Barbara, CA. Again I
search for other pictures in the same location
51
Event augmentation Santa Barbara

• I receive the following pictures
• Then I filter out to just those results that are
  visually similar

52
Event augmentation - Chalkidiki
Heres an image from my SenseCam at a beach in
Chalkidki in Greece. Id really like to see other
peoples pictures of this beachTherefore I
search by location firstly
53
Event augmentation - Chalkidiki

• I receive the following pictures
• Then I filter out to just those visually similar
  results

54
Event augmentation New York
Heres an image from my SenseCam looking towards
the Statue of Liberty in New York. Id really
like to see other peoples pictures that are
similarTherefore I search by location firstly
55
Event augmentation New York

• I receive the following pictures
• Then I filter out to just those visually similar
  results

56
What have we done

• Event segmentation (v1) and event importance
• Event-based browser
• Event segmentation (v2) using multi-feature
  clustering
• Biometrics-influenced landmark image detection
• Setting detection using SIFT features
• Automatic event augmentation with images
• Concept detection for events
• Bluetooth logging and SenseCam images
• Location mapping of SenseCam images

57
Concept Detection

• Automatic concept detection in photo and image
  management is possible
• Concepts, semantic features, use low-level
  features (colour, texture, edges, shapes, motion,
  audio) and and train a SVM or somesuch
• Feature detection important in TRECVid
• Feature detection not accurate when examined
  independently, but useful
• Analogous to ASR, OCR, other noisy recognisers
• Concepts for SenseCam image, or events ?

58
Concept Detection

• Working with University of Amsterdam to define
  and build feature detectors (visually-based) for
  SC images
• These will then be combined with other sensor
  values to assign concepts to events
• This will be useful in SC image management
• Carefully chosen set of SC concepts, several
  iterations
• Work ongoing

59
Concept Detectors
Suite of Sensecam specific concept detectors
under development

• Steering wheel (driving)
• Shopping
• Inside of vehicle when not driving (airplane,
  taxi, car, bus)
• Toilet/Bathroom
• Giving Presentation / Teaching
• View of Horizon
• Door
• Staircase
• Hands
• Holding a cup/glass
• Holding a mobile phone

• Food (eating)
• Screen (computer/laptop/tv)
• Newspaper/Book (reading)
• Meeting
• Road
• Vegetation
• Snow
• Office Scene
• Faces
• People
• Animal
• Grass
• Sky

60
What have we done

• Event segmentation (v1) and event importance
• Event-based browser
• Event segmentation (v2) using multi-feature
  clustering
• Biometrics-influenced landmark image detection
• Setting detection using SIFT features
• Automatic event augmentation with images
• Concept detection for events
• Bluetooth logging and SenseCam images
• Location mapping of SenseCam images

61
Bluetooth Context Logging

• Weve built an infrastructure for logging
  Bluetooth device occurences
• Runs on fixed and mobile devices
• Applications are context-sensitive advertising on
  billboards, and also enabling people-based search
  of SenseCam events
• Provides additional detail to annotate events
• Who was present ?
• How long for ?
• How important are they based on Bluetooth log

62
Bluetooth Familiarity

• We can determine how important an encountered
  device/person is by logging their presence over
  time
• This forms a.k.o. social network based on
  co-occurrence and co-location
• Our algorithms provide a measure of device/person
  familiarity
• Once calculated we can classify devices as
• Familiar (really well known)
• Familiar strangers (somewhat known)
• Strangers (unknown)

63
SenseCam Context Data
64
What have we done

• Event segmentation (v1) and event importance
• Event-based browser
• Event segmentation (v2) using multi-feature
  clustering
• Biometrics-influenced landmark image detection
• Setting detection using SIFT features
• Automatic event augmentation with images
• Concept detection for events
• Bluetooth logging and SenseCam images
• Location mapping of SenseCam images

65
Location Mapping Sensecam
An external GPS device maintains a log of where
the wearer has been. Photos are later location
stamped by matching timestamps.
66
Location Mapping Sensecam
67
Location Mapping Sensecam

• Experimental integration of location stamped
  visual lifelog with Visual Mapping Software
• Typical Scenario
• Requires
• GPS location stamping
• Feature detectors for (people, food, eating,
  etc)
• Event Segmentation
• With Key photo selection

I recently visited Asia, find me a sequence of
events where I was eating with other people in
both Korea and China.
68
Retrospective

• So how good were our guesses for user-pull usage
  ?
• Most of the things we developed are useful
• Where are we headed now that we know a few things
  ?

69
INTERFACE STRATEGIES
OUTCOME OF THE PROCESSING/SENSING
SUPPORTED TASKS

• Event Segmentation
• Event Classification
• Novelty Calculation
• Landmark Selection
• GPS location tagging

Comic-book style

• Standing/Sitting
• Walking
• Running
• Emotion intensity

• Gisting the day
• Re-living the day
• Searching for event
• Geographic browsing

Fast playback
Matching
Map navigation
70
INTERFACE STRATEGIES
OUTCOME OF THE PROCESSING/SENSING
SUPPORTED TASKS

• Event Segmentation
• Event Classification
• Novelty Calculation
• Landmark Selection
• GPS location tagging

Comic-book style

• Standing/Sitting
• Walking
• Running
• Emotion intensity

• Gisting the day
• Re-living the day
• Searching for event
• Geographic browsing

Fast playback
Map navigation
71
INTERFACE STRATEGIES
OUTCOME OF THE PROCESSING/SENSING
SUPPORTED TASKS

• Event Segmentation
• Event Classification
• Novelty Calculation
• Landmark Selection
• GPS location tagging

Comic-book style

• Standing/Sitting
• Walking
• Running
• Emotion intensity

• Gisting the day
• Re-living the day
• Searching for event
• Geographic browsing

Fast playback
Map navigation
72
INTERFACE STRATEGIES
OUTCOME OF THE PROCESSING/SENSING
SUPPORTED TASKS

• Event Segmentation
• Event Classification
• Novelty Calculation
• Landmark Selection
• GPS location tagging

Comic-book style

• Standing/Sitting
• Walking
• Running
• Emotion intensity

• Gisting the day
• Re-living the day
• Searching for event
• Geographic browsing

Fast playback
Map navigation
73
Requirements

• Integrate a number of different SenseCam browsing
  styles that we have come up with so far
• Simple, easy to use (non-techy style)
• User starts with a task
• Swapping between different UIs maintains the
  currently viewed date/event/photo

74
(No Transcript)
75
(No Transcript)
76
(No Transcript)
77
...with stronger daily progression cue to
increase the feeling of re-living the day
78
(No Transcript)
79
(No Transcript)
80
(No Transcript)
81
(No Transcript)
82
(No Transcript)
83
New browsing ideas easily integrated in a way
that can support a particular user task Swapping
between browsers maintains the currently viewed
event/date
84
Summary

• This is the roadmap for future work much is
  already done.
• As to the SenseCam what would we like ?
• Less conspicuous
• It has a useful battery life of only 6-9 months
• Replaceable batteries ?
• Lens gets scratched after a year
• Replaceable lenses ?
• Sleep button (2 mins. off)
• Short-term audio capture when required
• and when manually capturing photo
• Mood Sensor ? Important axis for organisation
• Higher resolution would be good
• Difficult to print for a photo album!

85
The DCU SenseCam team

• Alan Smeaton
• Noel OConnor
• Gareth Jones
• Cathal Gurrin
• Hyowon Lee
• Michael Blighe
• Daragh Byrne
• Aiden Doherty
• Liadh Kelly
• Ciarán ÓConaire
• Juncheng Lu (James)
• Tim Kersten