SenseCam Work at Dublin City University

1
SenseCam Work at Dublin City University Alan F.
Smeaton, Gareth J.F. Jones and Noel E. OConnor
(PIs) Georgina Gaughan, Cathal Gurrin, Hyowon
Lee, Hervé Le Borgne (PostDocs)Aiden Doherty,
Michael Blighe, Ciarán ÓConaire, Michael McHugh,
Saman Cooray (PhD students) Barry Lavelle, Paul
Reynolds (Masters students)Sandrine Áime (Summer
student) 15 people working on SenseCams in
some way at DCU Center For Digital Video
Processing,Dublin City University, Ireland
2
Overview

• Our contribution to developing SenseCam work
• Automatic event segmentation - 3 approaches
• Application generation of rolling weekly summary
  based on Addenbrooks
• Face detection and body patch matching
• Arizona data
• Using BT and other sensors for context
• Alternative way to presenting SenseCam images

3
Our (DCU) Contribution

• We do image/video analysis, indexing,
  summarisation, etc. and we apply this to SenseCam
  data
• We have no particular SenseCam application, we
  will develop underlying technology
• Were keen to hear about the real problems of
  SenseCams in practice, and to offer
• We consider the typical full-day SenseCam images,
  do event segmentation and summarisation

4
A days SenseCam images (3,000 4,000)
5
Automatic Event Segmentation

• Task automatically determine events from a
  collection of SenseCam image data
• Based around image-image similarity using
  MPEG-7 features where differences may indicate
  events
• Similar problem to shot bound detection in video
  but more challenging given the fish-eye view and
  lesser similarities within an event vs. a shot
• Several approaches can be taken

6
Similarity Calculation between 2 Images
7
Event Segmentation Approach I
One Days Images
......
......
......
8

• Stage 1
• comparison of adjacent images
• Stage 2
• Comparison every 2nd image
• Stage 3
• Comparison of blocks of images
• Incorporation of a face detector

9
Preliminary Results Images from 1 day
Number of pictures 2685Manually detected
events 27 Lots more to do, including
fusion of descriptors and optimising windowing
10
Event Segmentation II

• Use similarity clustering, and time
• Combine low-level content analysis and context
  information (i.e. metadata provided by the
  SenseCam and temporal data)
• Generate a similarity matrix by fusing low-level
  and metadata information
• Implement time constraints to constrain
  clustering
• Simple hierarchical clustering of images into
  events

11
Event Segmentation Approach II
One Days Images
12
Event Segmentation Approach II
One Days Images
13
Event Segmentation Approach II
One Days Images
14
Approach II Results
15
Approach III Group Images into 3 Classes

• Static Person
• Person performing one activity
• E.g. at computer, meeting, eating etc.
• Moving Person
• Travelling between locations
• Static Camera
• Sense Cam is put down
• User is not wearing it

16
Features Used

• Block-based Cross-Correlation
• Spatiogram image colour similarity
• Compares image colour spatial distribution
• Accelometer motion
• Feature-based training
• Using Bayesian approach to classification
• Viterbi algorithm used to smooth results
• Applied to 1 day SenseCam images so far

17
Event Segmentation Approach III
One Days Images
18
Accelerometer Data Example
19
Generation of Weekly Summaries

• Assume events already segmented
• Calculate average values for events of low level
  features from all images
• Generate similarity matrix using the average
  value from each event
• Visually similar events can then be detected, and
  the time period (week) structured automatically
  into a short movie
• Why a movie week Addenbrookes Cambridge
  application

20
Generation of Weekly Summary
Event-Segmented image sets
21
Generation of Weekly Summary
Event-Segmented image sets
Similar Events - Aiden working on the desk
Mon
Tue
Wed
Thr
Compare Event-Event similarity within a week
Fri
Sat
Sun
22
Generation of Weekly Summary
Event-Segmented image sets
Similar Events - Aiden waiting for bus
Mon
Tue
Wed
Thr
Compare Event-Event similarity within a week
Fri
Sat
Sun
23
Generation of Weekly Summary
Event-Segmented image sets
Similar Events - Aiden at the office corridor
Mon
Tue
Wed
Thr
Compare Event-Event similarity within a week
Fri
Sat
Sun
24
Generation of Weekly Summary
Event-Segmented image sets
Mon
Unique Event 1
Tue
Wed
Thr
Compare Event-Event similarity within a week
Fri
Sat
Sun
25
Generation of Weekly Summary
Event-Segmented image sets
Similar Events - Aiden waiting for bus
Mon
Similar Events - Aiden at the office corridor
Tue
Similar Events - Aiden working on the desk
Unique Events
Wed
Thr
Compare Event-Event similarity within a week
Fri
Sat
Sun
26
Generation of Weekly Summary
Event-Segmented image sets
Similar Events - Aiden waiting for bus
Mon
Similar Events - Aiden at the office corridor
Tue
Similar Events - Aiden working on the desk
Unique Events
Wed
Thr
Fri
Compare Event-Event similarity within a week
1 Week summary
(on Sunday)
Select images
Sat
Sun
27
Generation of Weekly Summary
Event-Segmented image sets
Similar Events - Aiden waiting for bus
Mon
Similar Events - Aiden at the office corridor
Tue
Similar Events - Aiden working on the desk
Unique Events
Wed
Thr
Fri
1 Week summary
(on Sunday)
Select images
Sat
Compare Event-Event similarity within a week
Sun
Select images
(on Monday)
Mon
28
Generation of Weekly Summary
Event-Segmented image sets
Similar Events - Aiden waiting for bus
Mon
Similar Events - Aiden at the office corridor
Tue
Similar Events - Aiden working on the desk
Unique Events
Wed
Thr
Fri
1 Week summary
(on Sunday)
Select images
Sat
Sun
Select images
(on Monday)
Compare Event-Event similarity within a week
Mon
Select images
(on Tuesday)
Tue
29
Preliminary Results
Number of similar images to a known event, from
top 10 retrieved
30
Face Detection Body Patch Matching

• Apply face detection software to detection the
  presence of a face in the SenseCam image
• Body Patch Matching
• Identify similar body patch by color to detect
  subsequent appearances within an event
• This works well for personal photos, but SenseCam
  images are lower quality

31
Similarity Comparison by Person Detection
828am, 7 June 2006
503pm 30 May 2006
32
Arizona State U. Data

• ASU gave us some SenseCam data 2 weeks ago
• Session rather than all-day images
• Applied automatic event detection using 4x MPEG-7
  low-level feature descriptors
• Both Color Structure and Color Moments outperform
  others
• Face Detection software performs badly on this
  data
• Blurred Images cause standard face detection
  software to fail

33
Event detection using ASU data 28-June-2006
Number of pictures 357 Manually detected events
28
34
Event detection using ASU data 28-June-2006
Number of pictures 434 Manually detected events
11
35
Using BT to provide context

• Achieved by logging Bluetooth devices in close
  proximity to the SenseCam wearer
• May be useful in determining which individuals
  are present around each picture
• Application created to poll and log Bluetooth
  devices on phone
• Currently developing host application to
  interface with mobile device and retrieve log
  file
• Next step synchronize time-stamps between
  SenseCam images and Bluetooth log file

36
Use of Multi-Sensor Data

• Concept To determine whether events can be
  identified based on multiple sensor data
• Data collected from
• GPS Device
• BodyMedia Device
• Heart Rate Monitor
• SenseCam
• Development of a framework to extract the
  relevant data from the different data sources
• CSV files, XML files, text files, Excel files

37
Presenting SenseCam Images?
E.g. intelligent summary of one day (playback for
1 minute)

• ... watching the fast playback of image sequences
  is not an ideal interaction
• Intensive concentration required during playback
• Event boundaries cannot be clearly presented
• Sense of time is skewed (more images of an
  important event, even if it lasted only 1
  minute less images of unimportant regular
  events even if they last many hours during the
  day)

38
Turn sequential playback into an interactive,
spatial browsing interaction (similar to the way
we turn video playback into keyframe browsing) gt
39
31 May 2006

• Approach
• 1-page visual summary of a day
• Each image represents each event
• Size of each image represents the importance
  or uniqueness of the event
• Timeline on top orientates the user about time
  when each event happened
• Mouse-Over activated

40
31 May 2006
41
31 May 2006
42
31 May 2006
43
31 May 2006
44
31 May 2006
Then my last desk-work of the day (2 hours) just
after lunch time
45
31 May 2006
My lunch break
46
31 May 2006
My dinner time
47
31 May 2006

• Conclusion
• More relaxed, interactive, inviting summary of
  the day than fast-forwarding, while still taking
  advantage of playback synergy effect
• Playing each of the Events in its location
  might be also good (without having to Mouse-Over)
• Importance is not by playing more images in
  that Event (this skews time), but by larger image
  size

48
Papers written

• Exploiting context information to aid landmark
  detection in SenseCam images, submitted to
  ECHISE - 2nd International Workshop on Exploiting
  Context Histories in Smart Environments
  Infrastructures and Design to be held at 8th
  UbiComp, Sept. 2006, Irvine, CA, USA
• Structuring a Visual Lifelog Diary by
  Automatically Linking Events, submitted to 3rd
  ACM Workshop onCapture, Archival and Retrieval of
  Personal Experiences (CARPE 2006) October, 2006,
  Santa Barbara, California, USA.
• Organising a daily visual diary using
  multi-feature clustering, submitted to SPIE
  Electronic Imaging, San Jose, January 2007

49
Future Work

• EVERYTHING !