SenseCam Project

1
SenseCam Project Under the supervision of Alan
Smeaton et Georgina Gaughan 12 June 2006 1
Sept. 2006 DCU, Dublin, Ireland
2
Overview

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images
• Comparison of blocks of images
• Results
• Evaluation and measures
• Comments
• Conclusion

3

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

Introduction

• - The SenseCam
• Little wearable camera
• Taking about 3, 000 pictures per day
• Responding to changes such as bright lights or
  temperature and sudden movements
• Prototype designed by Microsoft Research
• Acting like a visual diary

The SenseCam
4

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

Introduction
- Aim -
5

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

Introduction

• - Aim -
• Automatically detect the various events in a
  recorded day by extracting some landmark images
  which summarise each activity (breakfast, work,
  lunch etc.)
• - Why? -
• Watching an entire
• day takes approximately 3-4 minutes
• week takes approximately 15-20 minutes
• Importance of quickly and efficiently having
  access to the main significant events of a day

6

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

Introduction

• - Main ideas -
• Compare the content of the images, using a
  variety of low-level metrics (colour moment,
  colour structure)
• Calculate the similarity coefficient between the
  images to see how different they are from each
  other

7

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

The different approaches
- Comparison of adjacent images -
If similarity score gt threshold Then possible
event
We have to consider that it may have some
interruptions due to a movement, or a change in
the light.
8

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

The different approaches

• - Comparison of blocks of images adaptation of
  theText Tiling algorithm -
• Aim find boundaries between different topics in
  a text
• Method
• Compare each pair of adjacent blocks (usually
  3-5 sentences long) of text according to how
  similar they are lexically

block 1
block 2
block 2
block 3

9

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

The different approaches

• Order the similarity values according to how
  steeply the slopes of the plot are to either side
  of the gap, rather than by their absolute
  similarity score, as in the first approach

10

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

Results

• - Evaluation -
• One way to evaluate how to perform these
  approaches is to compare them against a judgment
  made by a human being
• Put landmarks manually in a collection of
  SenseCam images
• Use the precision and recall measures
• - Measures -
• Precision ability of a system to present only
  relevant items
• Recall ability of a system to present all
  relevant items

11

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

Results

• - Day 1 -
• - Day 2 -

Number of pictures 2679 Number of manual
landmarks 27

• Legend
• Green first approach
• Red second approach

Number of pictures 3457 Number of manual
landmarks 35
12

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

Results

• - Comments -
• To balance the gap between recall and precision
  measures, an alternative approach is to mix the
  two other ones
• First comparing adjacent images
• Then using the second approach to find the
  boundary images

• Légende
• Vert jour 1
• Rouge jour 2

13

• Introduction
• The SenseCam
• Aim
• Why?
• Main ideas
• The different approaches
• Comparison of adjacent images

Results

• - Conclusion -
• Mix of the two approaches
• good compromise
• not sufficient to get all landmarks images
• Future work incorporate the other data recorded
  by the SenseCam, like the temperature and to
  combine them with the existent approaches
• Could be useful to detect changes between inside
  and outside events