Survey on Face and Context Based Photo Clustering

1
Survey on Face and Context Based Photo Clustering

• Zhu Jianwei
• is04zhjw_at_mail2.sysu.edu.cn
• 2009.3.2_at_Selab

2
Outline

• Introduction
• Related Works
• Motivations
• Experiment Example
• Conclusion and Future Work

3
Introduction

• Family photo management
• A few persons families and friends (lt 50)
• Most of photos have person in it.
• User is interested in When, Where, Who and What.

4
Introduction

• Why clustering?
• Annotating photo one by one is tedious.
• Automatic (semi-automatic) clustering tool for
  batch annotation alleviates users burden.
• Clustering Methods
• Time (Event) / Spatial / Person

5
Introduction

• Clustering Methods
• By Date/Time
• Sort the photos by time, and cluster them into
  events.
• By Position
• Using GPS information to locate the place.
• They can be extracted from the EXIF of a photo.

6
Introduction

• Clustering Methods
• By Content
• Extract the low-level features from photo.
• Especially by person, usually combine the face
  detection , recognition and CBIR technology.
• By Tag (Label)
• High level semantic features.

7
Related Works

• Related researches
• Face detection and recognition
• Robust face detection is available now.1
• Face recognition precision is low and its easily
  affected by
• lighting, pose and variant expression.2
• Face recognition can not be directly applied to
  photo clustering.

1 Paul V.,Michael J., Robust Real-Time Face
Detection, International Journal of Computer
Vision 57(2), 137154, 2004 2 Zhao W.,
Chellappa R., Rosenfeld A. and Phillips P., Face
recognition A literature survey, Technical
Report, Maryland University, CfAR CAR-TR-948,
2000
8
Related Works

• Related researches
• Content based image retrieval.
• Partially it targets at solving the general image
  retrieval problems.
• No practical solutions to automatic family photo
  management. 3

3Zhang, L., Chen, L., Li, M., and Zhang, H.J.
(2003) Automated annotation of human faces in
family albums. Proceedings of ACM Multimedia,
355-358
9
Related Works

• 1. Semi-Automatic Image Annotation Using Event
  and Torso Identification 4
• Use time to cluster photos into events.
• Use face detection to find faces.
• During an event, use torso (body) information to
  cluster by person.
• (Usually a event is less than a day, one will
  wear the same clothes during the day)

4Suh, B., and Bederson, B.B. (2004)
Semi-Automatic Image Annotation Using Event and
Torso Identification, Tech Report HCIL-2004-15,
Computer Science Department, University of
Maryland
10
Related Works

11
Related Works

• 2. EasyAlbum An Interactive Photo Annotation
• System Based on Face Clustering and Re-ranking 5

Cluster annotation
5 J. Cui, F. Wen, R. Xiao, Y. Tian, and X.
Tang. EasyAlbumAn interactive photo annotation
system based on face clustering and re-ranking.
Proc. CHI 2007. ACM Press, 2007.
12
Related Works

• 2. Easy Album An Interactive Photo Annotation
• System Based on Face Clustering and Re-ranking 5

Contextual re-ranking
5 J. Cui, F. Wen, R. Xiao, Y. Tian, and X.
Tang. EasyAlbumAn interactive photo annotation
system based on face clustering and re-ranking.
Proc. CHI 2007. ACM Press, 2007.
13
Related Works

• Framework

14
Related Works

• Evaluation Method
• Compare with Adobe Photoshop Elements 4.0 .
• With large number of photos, compare the time
  performance.
• Limit the time to see how fast users can organize
  their photos into right clusters.

15
Related Works

• 3. Approaches to Consumer Image Organization
  Based on Semantic Categories 6
• System overview

6 Cathleen D., Madirakshi D. Alexander C.Loui,
Approaches to Consumer Image Organization Based
on Semantic Categories, Proc. of SPIE
Vol.6391,63910L-1,2006
16
Related Works

• 3. Approaches to Consumer Image Organization
  Based on Semantic Categories 6
• Group by event

6 Cathleen D., Madirakshi D. Alexander C.Loui,
Approaches to Consumer Image Organization Based
on Semantic Categories, Proc. of SPIE
Vol.6391,63910L-1,2006
17
Related Works

• 3. Approaches to Consumer Image Organization
  Based on Semantic Categories 6
• Group by person

6 Cathleen D., Madirakshi D. Alexander C.Loui,
Approaches to Consumer Image Organization Based
on Semantic Categories, Proc. of SPIE
Vol.6391,63910L-1,2006
18
Related Works

• 3. Approaches to Consumer Image Organization
  Based on Semantic Categories 6
• Similarity Calculation
• (1)Use color histogram for facial feature
  measurement.
• (2)The distance of two clusters if defined as
• (3) Use nearest-neighbor algorithm to do
  clustering.
• (Like Kruskal MST)

6 Cathleen D., Madirakshi D. Alexander C.Loui,
Approaches to Consumer Image Organization Based
on Semantic Categories, Proc. of SPIE
Vol.6391,63910L-1,2006
19
Related Works

• 3. Approaches to Consumer Image Organization
  Based on Semantic Categories 6
• User Interface

20
Related Works

• Evaluation Method
• Recruit 10 digital camera users, each with more
  200 photos over the past year. (totally about
  2000 photos)
• Random select 40 photos from each user.
• Record users edit (correct wrong clustering)
  operations. (less is better)
• Collect satisfaction feedback.1 for Not very
  satisfied, 7 for very satisfied.

21
Motivations

• With Known face location, hair? body and legs s
  positions can be estimated.
• To do this, good image segment technique is need.
  
• Use eye locations to do face alignment ,
  alleviating the bad effects by the leaning pose.

22
Experiment Example

• Here is an experiment coarsely implementing the
  clustering method motioned at Related works.
• Semi-Automatic Image Annotation Using Event and
  Torso Identification 4

23
Experiment Example

• Framework
• SELAB photo sharing framework.
• Data Set a few personal photos
• Algorithm
• Single algorithm. (by person)
• Combine algorithms. (by time/event, and by person)

24
Experiment Example
25
Experiment Example
26
(No Transcript)
27
Experiment Example

• The experiment result is not very good.
• It did not implement image segment, just estimate
  the body as a rectangle.
• Facial feature and body cloth feature are both
  calculate by histogram.

28
Conclusion and Future Work

• Add image segment.
• Add face alignment.
• Make use of facial feature better. (not know how
  to do yet).

29
References

• 1 Paul V.,Michael J., Robust Real-Time Face
  Detection, International Journal of Computer
  Vision 57(2), 137154, 2004
• 2 Zhao W., Chellappa R., Rosenfeld A. and
  Phillips P., Face recognition A literature
  survey, Technical Report, Maryland University,
  CfAR CAR-TR-948, 2000
• 3Zhang, L., Chen, L., Li, M., and Zhang, H.J.
  (2003) Automated annotation of human faces in
  family albums. Proceedings of ACM Multimedia,
  355-358
• 4Suh, B., and Bederson, B.B. (2004)
  Semi-Automatic Image Annotation Using Event and
  Torso Identification, Tech Report HCIL-2004-15,
  Computer Science Department, University of
  Maryland
• 5 J. Cui, F. Wen, R. Xiao, Y. Tian, and X.
  Tang. EasyAlbumAn interactive photo annotation
  system based on face clustering and re-ranking.
  Proc. CHI 2007. ACM Press, 2007.
• 6 Cathleen D., Madirakshi D. Alexander C.Loui,
  Approaches to Consumer Image Organization Based
  on Semantic Categories, Proc. of SPIE
  Vol.6391,63910L-1,2006