Contextual Wisdom

1
Contextual Wisdom

• Social Relations and Correlations for Multimedia
  Event Annotation
• Amit Zunjarwad, Hari Sundaram and Lexing Xie

2
I dont want to spend time annotating ( help!
3
Talk Outline
Observations
Events
Similarity, Co-Occurrence and Trust
Experimentscompare against SVM
Generalization Sum of Partial Observations
Conclusions
4
An Annotation Puzzle
5
Observing Flickr Data
6
Learnability

• The pool statistics reveal a power law
  distribution
• Less than 11 of the tags have more than 10
  photos
• There are not enough instances to learn most of
  the concepts!
• The global flickr pool is interesting

7
Learnability
8
Learnability

• The pool statistics reveal a power law
  distribution
• Less than 11 of the photos have more than 10
  instances
• There are not enough instances to learn most of
  the concepts!
• The global flickr pool is interesting
• Most of the tags have over 100 instances
• Photos reveal very high visual diversity
• The Power law is a fundamental property of online
  networks cannot be wished away.

9
Scalability

• Singapore
• People
• Walking
• Orchard rd.
• After MRT
• Experimenting
• Walking
• Day
• Outdoor..

10
The Role of context

• The assumption of consensual semantics
• Search for yamagata

11
What if the answer didnt completely lie in the
pixels?
12
Events

• What are they?

13
Defining Events

• An event refers to a real-world occurrence,
  spread over space and time.
• Observations form event meta data Westermann /
  Jain 2007
• Images / text / sounds describe events

14
Context

• Event context refers to the set of attributes
  that help in understanding the semantics
• Images / Who / Where / When / What / Why / How
• Context is always application dependent
• Ubiquitous computing community location,
  identity and time are main considerations

Mani and Sundaram 2007
15
Four Problems

• Event archival events involve people, places
  and artifacts
• Exploit different forms of knowledge
• (Global) Similarity media, events, people.
• (Personal) Co-occurrence what are the joint
  statistics of occurrence?
• (Social) Trust determining whom to trust for
  effective annotation?

16
Similarity

• Global, Systemic knowledge

17
Event similarity

• A bottom up approach
• Edge, color and texture histograms for images.
• Rely on ConceptNet for text tags
• Why ConceptNet and not WordNet?
• Expands on pure lexical terms, to compound terms
  buy food
• Expands on number of relations from three to
  twenty
• Contains practical knowledge we can infer that
  a student is near a library.

18
A base similarity measure

• ConceptNet provides three functions
• GetContext(node) the neighborhood of the concept
  book includes knowledge, library
• GetAnalogousConcepts(node) concepts that share
  incoming relations analogous concepts for the
  concept people are human, person, man
• FindPathsBetweenNodes(node1,node2) returns a
  set of paths.
• Our similarity measure is built using these
  functions.

19
Concept similarity

• The similarity between two concepts (e,f) is
  defined as follows
• We current use a uniform weighting on all three
  as the composite measure

context
analogous
path based
20
Computing similarity between sets

• The distance between two concept sets is a
  modified Haussdorf similarity.

B
A
21
Facet similarity (4w)

• Similarity between facets are computed using a
  weighted sum of frequency and the concept
  similarity measure
• Time distance is based on text tags, not actual
  time data allows for temporal descriptions as
  summer, holidays etc.
• Only frequency is used for who facet.

22
Image facet similarity

• Color, texture and edges are computed
• 166 bin HSV color histogram
• 71 bin edge histogram
• 3 texture features
• Euclidean distance on the composite feature
  vector.
• The distance between two events is then a
  weighted sum of distances across all event
  facets.

23
The global similarity matrix Ms
24
Co-occurrence

• Personal, statistical knowledge

25
Statistics are computed per person

• The concept co-occurrences are just frequency
  counts.
• (i fun , j new york) then the index (i,j)
  contains the number of occurrences of this tuple.
• Notes
• Each concept is given a globally unique index
• Co-occurrence matrixes are locally compact
• Each user k, has a co-occurrence matrix Mck
  associated with the user.

26
Trust

• People we like

27
Activity based trust

• Narrow understanding of trust
• a priori value is important
• Computing trust
• Compute event-event similarity
• Trust propagation
• Biased PageRank algorithm
• Trust vectors are row normalized

28
The recommendation algorithm
29
A review of what we know

• The framework is event centric
• We know
• How to combine the three?

similarity
co-occurrence
trust vectors
social
personal
global
30
details

• Compute the social network trust vector (t) for
  the current user.
• Compute the trusted, global co-occurrence matrix,
  for all tuples.
• Iterate

31
Experiments
32
Details

• Developed and event based archival system
• 8 graduate students
• 58 events, 250 images, over two weeks
• SVM baseline comparison
• Two cases
• Uniform trust (global)
• Personal trust

33
SVM training

• Training is difficult very small pool.
• Modified bagging strategy
• Train five symmetric classifiers
• Pick one which maximizes the F-score

34
Uniform trust

• Global Case
• 31 classifiers (who8, when 6, where 10, what
  7)
• Minimum number of images 10
• Tested on 50 images (why?)

35
Personal Network

• Trained classifiers per person
• Very small pool
• Min images 5
• 28 classifiers (who9, when 4, where 6, what
  9)

36
Positive examples
SVM sky diving
Social Network based method fun
37
The Sum of Partial Observations

• Beyond web 2.0 hype

38
Experiential fragments

• Which media object summarizes my trip to
  Singapore?

39
A reconsideration of a traditional idea
40
The Creation of participatory knowledge
41
Conclusions
42
summary

• An event based annotation system
• Media are event meta-data
• Issues learnability, scalability, context
• Employ three kinds of knowledge
• Global conceptnet, image similarity
• Personal statistical co-occurrence
• Social trust
• Recommendations
• Employ iterative schemes (HITS / PageRank)
• Results
• Outperform SVM in small pools

43
Conclusions

• Power law tag distribution
• Data pool will remain small for most tags
• Fundamental issue
• Participatory knowledge is powerful trust
  within context is important issue.
• Future work
• Careful math analysis of coupling equations
• Event structure / relationships need to be
  incorporated
• Multi-source (email / Calendar / IM / blogs)
  integration.

44
Thanks!

• Esp. Dick Bulterman, Mohan