Intelligent Story Picturing with computers

1
Intelligent Story Picturing with computers

• Goals
• Picturize a story with suitable images.
• Choose
• real still pictures from database.
• the best representative pictures.

2
Intelligent Story Picturing with computers

• Need for Story Picturing
• News picturing - choose best images to convey
  news.
• Educational choose best images to teach
  children.
• Discovery Channel documentaries.
• Animal Planet stories.
• A picture is worth a thousand words.

3
Intelligent Story Picturing with computers

• Why is it difficult.
• Choice of representative image is subjective.
• No fixed criteria for choosing best.
• Humans select best images based on learnt
  knowledge.

4
Intelligent Story Picturing with computers

• Choose the most representative image

5
Intelligent Story Picturing with computers

• Choice of representative
• Qualititatively
• most perfect image.
• one that visually captures the concept best.
• Quantitatively
• image which is visually most similar to important
  images recursive definition.

6
Intelligent Story Picturing with computers

• Representation in Equations
• is a collection of images,
• is a similarity measure between two images,
• We define rank of an image as the solution
  of the equation

7
Intelligent Story Picturing with computers

• Properties of this equation
• Non-linear, but can be solved recursively.
• There is proof that the recursive procedure
  converges to the principal eigen-vector of S.

8
Intelligent Story Picturing with computers

• Idea referred to as Co-citation in search engine
  research.
• Google, Citeseer use co-citation idea.
• We propose co-citation for image ranking.

9
Intelligent Story Picturing with computers

• Every image cites/votes towards the rank of every
  other image.
• Citation/vote enhances the rank.
• Effect of a vote is determined by rank of voting
  image.
• better rank of voting image greater effect of
  vote

10
Intelligent Story Picturing with computers

• Vote of towards the rank of
• is
• Highly voted images
• will have higher numerical ranks.
• are expected to be important representatives.

11
Intelligent Story Picturing with computers

• Choice of
• Semantic similarity - similarity in content.
• Following measures are useful.
• Visual similarity - IRM measure.
• Annotation based similarity how similar are
  annotations.
• we assume images are annotated.

12
Intelligent Story Picturing with computers

• Integrated Region Matching (IRM) measure
• Extract wavelet features.
• Segment images into a number of regions.
• Calculate overall region based similarity.
• Details can be found in
• James Z. Wang, Jia Li and Gio Wiederhold,
  SIMPLIcity
• Semantics Sensitive Integrated Matching for
  Picture Libraries,
• '' IEEE Transactions on Pattern Analysis and
  Machine
• Intelligence,vol. 23, no. 9, pp. 947-963, 2001

13
Intelligent Story Picturing with computers

• IRM distance measure
• Converted to similarity measure
• conversion
• use percentile values
• fraction of all d values which are
  greater than

14
Intelligent Story Picturing with computers

• Keyword based distance using WORDNET
• If and two keywords
• Case1 if and are same or synonyms
• Case2 - if appears in s list of
  hypernyms at level t
• Case3 if unrelated or antonyms

15
Wordnet - A lexical database for the English
Language Lexical Inherence of Nouns

• Lexicographers impose tree structure on the
  semantic memory of nouns.
• Consider the following oak-gttree-gtplant-gtorganism
  .
• Asymmetric, transitive semantic relation
• Hypernymic relation.

16
Intelligent Story Picturing with computers

• Justification
• Authors tend to use similar words when describing
  ideas.
• dinosaur, animal, beast
• oak, tree, plant
• Expected to be conveying similar ideas when using
  similar words.

17
Intelligent Story Picturing with computers

• Keyword based similarity
• - weight associated with keyword

18
Intelligent Story Picturing with computers

• Algorithm
• Eliminate
• stop words from the text.
• nouns with very high polysemy count.
• verbs, adjectives with high polysemy count
• Keyword weights determined by frequency
• Use as the similarity measure
• Calculate image ranks based on

19
Intelligent Story Picturing with computers

• Some issues
• How to evaluate the system.
• relevance is subjective
• Keyword based distance used is crude.
• NLP people construct lexical chains.
• word weighting is not easy (tf, idf measures
  exist)