Image Subject Searching: What We Know and Where We Need to Go

1
Image Subject Searching What We Know and Where
We Need to Go

• Rachael Bradley
• April 2007
• Acknowlegement This presentation builds upon
  research conducted with CLiMB (Computational
  Linguistics for Metadata Building), supported by
  the Mellon Foundation

2
Purpose

• Images are used in design, journalism, education,
  medicine, entertainment and many other areas.
• Increasing numbers of images are available in
  digital format and can be searched online.
• This presentation focuses on image subject
  retrieval in order to generate evaluation
  criteria and future research needs in image
  retrieval.

3
Outline

• Introduction
• Content and Meaning
• User Studies
• Key Characteristics
• Current Technology
• Evaluation
• Future Research
• So What?

4
Introduction

• Introduction
• Context
• Example (1)
• Content and Meaning
• User Studies
• Key Characteristics
• Current Technology
• Evaluation
• Future Research
• So What?

5
Context

• Four types of image attributes
• Biographical
• Birth (Examples Creator, Date, Title)
• Travels (Examples Who has owned it, Cost )
• Exemplified (Examples Painting, .jpg, Sculpture)
• Relationship (Examples sketches-final painting,
  image-critiques)
• Subject
• Of/About (Examples Of a lion/About pride)
• This presentation focuses on subject attributes
• Additional attributes may be as important or more
  important to the end user when searching for an
  image

6
Example (1)
Artemisia Gentileschi Judith Slaying Holofernes,
1612-13 Naples, Museo di Capodimonte
7
Content and Meaning

• Introduction
• Content and Meaning
• Levels of Content
• Image Analysis
• Establishing Meaning
• Types of Meaning
• Example (2)
• Data Sources for Establishing Creators Intended
  Meaning
• Language of Images
• Symbols
• Example (3)
• Data Sources for Establishing Audience
  Interpretation
• Evolution of Audience Interpretation
• User Studies
• Key Characteristics
• Current Technology
• Evaluation
• Future Research
• So What?

8
Levels of Content

• Pre-iconography
• Generic description of objects and events in an
  image
• Knowledge gained by everyday experience is all
  that is needed
• Iconography
• Specific information, conventional matter
• Requires familiarity with a specific culture
• Iconology
• Intrinsic meaning or content
• Requires synthesis of information
  (Pre-iconography Iconography Knowledge of
  culture, artist, etc)
• Each level includes time, space,
  activities/events, and/or objects

9
Panofsky-Shatford Matrix1

• Each level can be divided into factual or
  expressional
• Simplified into Specific Of, Generic Of, and About

1. Shatford, 1986 Armitage and Enser, 1997
10
Image Analysis

• Descriptive Analysis
• Recognition and description of visual elements in
  a work of art
• Shapes, forms, lines and colors
• Formal Analysis
• Recognizing visual relationships between shapes,
  forms, lines and colors
• Images have coherent structure held together and
  ordered by the use of similar shapes, forms and
  colors
• Internal Analysis
• Focus on works inherent aspects (iconographic,
  narrative, symbolic)
• External Analysis
• Analysis of work within a larger context
  (historical, ideological, political,
  psychological, etc)

11
Establishing Meaning

• 1. Traditions of Representation
• Known to the artist and to the actual or intended
  beholders
• Recorded in symbolic dictionaries or recognized
  through repeated use in art
• 2. Pictoral Context and Location
• Visual design in context of the rest of the
  picture
• Location of the artwork in relation to other art
  or the building itself
• 3. Social and political background
• Historical knowledge of events contemporary to
  the painting
• 4. Situation of the artist
• Training, interests, emotional conflicts,
  attitudes, beliefs, economical and psychological
  relations to the patron and to the beholders
• 5. Intentions
• Intentions of the particular artist
• Intentions of most artists in a particular period
• 6. Responses of the beholders
• Response of particular persons in particular
  situations
• Response of normal people in normal situations
• Conscious vs unconscious response

12
Types of Meaning

• Creators intended meaning
• Traditions of representation
• Pictoral context and location
• Social and political background
• Situation of the artist
• Intentions
• Audience Interpretation
• Responses of the beholders
• Pictoral context and location
• Social and political background

13
Example (2)
Caravaggio, 1599
For Discussion 1. Traditions of
Representation 2. Pictoral Context and
Location 3. Social and political background 4.
Situation of the artist 5. Intentions 6. Response
of the beholder
Donotello, 1460
14
Data Sources for Establishing Creators Intended
Meaning

• Text sources
• Associated metadata
• Primary sources
• Diaries
• Announcements
• News articles
• Contracts
• Religious works/ fictional texts
• Symbolic dictionaries
• Histories

• Image Sources
• Original Image
• Related Images
• Preliminary drawings
• Other works by creator
• Images the creator was aware of
• Architectural drawings

15
Language of Images

• Images can never mimic reality
• Limited physical media do not allow for exact
  representation of reality
• Images are information encoded by the creator and
  decoded by the viewer
• To say a drawing is a correct view...means that
  those who understand the notation will derive no
  false information from the drawing (90).
• ...the correct portrait, like the useful map, is
  an end product on a long road through schema and
  correction. It is not a faithful record of a
  visual experience but the faithful construction
  of a relational model (181).
• Style defines the visual possibilities
• Styles, like languages, differ in the sequence
  of articulation and in the number of questions
  they allow the artist to ask (90).

Image Internal Meaning (Style - Artist
Variations) Symbolism Relationships
16
Symbols

• Symbols
• Visual Elements
• Contents Time, Space, Activities/Events, and/or
  Objects
• The symbol makes an informed viewer will think of
  what it symbolizes
• The viewer can specify what it symbolizes
• The symbol does not depict what it symbolizes
• Natural Symbols
• A natural connection exists between the symbol
  and what it symbolizes
• Conventional Symbols
• A tradition exists connecting the symbol and what
  it symbolizes
• Identifying Symbols
• Care in representation
• Central/conspicuous position
• Someone points to the motif
• Presence is out of place

17
Example (3)
Judith I Klimt, 1901
18
Data Sources for Establishing Audience
Interpretation

• Text sources
• Primary sources
• Critiques
• Diaries
• Announcements
• News articles
• Histories
• Accession records
• Changes in use
• Price

• Image Sources
• Original Image
• Personal response
• Related Images
• Derived images
• Later works by artist

19
Evolution of Audience Interpretation

• Interpretation changes over time
• Creation
• Artist Birth to Death
• Quotation
• Subsequent artists emulate images, style and
  technique
• Interpretation
• Frame Classify, organize and interpret life
  experiences (Artist Anecdotes)
• Artist Anecdote Story of artists life and work
  that
• Recontextualization
• Work enters broader cultural/commercial context
• Appropriation, Commercialization, Commodification
• Consumption
• Currency exchanged for some form of artist
  experience
• Interpretation is based on individual and
  cultural factors

20
User Studies

• Introduction
• Content and Meaning
• User Studies
• Image Study Methodology
• Visual Elements
• Pre-Iconographic and Iconographic Terms
• Variation in Search Terms
• Image Constructs
• Iconological Terms
• Image Selection
• User Confidence
• Query Modification
• Browsing
• Additional Findings
• Key Characteristics
• Current Technology
• Evaluation
• Future Research
• So What?

21
Image Study Methodology

• Analyzing email requests to a reference service
• Requests created independently from a retrieval
  system
• Provides some contextual information
• Analyzing query logs from image search engines
• Interface dependent
• Large samples
• No contextual information available
• Possible bias because only queries with
  pre-identified image terms are selected
• Self administered questionnaires describing
  searches
• Contextual information available
• Testing risk
• User studies involving questionnaires, interviews
  and/or observations
• Provide rich information on entire search process
• Focuses on specific groups, possibly transferable
  but not generalizable
• Keister found that requests varied by user types

22
Visual Elements

• Descriptive Analysis
• Color, Line, Shape, Style, Focal Point

In user studies, use of visual elements for
search has been limited.
Used to distinguish between color and black and
white photos
23
Pre-Iconographic and Iconographic Terms

• Pre-Inconographic NonUnique, Noun, Generic
• Iconographic Unique, Proper Noun, Specific
• Refiners, used in many studies, confuse these
  analyses

The level of content description in search terms
is highly variable, likely due to task and
collection differences
24
Variation in Search Terms

• Observations
• "It is not so much that a picture is worth a
  thousand words, for many fewer words can describe
  a still picture for most retrieval purposes. The
  issue has more to do with the fact that those
  words vary from one person to another (p.17).1
• No attempts to technically reduce such a notion
  to thesaurus or subject headings could ever
  encompass the richness of human induction when
  exposed to an image. If a picture is worth a
  thousand words to one viewer, it is worth a
  million words to 1,000 viewers. No individual or
  small group of individuals, no matter how
  professional or rule intensive the approach,
  could ever capture a full panoply of impressions
  invoked by an image (p.7).2
• Results
• In a study of 33,149 queries on Excite search
  engine from 9855 users, most terms only occurred
  once. The most frequently used terms occurred
  less than 10 of the time.3
• In a study of image professionals use of a
  commercial image provider over one month, the top
  term (woman and women) occurred 7 of the time.4
  

In all levels of content, vocabulary varies
greatly
25
Image Constructs

• Similar to Risattis Formal Analysis
• Introduced by Keister from analysis of reference
  requests at NLM
• In an Image Construct Query the terms are used
  as a visual construction rather than simply
  isolated terms.
• Examples Man sitting in the chair with a box on
  his head
• People racing in wheelchairs
• Surgeons standing
• Results
• Image constructs make up 1/3-1/2 of image
  requests1
• Visual constructs made up 83 requests2

Many searchers describe the object relationships
within an image
26
Iconological Terms

• Only one study has specifically examined use of
  Iconological Terms
• 1,749 requests from 7 different image archives1
• Who Mean 1.9 (Standard Deviation 3.8)
• What Mean 1.2 (Standard Deviation 2.3)
• Where Mean 0 (Standard Deviation 0)
• When Mean 0 (Standard Deviation 0)

In the only study to examine iconology, use of
iconological terms was limited.
27
Emotional Response
Five studies reference search by emotional
response
Use of emotional response terms in search has
been limited but has been used more than other
iconological terms
28
Image Selection

• Study of 38 faculty and graduate students of
  American History1
• Topicality most important factor in making
  relevance judgments.
• Most users did not feel comfortable making a
  relevance judgment based on image alone.
• To make a final judgment users used both image
  and text.
• User study of journalism related requests2
• Selection based on (in order)
• 1) Topicality, often based on caption
• 2) Technical and biographical criteria
• 3) Impression to be conveyed (Difficult to convey
  in words)
• User study of journalism related image queries3
• Topicality was a necessary but insufficient
  criterion for relevance...Final selection
  criteria could also be preferential or reactive
  selections were based on personal impressions of
  images being more interesting', funny,
  different, most dramatic'. (p. 107)
• Searchers tended to alternate between viewing
  the textual description and the actual image
  during the selection process (p. 106).

Iconological factors become increasingly
important during selection. Associated text is
necessary during selection.
29
User Confidence

• Image needs were often fuzzy and could not be
  fully explicated. Most often it was however
  possible to name a critical object that should
  appear in the image. The search was then based on
  querying for this object (105).1
• No consistent rational manner for asking for
  pictures(9).2
• The difficulty users often have in translating
  their image needs into verbal or written
  expressions is exemplified by the patron who
  states, I cant tell you what I want, but Ill
  know it when I see it! (46).3
• The selection of search keys for general search
  topics was considered difficult. Journalists
  presumed that the archive contained photos
  relating to topics of interest, but they just had
  not discovered the right way to retrieve them
  (275)4.

Users find it difficult to express image
information needs in words
30
Query Modification

• Study of 33,149 queries on Excite search engine
  from 9855 users1
• 40 of queries were first time queries and 60
  were modified
• Study of image professionals use of a commercial
  image provider
• From 420 image search sessions, the mean number
  of queries per search 2.1
• 48 of queries were modified
• 14 added one or more terms
• 5.6 eliminated one or more terms
• 28 changed one or more terms

Approximately half of all queries are modified.
31
Browsing

• Study of 64 university students online image
  queries1
• Browsing was the primary strategy in satisfying
  20 of information needs (199)
• User study of journalism related requests2
• Browsing was the main search strategy
• "General search topics easily led to multiple
  queries and heavy browsing. Specific needs led
  more likely to just one or two queries and
  browsing sessions (274).
• Trial and error method rather than carefully
  constructed queries
• Study of 1852 journalism related image queries3
• Browsing was the main search strategy after the
  initial query and especially important in
  abstract image needs and collaborative retrieval
  (105).
• Study of image professionals use of a commercial
  image provider4
• Browsing took place in 90 of sessions in Sample
  1 and 81 of sessions in Sample 2.
• An average of 93 thumbnails were browsed per
  session in Sample 1 and 129 in Sample 2 (1354).

Browsing is important during search and selection.
32
Additional Findings

• Query by Example
• Study of 404 queries to Google Answers Visual
  Arts1
• 10 provided examples (Cunningham, Bainbridge and
  Masoodian, 48)
• Study of 1 Month of search logs from a commercial
  image provider2
• Other changes to queries included using terms
  that appear in image captions as additional terms
  ...these represent a change in search strategy to
  a query by example (QBE) form of search, but
  using text associated with the image rather than
  the image itself. (Jorgensen and Jorgensen,
  1355)
• Query for All Existing Material
• User study of journalism related image queries3
• Requests for all existing material on a certain
  topic accounted for nearly tenth of all image
  requests. This type of request has yet to receive
  any attention, even though it might affect
  retrieval measures such as recall (109)

Query by example and specifying all existing
material may be important to some users.
33
Key Characteristics

• Introduction
• Content and Meaning 
• User Studies
• Key Characteristics
• Access Characteristics
• Search and Selection Characteristics
• Current Technology
• Evaluation
• Future Research
• So What?

34
Access Characteristics

• Increasingly complex and variable access points
• Visual Elements
• Rarely used
• Pre-iconography and Iconography
• Often used
• Use likely varies based on collection and tasks
• Level of description can vary by individual,
  collection and task
• Terminology can vary by individual and collection
• Relationships between items is often important
• Iconology
• Rarely used
• Level of description can vary by individual,
  collection and task
• Interpretations vary widely by individual
• Terminology also varies

Although use of Visual Elements and Iconology has
been rarely observed to date, this may be a
result of testing limitations.
35
Search and Selection Characteristics

• Users lack confidence in expressing their image
  needs
• Users often modify queries based on results
• Browsing is a key strategy in image search and
  selection
• Iconology becomes increasingly important during
  selection
• Both the image and associated text are important
  during selection

36
Current Technology

• Introduction
• Content and Meaning 
• User Studies
• Key Characteristics
• Current Technology
• Concept-Based Retrieval
• Content-Based Retrieval
• Social Tagging
• Evaluation
• Future Research
• So What?

37
Concept-Based Retrieval

• Text to Text retrieval
• Text Associated with Images
• Metadata
• Ontologies and Classification Schemes
• Keyword search on associated texts
• Challenges
• Term agreement
• Subjectivity
• Level of agreement

38
Content-Based Retrieval

• Image to image retrieval
• Color
• Possible users medical diagnosis, fashion and
  interior design, art history, journalism and
  advertising
• Overall color or color by location
• Texture
• Coarseness, contrast and directionality
• Shape
• Boundaries or regions
• Face Recognition
• Difficulties disambiguating foreground and
  background?
• Query by Example
• Input an example image or better yet set of
  images (typically selected)
• Model the desired color, texture or shape
  (selected or created)
• Challenges
• 3-Dimensions
• Boundary delineation (foreground and background)
• Variations in angles

39
Social Tagging

• Allows the general public as well as professional
  community to apply text descriptions to images
• Steve.museum
• At The Metropolitan Museum of Art, early studies
  indicate a significant variation between the
  existing collections documentation recording
  artist, date, medium, dimensions, and iconography
  and the words that are supplied by naïve
  viewers, describing the visual elements of an
  image and what it literally depicts.1
• Challenges
• Vocabulary quality
• Interface design

40
Evaluation

• Introduction
• Content and Meaning 
• User Studies
• Key Characteristics
• Current Technology
• Evaluation
• Evaluation Criteria
• Future Research
• So What?

41
Evaluation Criteria

• Does the image retrieval system support
• searching by visual elements (likely using
  content based retrieval methods)?
• query expansion methods for pre-iconographical
  and iconographical terms?
• keyword searching of associated text for
  pre-iconographical, iconographical and
  iconological terms?
• social tagging a large number of users can apply
  iconological and other terms to images?
• browsing both images and associated text?
• query modification?
• query by example?

42
Future Research

• Introduction
• Content and Meaning
• User Studies
• Key Characteristics
• Current Technology
• Evaluation
• Future Research
• Task Types
• Type of Image Need
• Additional Technology
• So What?

43
Task Types

• Attentional Maintain or draw attention
• Retentional Assist with recall
• Explicative Explain visually what would be
  cumbersome to explain verbally
• Descriptive Show what an object looks like
• Expressive Make an impact on a reader
• Constructional Explain how various components
  fit together
• Functional Enable the viewer to follow a process
  or organization
• Logico-mathematical Diagram mathematical
  concepts
• Algorithmic To show possibilities
• Data-display Allow quick comparison and easy
  access to data
• The majority of studies have focused on
  descriptive tasks.

Research Question 1 How do search and selection
strategies change across task types?
44
Type of Image Need

• Image Needs
• A preliminary experimental study indicated that
  keyword searching increased and browsing
  decreased with the specificity of the image
  need1.
• Studies of journalism image search indicate that
  selection strategies differ for single and
  multiple images2.

Research Question 2 How do search and selection
strategies change with image needs?
45
Additional Technology

• Style recognition has not been addressed
• Social tagging has not been fully exploited as a
  mechanism for broadening iconological terminology
• Available technologies have not been combined to
  create an overall image search experience

Research Question 3 Is style recognition
technologically feasible? Research Question 4
How can social tagging be used to improve the
search experience? Research Question 5 How can
concept-based retrieval, content-based retrieval,
and social tagging retrieval be combined
successfully?
46
So What?

• Introduction
• Topicality vs Contents 
• Image Subject Search
• Image Relevance
• Key Characteristics
• Current Technology
• Evaluation Methods
• Future Research Studies
• So What?

47
Take Home Message

• Research is currently being conducted in both
  content-based and context-based image retrieval
  but they are not coordinated
• Variations in terminology and categorization
  across theory, user studies, and technology
  studies make it difficult to build on previous
  knowledge
• Combining theory from various disciplines and
  empirical knowledge in image retrieval will
  provide the best chance of creating a successful
  search and selection experience.