General architecture

1
Generalarchitecture
2
Minimal Subscene

• Working definition The smallest set of objects,
  actors and actions in a dynamic visual scene that
  are relevant to present behavior
• For now we will assume
• Bottom-up objects/actors/actions must be visible
• Top-down relevance to present behavior
  explicitly specified,
• e.g., by specifying a question or task
• Knowledge base the system may supplement
  explicit knowledge
• with long-term acquired knowledge

3
MotivationHumans

• 1) Free examination
• 2) estimate material
• circumstances of family
• 3) give ages of the people
• 4) surmise what family has
• been doing before arrival
• of unexpected visitor
• 5) remember clothes worn by
• the people
• 6) remember position of people
• and objects
• 7) estimate how long the unexpected
• visitor has been away from family

Yarbus, 1967
4
Beobot
5
VisualAttention

• see
• http//iLab.usc.edu

6
ObjectRecognition
Riesenhuber Poggio, Nat Neurosci, 1999 (MIT)
7
Action Recognition
Oztop Arbib, 2001
8

• Start
• Issue question
• Parse question
• Extract keywords
• Expand to related concepts,
• using ontology/KB
• -Fill initial task list

9
Task list

• Working list of currently relevant
  objects/actors/actions
• Initially empty
• Question/task specification provides initial
  filling-in
• As the scene is scanned and objects/actors/actions
  are
• recognized, contents of task list are updated

10
Where attention, saliency map and task map

• Input video stream
• Low-level vision massively parallel extraction
  of simple visual features from video input
• Saliency map localizes conspicuous (potentially
  interesting) objects irrespectively of why they
  are salient
• Task map acts as spatial filter to saliency map
  only locations in the current minimal subscene
  can easily pass through. Other locations need to
  be exceptionally salient to pass through.

11
What memory

• Relates concepts to visual properties
• Bridge between visual and semantic knowledge

12
Generalarchitecture
13
Examples / experiments

• Examine video clips
• For each scene, please write down
• Most salient object
• Most salient action
• Minimal subscene
• Who is doing what to whom

14
Scene 001
15
Scene 001 Attentional Trajectory
16
Scene 002
17
Scene 002 Attentional Trajectory
18
Scene 003
19
Scene 003 Attentional Trajectory
20
Scene 004
21
Scene 004 Attentional Trajectory
22
Scene 005
23
Scene 005 Attentional Trajectory
24
Scene 006
25
Scene 006 Attentional Trajectory
26
Scene 007
27
Scene 007 Attentional Trajectory
28
Scene 008
29
Scene 008 Attentional Trajectory
30
Scene 009
31
Scene 009 Attentional Trajectory
32
Scene 010
33
Scene 010 Attentional Trajectory
34
Scene 011
35
Scene 011 Attentional Trajectory
36
Scene 012
37
Scene 012 Attentional Trajectory
38
Scene 013
39
Scene 013 Attentional Trajectory
40
Scene 014
41
Scene 014 Attentional Trajectory
42
Scene 015
43
Scene 015 Attentional Trajectory
44
Scene 016
45
Scene 016 Attentional Trajectory
46
Scene 017
47
Scene 017 Attentional Trajectory