Closing the Loop for ISP using Performance Prediction Dec-05

1
Closing the Loop for ISPusing Performance
PredictionDec-05

• Greg Arnold, Ph.D.
• Gregory.Arnold_at_wpafb.af.mil
• Sensors Directorate
• Air Force Research Laboratory
• AFRL/SNAT, Bldg 620 2241 Avionics Circle
• WPAFB OH 45433-7321 [phone]x4388

2
Trilogy of Thoughts / Goals

• Playground- Urban SASO (Security Stability Ops)
• ISP- context is UAV swarms S-S fusion
• Need multiple sensors
• Confirmatory Sensing and Interrogation
• Anomaly detection backtracking
• Understand the problem
• Active Vision- manipulate the sensor to improve
  perf
• Offline ATR-driven sensing
• Online Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!
• Reasoning in 3D (requires metrics)
• Images are samples from world
• General gt Specific for robustness

Uninhabited Air Vehicle
3
Target RecognitionLevels of Discrimination

• Detection the level at which targets are
  distinguished from non-targets,
• i.e., clutter objects such as trees, rocks, or
  image processing artifacts.
• Classification the level at which target class
  is resolved, e.g., building,
• vehicle, or aircraft.
• Recognition the level at which target subclass
  is determined, e.g., for
• a tracked vehicle, tank, APC, or ADU.
• Identification the level at which the
  model/make of a target is resolved,
• e.g., for a tank M60, M1, or T72.
• Fingerprint the serial number of a particular
  instance of a target,
• i.e. Vinces Caravan vs. Loris Caravan.

4
Target RecognitionLevels of Automation

• Interactive decision aid
• Human and machine work interactively
• Automatic decision aid
• Machine is autonomous from input of data to
  output to human
• Human makes final decision (Human-in-the-loop)
• Autonomous system
• Machine makes the final decision
• Human is NOT in the loop

5
What Does ATR Mean?
6
ISR Goals
Intelligence, Surveillance, and Reconnaissance

• ISR Goals
• No Sanctuary
• Persistent (PISR)
• All Weather
• Day / Night
• All Terrain (city, country,
• forest, desert, ocean)
• Moving Stationary
• Safety !!!

Found Something
Go Get It
Shot
Kill Chain
Hit It!
7
Automated Target Recognition (ATR) Insights

• Information Limited Believe current performance
  is information limited
• Human (Data gtgt Information)
• Pixels/Pupils ratio
• Better SNR, resolution, modalities
• Machine
• False Alarms (Google Search)
• Finer Discrim./Obscuration (gtgt higher resolution)
• 3-D Intuitively understand geometric (3-D)
  information
• UAVs UAVs transform the CID problem!

8
Sensors Directorate Structure

• SN Directorate
• SNA Sensor ATR Technology
• SNJ Light (EO) Sensors
• SNR Radio (RF) Sensors
• SNZ Applications
• SNA Division Mike Bryant, Lori Westerkamp (Ed
  Zelnio)
• SNAA Evaluation
• SNAR Applications
• SNAS Modeling and Signatures
• SNAT Innovative Algorithms
• SNAT Branch Dale Nelson, Rob Williams
• Generation After Next Technologies Algorithms
  (Greg Arnold)
• Tracking and Registration (Devert Wicker)
• Vigilance (Kevin Priddy)

9
ATR Thrust Scope
ID
M60
TRACK
FUNCTIONS
GEOLOCATE
FIND
SENSORS
Algorithms
Signatures
MATURATION PROCESS
Assessment
10
ATR Thrust Approach - Subthrusts
FIND FIX TRACK ID
Operational Target Models/Databases
Characterized Performance High Performance
Computing Operational Databases
ASSESSMENT FOUNDATION
SIGNATURES MODELING
INNOVATIVE ALGORITHMS
Sensor Data Management System (SDMS)
Signature Center
Phenomenology Exploration EM Modeling Synthetic
Data
Challenge Problems Standard Metrics ATR Theory
11
(No Transcript)
12
ATR
We need a generalized pattern recognition
capability that will classify things previously
unseen, actively manage assets, and predict the
intent and actions of combatants.

• ATR for Anticipatory ISR
• Multi-X fusion for PISR/TST
• Dynamic GIG Sensor Management
• ATR Theory for Anticipation

Capability / Difficulty
Spiral Development

• Adaptive ATR
• On-the-fly modeling / reacquisition
• Reasoning with uncertainty
• Adaptive metrics derived from user

• 3-D ATR
• 3-D Imaging for RF Floodlight
• 3-D for urban context
• ATR Theory challenge problem

Goals
13
Assumptions / BeliefsBackground / Framework

• Must Define Problem EOCs
• Whether or not applying model-based vision
• Necessary for testing algorithm capabilities
• Model-Based Vision
• More than just CAD models
• Characterization of the data
• and the system at some level
• If I cant model it, I dont understand it
• Physics-Based Vision
• What can we do before appealing to statistics

14
Operating Conditions (OCs)
OCs Everything that changes the sensor
response. Most OCs have infinite variation
15
Real world variabilityExtended Operating
Conditions (EOCs)
20 Target Types
. . .
Squint Depression Angle
Articulation
6 DOF Pose
Configuration
Obscuration
z
Variants
y
x
16
Discrimination vs. Robustness
Data
Models
Discrimination
Robustness
17
Challenge Space
Using Information More Effectively
Bio-Inspired
Adaptive ATR
ATR Driven Sensing
Multisensor Approaches
More Information
17
18
Trilogy of Thoughts / Goals

• Playground- Urban SASO
• ISP- context is UAV swarms S-S fusion
• Need multiple sensors
• Confirmatory Sensing and Interrogation
• Anomaly detection backtracking
• Understand the problem
• Active Vision- manipulate the sensor to improve
  perf
• Offline ATR-driven sensing
• Online Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!
• Reasoning in 3D (requires metrics)
• Images are samples from world
• General gt Specific for robustness

19
Vertically Integrated Sensor Exploitation for
Generalized Recce Instant Prosecution
(VISEGRIP)
20
Confirmatory Sensing Interrogation
Background
Goal Quantify the accuracy, completeness,
relevance of information with demonstrable
authority. Challenge problem support
counter-WMD Objective Theory algorithm
research to incorporate ATR Theory principles
into Sensor Mgmt infrastructure modified to
implement confirmatory sensing interrogation
Payoff Pattern Recognition discipline that is
more expressive to assure users that source is
authoritative and information is actionable
Theory
Algorithms

• ATR Theory
• Aims to design and predict performance of sensor
  data exploitation systems
• Includes all forms of sensor data exploitation
  i.e. target detection, tracking, recognition, and
  fusion
• Information Theory
• Studies the collection and manipulation of
  information

Query Generation What question to ask Query
Processing When, How, Who to ask Data Fusion
Align redundant information Assess unique or
contradictory information Assimilate valuable
information Evidence Assessment Quantify
accuracy and completeness of assertions Predict a
window of opportunity
21
(No Transcript)
22
Trilogy of Thoughts / Goals

• Playground- Urban SASO
• ISP- context is UAV swarms S-S fusion
• Need multiple sensors
• Confirmatory Sensing and Interrogation
• Anomaly detection backtracking
• Understand the problem
• Active Vision- manipulate the sensor to improve
  perf
• Offline ATR-driven sensing
• Online Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!
• Reasoning in 3D (requires metrics)
• Images are samples from world
• General gt Specific for robustness

23
ATR-Driven Sensing
Cueing, Prioritization for the Human
24
Trilogy of Thoughts / Goals

• Playground- Urban SASO
• ISP- context is UAV swarms S-S fusion
• Need multiple sensors
• Confirmatory Sensing and Interrogation
• Anomaly detection backtracking
• Understand the problem
• Active Vision- manipulate the sensor to improve
  perf
• Offline ATR-driven sensing
• Online Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!
• Reasoning in 3D (requires metrics)
• Images are samples from world
• General gt Specific for robustness

Uninhabited Air Vehicle
25
What is your Objective Function?

• L-p (L-1, L-2, L-infinity)
• Diffusion Distance
• Hausdorff
• Chamfer
• Ali-Silvey
• Earth Movers Distance
• Chi Squared
• Entropy
• Kullback-Liebler
• Mutual Information
• Maximum Likelihood
• Renyi

• Pd (Prob. of Detection)
• Pcc (correct classification)
• Pe (Prob. of Error)
• Pfa (False Alarm)
• Confusion Matrix
• Precision
• Recall
• ROC Curve

26
Clear Box View of ATR
Environment
Detect Track Geolocate ID
Sensor
ATR Decisions
Human Decisions
Target
Target Models Database
Feature Extractor
Target Knowledge
Discriminator
Target Knowledge
Decision Rule
Trained Features
Templates
Models
27
ATR/Fusion Processes
Environment
Detect Track Geolocate ID
Sensor(s)
ATR Decisions
Human Decisions
Target
Target Models Database
28
Performance Model is the Lynchpin

• ATR System is dependent on the Performance Model
• Need performance prediction
• Determine where / when to use sensors
• Estimate effectiveness of sensors for given task
• Sensor Management
• Registration
• Learning

29
If Somebody Asks

• Typical DARPA question
• Is it physically possible to do X?
• Weve invested K and achieved P performance, is
  it worth investing more?
• Examples
• How likely are we to detect a dismount with an
  HSI system with 1m spatial resolution? 1ft?
  1in?
• We spent 40M and achieved 80 of
  perfection.Have we reached the knee in the
  performance curve?
• Organization X says it can build a system to do
  Y. Does this violate physics?

30
Aspects of ATR Theory Objectives
Data Assessment
Design
System Evaluation

• Measure the information content of sensor imagery
• Given a set of data and a MOP, determine
  attainable performance range
• What are the critical design constraints to
  achieve a desired outcome, using this data?

• Estimate exploitation level of available
  information
• Establish feedback loop between ATR designers
  and sensor developers
• What are the critical design constraints to
  achieve a desired outcome at a particular level
  of confidence?
• Information gain from using models and data
  adaptively (learning)

• Determine theoretical upper bound on performance
  of given ATR
• Given an ATR system and a set of data, determine
  how much information can be exploited
• Determine how close a given system comes to
  achieving the optimal bound
• What are the critical design constraints to
  achieve a desired outcome, using this sensor and
  algorithm?
• What was the benefit of adding this (additional
  data/processing)?

31
Problem Simplification

• Having said all that, lets examine a problem for
  which we have some intuition
• 4 or 5 points undergoing rotation, translation,
  and maybe scale and skew
• 1-D, 2-D, and 3-D
• Understand the projection from world to sensor

32
What is Shape?

• Pose and scale invariant, coordinate independent
  characterization of an arrangement of features.
• Residual geometric relationships that remain
  between features after mod-ing out the
  transformation group action.
• Captured by a shape space where each distinct
  configuration of features (up to transformation)
  is represented by a single point.

33
Beyond Invariants

• Invariants
• Projection

Object-Image Relations
34
Generalized Weak Perspective

• Projection model applicable to optical
  images(pinhole camera)
• Approximates full perspective for objects in far
  field
• Affine transformations on 3-space, and in the
  image plane (2-space)
• Denoted GWP

35
Affine Transformations

• In 3D

(Rotate, Scale, Skew Translate) (3-D Point)
36
GWP Projection3D to 2D
Projection
Image
Object
37
Object-Image Relation Motivation
Object 1

• Object 1 is not equivalent to Object 2 (in 3-D)

Object 2

• Image 1 is not equivalent to Image 2 (in 2-D)

38
Object - Image Relations Concept
The relation between objects and
images expressed independent of the camera
parameters and transformation group
(1) Write out the camera equations (geo or
photo) (2) Eliminate the group camera
parameters (3) Recognize the result as a relation
between the object and image invariants.
But pure elimination is VERY difficult even for
polynomials.
39
Weak PerspectiveObject - Image Relations
WEAK PERSPECTIVE

• (Generalized)
• Parallel things remain parallel
• The object size is 1/10 the distance from the
  camera
• (Standard Position Method)

40
Weak Perspective Camera
3-D Model Pixi,yi,zi N-points (3N
DOF) Rotate,Translate,Scale,Shear (12
Constraints) 3N-12 Absolute Invariants
2-D Image qiui,vi N-points (2N
DOF) Rotate,Translate,Scale,Shear (6
Constraints) 2N-6 Absolute Invariants
Camera Model N-points (2N DOF) Union 2-D 3-D
(8 Constraints) 2N-8 relations
Need 5 corresponded points (minimum)
41
3-D Invariants
3-D Model Pixi,yi,zi,1 5-points GL3Translatio
n (12 Constraints) 3N-12 Absolute Invariants
Invariant is a function of the Ratio of
Determinants
A useful standard position is
42
2-D Invariants
2-D Image qiui,vi,1 5-points GL2Translation
(6 Constraints) 2N-6 Absolute Invariants
Invariant is a function of the Ratio of
Determinants
A useful standard position is
43
Object - Image RelationGeneralized Weak
Perspective Camera
(2-D Standard Position) (Camera Transform) (3-D
Standard Position)
Eliminate camera transform parameters
The camera transforms the first 4 object point to
image points, the remaining points satisfy the
object - image relation iff
44
Object-Image Relation Abstraction
All objects that could have produced the image.
Object- Image Relations
All images of the object.
45
GWP Shape Spaces

• The shape spaces in the GWP case are Grassmann
  manifolds
• In 3D
• Gr(n-4,H) or dually the Schubert cycle of
  4-planes in Gr(4,n) which contain (1,.,1)
• Manifold has dimension 3n-12
• In 2D
• Gr(n-3,H) or dually the Schubert cycle of
  3-planes in Gr(3,n) which contain (1,.,1)
• Manifold has dimension 2n-6
• H is the subspace of n-space orthogonal to the
  vector (1,,1)

46
Why

• We associate to our object data, viewed as a
  linear transformation from n-space to 4-space,
  its null space K of dimension n-4.
• Likewise to our image data in 2D we associate
  the null space L of dimension n-3.

47
Global Shape Coordinates

• Better than local invariants
• Come from an isometric embedding of the shape
  space in either Euclidean space or projective
  space.
• Matching expressed in these coordinates will
  gracefully degrade

48
Example in GWP

• 3D, n 5 feature points
• Global shape coordinates are the Plucker
  coordinates (or dual Plucker coordinates) of the
  4xn object data matrix or the 3xn image data
  matrix.

49
Global Object-Image Relations

• General
• If and only If conditions
• Overdetermined set of equations
• GWP
• To match, K must be contained in L (iff)
• This incidence condition can be expressed in
  terms of the global shape coordinates
• For n5, 10 (non-independent) relations that
  look like
• 1234125-12351241245123
• Locally only 2 of the 10 are independent,
  because the locus V of matching pairs (object
  shape, image shape) in the 7 dimensional product
  space XxY has dimension 5, codimension 2.

50
Beyond Object-Image Relations

• Object-Image Relations
• Matching

Object-Image Metrics
51
Why Metrics?

• We intuitively know that if we want to measure
  something we need a metric ATR is no different.
• How far apart are these points?
• The triangle inequality provides efficient match
  searching
• Reliable predictable Unknowns rejection
• Theoretical performance prediction

52
The Triangle Inequality Advantage
u image, x prototype object, xk object from
group
Measure the distance from the image to each shape
object?
Shape Space
53
Using the Triangle Inequality
Equivalent Grouping Decision
u image, x prototype object, xk object from
group
Reject beyond Thresholdnoise
u
Search the group iff the distance to the
prototype is less than the sum of the max
intragroup distance and noise threshold.
Shape Space
X
54
What are Shape and Distance?

• Shape What is left after translation rotation
  are removed (more generally, the group)
• This is the (Partial) Procrustes definition of
  distance
• R represents rotation and T represents
  translation
• Procrustes normalizes the size of the objects

55
New Metrics?

• Any ol metric just wont do
• Invariant to translation rotation of 3-D object
  ( more)
• Invariant to the camera projection (
  discretization)
• This leads to the concept of Object-Image
  Relations (O-IRs)
• Incomplete
• O-IRs are only surrogate metrics
• 0 iff the object and image features are
  consistent
• Object-Image Metrics satisfy all the metric
  properties
• Shape Space is NOT Euclidean!
• There is some evidence that human similarity
  perception is not always metric

56
Metrics on the Shape Spaces

• How to compare objects to images!
• We want a natural shape matching metric
• Invariant to transformations of the 3D or 2D
  data,
• e.g. Rotations, translations, or scale of the
  object or image
• Generalize Weak Perspective
• We use the natural Riemannian metric on the
  Grassmannian to measure distances between object
  shapes and image shapes
• This involves the so called principal angles
  between subspaces and is easily computed from the
  original data matrices via QR decomposition and
  SVD.

57
Object-Image Metrics

• Two ways to compute an object to image distance
• 1. Object Space
• Compute the minimum distance in object space
  from the given object to the set of all objects
  capable of producing the given image
• 2. Image Space
• Compute the minimum distance from the given
  image to the set of all images produced by that
  object

58
Object-Image Metrics Duality
Object Shape Space
Image Shape Space
xu all objects that could have produced the
image.
Object- Image Relations
ux all images of the object.
Duality Theorem
Matching can (in principle) be performed in
either object or image space without loss of
performance !
59
Duality

• Theorem - with suitable normalization
• These metrics are the same!
• In the GWP case this distance turns out to
• be the distance between two subspaces of
• different dimension defined again by using
• principal angles.

60
Image Geodesics

• 2 random images
• Geodesic between them
• Not Linear
• Not the projection of a line
• Not even coplanar
• Geodesics on the this cone have the same length
  as the calculated image distance!

61
Orthographic Shape Space3 Points in 1-D 2-D

• 3 points modulo translation, rotation, reflection
  yields
• 1-D Surface of a 30o cone w/ axis along
  1,1,1
• 2-D Interior of the cone
• 0,0,0 object _at_ origin
• a,a,b objects partition cone
• Scale lines through origin
• Geodesics on the this cone have the same length
  as the calculated image distance!

Rotation on the wrong side aboutthe centroid
rotates the cone (isotropy condition).
62
Object-Image Relations (1)

• Fix an Object
• Set of Images it can produce
• Always circumscribe the cone
• Not conic sections!
• Equilateral triangle produces a slightly smaller
  circle
• Image produces line to origin

63
Object-Image Relations (2)

• Fix an Image
• Set of objects that could produce the given image
• Bent over cone
• Touches along line through origin and image
• Eventually converges to the cone surface
• Large objects must be nearly collinear to produce
  the image

64
Epsilon Balls Noise Analysis

• The set of all shapes of distance 1 from the
  given shape (image)
• The image Gaussian IID noise added to each
  image point location(std. dev. 0.5)

• Still working the analytic model of noise in the
  shape space

65
Intrinsic Separability

• How many different shapes can I hope to identify?
• Shape space as a unit volume
• Epsilon balls defined by metric
• Noise balls generated by Gaussian noise
• 5 points in 3D (Generalized Weak Perspective)
• Epsilon in 0, 0.73 (max radius of ball)
• P(Random Shape in Epsilon Ball)1.37Epsilon
• Example Epsilon0.01
• P(Random Shape in Epsilon Ball)0.014
• Requires as least 73 balls to cover shape space
• (Could be more or less efficient coverings)
• Separability on a gross level

66
Summary

• Integration of Sensing and Processing
• Active Vision
• ATR Theory
• These are all connected overlapping areas
• Provides a rich field of problems and applications

ISP
Active Vision
ATR Theory
67
Open Problems in Inf. Exploitation

• Open Problems ATR Theory
• Classification before Recognition before ID
• Representation
• Modeling free forms spanning discrete-continuous
• Uncertainty in models (i.e. target variability)?
• Cross-sensor phenomenology (registration /
  fusion)
• Correspondence
• Unmixing automated methods for separating
  foreground / background in various scenarios (ID
  before segmentation or pose estimation)
• Intrinsic Separability
• How objects separate in quotient space as a
  function of sensor
• Confusers
• Unknowns How to separate knowns from unknowns
• (Object-Image) Metrics
• Efficient Search of Large Databases
• How to choose the metrics based on the expected
  noise model the type of quotient space derived
  from the choice of metric
• Long Poles
• Probabilities for Fusion / Reasoning with
  Uncertainty
• Recognition By Components (including
  construction/decomposition)
• Non-Gaussian, Nonlinear Analysis (i.e. most
  models and algorithms assume these two
  properties)