BLIND DETECTION OF PHOTOMONTAGE USING HIGHER ORDER STATISTICS

1
BLIND DETECTION OF PHOTOMONTAGE USING HIGHER
ORDER STATISTICS
By Tian-Tsong Ng, Shih-Fu Chang And Qibin Sun

• Presented By
• Barak Hurwitz
• Advanced IP color seminar
• With Dr. Hagit Hel-Or
• 2006

2
Outline

• Review
• Problem and Motivation
• Image Forgery Detection Approaches
• Bicoherence
• Definitions
• Bicoherence for splicing detection
• Enhancements
• Bipolar Perturbation Hypothesis
• Bicoherence of bipolar signal
• Texture Decomposition
• Conclusions

3
Problem Motivation

• General problem Image Forgery Detection
• Image Forgery Images with manipulated or fake
  content

Image Forgery Detector
Real
Fake
4
Problem Motivation-cont
Input
Output
Machine
http//www.ee.columbia.edu/ln/dvmm/trustfoto/
5
Problem Motivation-cont
How much can we trust digital images?
6
Problem Motivation-cont
How much can we trust ourself?

• Ron Muecks Australian sculpture

7
Problem Motivation-cont

• Famous examples

March 2003A Iraq war news photograph on LA Times
front page was found to be a photomontage
Feb 2004A photomontage showing John Kerry and
Jane Fonda together was circulated on the
Internet Adobe Photoshop (5 million registered
users)
Pictures From http//www.camerairaq.com/faked_phot
os/
8
Problem Motivation-cont
Home made examples
9
Photomontage and Spliced Image

• Specific problem Image Splicing Detection
• Photomontage A paste-up produced by sticking
  together photographic images, possibly followed
  by post-processing (e.g. edge softening and
  adding noise).
• Spliced Image (see figures) Splicing of image
  fragments without post-processing. A simplest
  form of photomontage.
• Why interested in detecting image splicing?
• Image splicing is a basic and essential operation
  in the creation of photomontage
• Therefore, a comprehensive solution for
  photomontage detection includes detection of
  post-processing operations and intelligent
  techniques for detecting internal scene
  inconsistencies

10
Image Forgery Detection Approaches

• Active approach
• Fragile/Semi Fragile Digital Watermarking
  Inserting digital watermark at the source side
  and verifying the mark integrity at the detection
  side.
• Authentication Signature Extracting image
  features for generating authentication signature
  at the source side and verifying the image
  integrity by signature comparison at the receiver
  side.
• Effective when there is
• A secure trustworthy camera
• A secure digital watermarking algorithm
• A widely accepted watermarking standard
• Passive and blind approach
• Without any prior information (e.g. digital
  watermark or authentication signature), verifying
  whether an image is authentic or fake.
• Advantages No need for watermark embedding or
  signature generation at the source side

11
What are the qualities of authentic images?

• Image Authenticity
• Natural-imaging Quality (NIQ)
• Entailed by natural imaging process with real
  imaging devices, e.g. camera and scanner
• Effects from optical low-pass, sensor noise, lens
  distortion, demosicking, nonlinear
  transformation.
• Natural-scene Quality (NSQ)
• Entailed by physical light transport in 3D
  real-world scene with real-world objects Results
  are real-looking texture, right shadow, right
  perspective and shading, etc.
• Examples
• Computer graphics and photomontages lack in both
  qualities.

NO NIQ
Computer Graphics
NO NSQ
Photomontage
12
Goal Image Splicing Detection using
Natural-imaging Quality (NIQ)

• NIQ Authentic images comes directly from camera
  and have low-pass property due to camera optical
  anti-aliasing low-pass??
• Deviations from NIQ Image splicing introduces
  arbitrarily rough edges/discontinuities in image
  signal??
• Characterize NIQ using bicoherence

13
Outline

• Review
• Problem and Motivation
• Image Forgery Detection Approaches
• Bicoherence
• Definitions
• Bicoherence for splicing detection
• Enhancements
• Bipolar Perturbation Hypothesis
• Bicoherence of bipolar signal
• Texture Decomposition
• Conclusions

14
Bicoherence

• Definition The bicoherence of a signal x(t)
  with its Fourier transform being X(?) is given
  by

Expectancy
From FFT of X(t)
Autocorrelation
Normalized
where f represents the complex conjugate and the
circle represents convolution. For a real
function, f f.
15
Prior work using BIC to detect speech splicing

• Farid99
• Assuming that speech signal is originally low in
  QPC
• Nonlinearity associated with splicing causes
  increase of BIC magnitude
• BIC features used for detecting the increase of
  QPC in spliced human speech signal are
• ??average BIC magnitude
• ??Variance of the BIC phase histogram

Human speech signal is originally weak in higher
order correlation, reflecting on the low value of
the bicoherence magnitude feature and a rather
randomly distributed bicoherence phase
16
Prior work using BIC to detect speech splicing-
cont
Farid99 splicing is a non-linear operation
which comprises a linear-quadratic operation that
could induce QPC.

• Definition Quadratic Phase Coupling (from HOSA)
• Phase coupling occurs due to nonlinear
  interactions between harmonic components.
• 3 harmonics with frequencies Fk and phases
  , k 1,2,3, are said to be quadratically phase
  coupled if

QPC is a phenomena where 3 harmonics have the
following frequency and phase respectively. It
can be seen that the relationship of the
frequency and phase are the same.
17
Challenges of Applying BIC to 2D images

• Krieger97?? empirical observation
• Due to the predominant image edge features,
  natural images exhibit concentration of energy in
  2-D BIS at regions with frequencies corresponding
  to
• Fackrell95b, Zhou96gt BIS energy implies QPC
• gt Krieger97s empirical observation predicts
  that image splicing detection using bicoherence
  magnitude and phase features would face a
  significant level of noise.

18
Extraction of BIC Features from image
To reduce noise effect, phase histogram is
obtained from the BIC components with magnitude
exceeding a threshold
19
Experiment with Plain BIC features

• Compute the plain BIC features and look at the
  feature distribution for our data set
• They found that the distribution for magnitude
  and phase are greatly overlapped!!!

Proposed Solutions?? To model the image-edge
effect on BIC?? To capture splicing-invariant
features
20
Outline

• Review
• Problem and Motivation
• Image Forgery Detection Approaches
• Bicoherence
• Definitions
• Bicoherence for splicing detection
• Enhancements
• Bipolar Perturbation Hypothesis
• Bicoherence of bipolar signal
• Texture Decomposition
• Conclusions

More
21
Theoretical Basis for Bicoherence for Image
Splicing Detection
Image splicing introduces rough edges at splicing
interface
Image splicing can be considered as a bipolar
perturbation on an authentic signal.
22
Theoretical Basis for Bicoherence for Image
Splicing Detection

• Theoretical analysis shows that bipolar
  perturbation of a signal results in an increase
  in BIC magnitude and phase concentration at 90
  degrees .

An example of BIC phase histogram
23
Bipolar perturbation model

• Original image signal is relatively smooth due to
  the low-pass anti-aliasing operation in camera or
  scanner.
• Spliced image signal can have arbitrary
  discontinuity

Definition (Bipolar signal)
24
Bicoherence of Bipolar Signal

• Results Bicoherence phase of bipolar signal is
  concentrated at 90

Resulting in 90 phase bias
When there is phase coherency, bicoherence
magnitude is close to unity
25
Bipolar Perturbation Effect on Phase Feature

• Bipolar perturbation

Numerator of the perturbed signal bicoherence
26
Empirical Support for Bipolar Perturbation Model

• Spliced averaged phase histogram
• Authentic averaged phase histogram

27
Phase Histogram
Typical examples of bicoherence phase histogram
from spliced images
Examine phase concentration at 90
Strong at 90
Near at 90
Non at 90
28
Modeling Image-edge Effect on BIC

• BIC depends on the image characteristics??
• Krieger97 shows image edges result in high BIC
  energy.??
• Classifier needs to consider image types??
• Categorize images according to region interface
  types
• textured-textured, textured-smooth and
  smooth-smooth??
• Experiment shows that BIC features have different
  separability for different interface types??
• They use canny edge pixel percentage (one of many
  ways) for determining interface types.

The scatter plot for BIC phase feature is similar!
29
Splicing-invariant Features Authentic
Counterpart (AC)

• AC is similar to the spliced image except that it
  is authentic

30
Texture Decomposition

• VeseOsher02 ??
• An image f is decomposed as u v ??
• u gt structure component
• v gt fine-texture component ??

u
v
31
Splicing Detection using Texture Decomposition

• They approximate the authentic counterpart (AC)
  using the structure component??
• Assumptions
• Structure component less contaminated by splicing
  (captures the splicing invariant features) ??
• Splicing artifacts (bipolar perturbation) are
  captured by the fine-texture component??
• 2 approaches for detecting image splicing??
• Detect the presence of splicing artifacts in the
  fine-texture component
• Does not work well because the value of BIC
  features of the fine-texture component vary in a
  very narrow range, hence not discriminative??
• Detect the absence of splicing artifacts in the
  structure component. (They adopted this
  technique)

32
Computing Prediction Residue Features
Structure-Texture Decomposition
Texture
Extract Plain BIC Features
Structure
Fs
Extract Plain BIC Features
F1- cFs
Prediction Residue Feature
F1
33
Columbia Image Splicing Detection Evaluation
Dataset

• 933 authentic and 912 spliced image blocks
  (128x128 pixels)
• Extracted from
• contributed by photographers- assume to be
  authentic
• Splicing is done by cut-and-paste of
  arbitrary-shaped objects and also
  vertical/horizontal strip.

Download URL http//www.ee.columbia.edu/dvmm/new
Downloads.htm
34
Authentic Category
35
Spliced Category
36
Performance Metrics

• RBF kernel Support Vector Machine (SVM) on 933
  Authentic and 912 Spliced images, 10-fold
  cross-validation to ensure no over-fitting.??
• 3 evaluation metrics over 100 runs of
  classification

37
Additional Results on Bicoherence Features
ISCAS04

• Features evaluated ??
• BIC magnitude feature??
• BIC phase feature??
• BIC magnitude predication residue??
• BIC phase prediction residue??
• Edge pixel percentage

38
ROC curve for image splicing detection
Receiver Operating Characteristic (ROC)
39
Conclusions

• A bipolar perturbation model for explaining the
  effectiveness of bicoherence in detecting image
  splicing
• Plain BIC features do not perform well
• Need to incorporate image characteristics and the
  splicing invariant component with respect to
  BIC??
• Improve the classification accuracy from 62 to
  72
• Still a large margin for innovation and
  improvement??
• Possible directions ????
• Combine with computer-vision analysis (dealing
  with scene and illumination consistency)??
• Other issues explore discriminative features
  other than BIC.

40

• THE END

41
Test their algorithm
http//apollo.ee.columbia.edu/trustfoto/trustfoto/
natcgV4.html
42
Test their algorithm
43
Test their algorithm- cont
44
More Definitions

• More Definitions

45
Skewness
Multivariate normality  is the assumption that
all variables and all combinations of the
variables are normally distributed.
when a distribution is normal both skewness and
kurtosis are zero. Kurtosis is related to the
peakedness of a distribution, either too peaked
or too flat. skewness is related to the symmetry
of the distribution, the location of the mean of
the distribution

• Getting asymmetric

46
A Brief Introduction to Higher Order Statistics
First Order Statistics
As a measure of the similarity of two time
series, one constructs the correlation, given by
In the general form shown above, C is called the
cross-correlation. It's Fourier transform S is
the cross spectral density and represents the
power common to the two input channels.
When x y, then C is referred to as the
auto-correlation, a measure of the
self-similarity, or periodicity, of a time
series. The Fourier transform of the
auto-correlation is the power spectral density,
S, which is the power present in a given channel
as a function of frequency.
One denotes the coherence of two signals by
calculating the percentage of the total power in
the two channels which is in the correlated
signal. The coherence is given by
47
A Brief Introduction to Higher Order Statistics-
cont

• Therefore using first order statistics, one
  learns about
• the power in a given channel,
• the correlated power for two channels, and
• the phase coherence of two channels.

Second Order Statistics
In higher orders, the cumulant is the general
term for the quantity that in first order
statistics is known as the correlation. The
second-order cumulant, known as the bicumulant,
is given by
The Fourier transform of the bicoherence is the
bispectrum, the second-order counterpart of the
power spectrum.
The cumulants at the origin, also known as
zero-lag cumulants, are useful quantities in
characterizing the input signal. The first four
zero-lag cumulants are

• Mean                   
• Variance                     
• Skewness              0 if Symmetric. But
  Skewness 0 does not prove symmetry.
• Kurtosis              0 if Gaussian. But
  Kurtosis 0 does not prove gaussianity

http//www.ligo-wa.caltech.edu/keithr/detcamp04/c
amp/BicoMonIntro.html
48
A Brief Introduction to Higher Order Statistics-
cont
The higher order cumulants have the advantage
that beyond the origin they are zero for i.i.d.
processes, like gaussian noise
If we normalized the bispectral density by the
power of the contributing channels, we arrive at
the bicoherence, or the auto-bicoherence if x
y.
The bicoherence yields the phase coherence of
coupled frequency processes. For the
auto-bicoherence, the frequency symmetries limit
the unique area to the region shown below
49
Hanning/Hamming window
Also known as the raised cosine window
The Hanning window for N points is defined as
where -N/2 lt i lt N/2
The Hamming window is
50
Properties of BIC

• For signals of low order moments like Gaussian ,
  BIC magnitude 0
• Fackrell 95b Quadratic Phase Coupling (QPC) vs.
  BIC

• A simultaneous occurrence of frequency harmonics
  at and (Quadratic Frequency Coupling- QFC)
• with respective phase being
• At with QPC ,
• BIC phase 0 and BIC magnitude ratio of QPC
  energy

51
Applications of Bicoherence (BIC) and Bispectrum
(BIS)

• BIC/BIS detects QPC/QFC as one form of
  non-linearity??
• Bullock97 Studying non-linearity in
  intracranial EEG signal??
• KimPowers79 Application in plasma physics??
• SatoSasaki77 Application in manufacturing??
• Hasselman63 Application in oceanography??
• Fackrell95a Detecting fatigue crack in
  structure through vibration
• BIC/BIS detect signal non- gaussianity??
• Santos02 Detecting non- gaussianity in the
  cosmic microwave background data

52
Bicoherence Features

• Phase feature
• Magnitude feature

53
Effect of Bipolar Perturbation on Magnitude
Feature (cont.)
54
Illustration of a general splicing model
55
Why BIC is Good for Splicing Detection?
A linear-quadratic of a signal of two harmonics
is illustrated below
Phase
Frequency
56
Hypothesis I (cont.)
Quadratic linear Operation
Argument Farid99 Quadratic-linear operation
gives rise to QPC and a nonlinear function, in
Taylor expansion, contains quadratic-linear term.
As splicing is a nonlinear operation, hence
bicoherence is good at detecting splicing.
Problems 1.No detailed analysis was given.
2.The quadratic-linear operation here is a
point-wise operation, it is not clear how
splicing can be related to a point-wise
operation?
57
More On Bicoherence
58
Definitions

• (1) Signal f(x)
• (2) Power spectrum P(w) F(w)F(w)
• (3) Bispectrum B(w1,w2) F(w1) F(w2) F(w1w2)
  with F(w) the Fourier transform and F(w) its
  complex conjugate.

A fractal signal f(x)
Power spectrum P(w)
Bispectrum B(w1,w2)
http//www.cs.dartmouth.edu/farid/research/blind.h
tml
59
Bicoherence (From Wikipedia)

• Bicoherence
• Bicoherence is a squared normalized version of
  the bispectrum .
• The bicoherence takes values bounded between 0
  and 1
• Convenient measure for quantifying the extent of
  phase coupling in a signal.
• Also known as bi-spectral coherency.
• Bispectrum
• The bispectrum is a statistic used to search for
  nonlinear interactions.
• The Fourier transform of the 2nd -order cumlant
  autocorrelation power spectrum
• The Fourier transform 3rd -order cumlant
  bispectrum.
• They fall in the category of Higher Order Spectra
  (Statistics), or Polyspectra
• They provide supplementary information to the
  power spectrum.

The prefix bi- in bispectrum and bicoherence
refers not to two time series xt, yt but rather
to two frequencies of a single signal. Setting x
y in the coherence yields the bicoherence.
60
Bicoherence (From Wikipedia)

• The bispectrum is the easiest to compute, and
  hence the most popular.
• Bicoherence vs. coherence
• coherence analysis is an extensively used method
  to study the correlations in frequency domain,
  between two simultaneously measured signals.
• The difference with measuring coherence is the
  need for both input and output measurements by
  estimating 2 auto-spectra and 1 cross spectrum.
• On the other hand, bicoherence is an
  auto-quantity, i.e. it can be computed from a
  single signal. The coherence function provides a
  quantification of deviations from linearity in
  the system which lies between the input and
  output measurement sensors.
• The bicoherence measures the proportion of the
  signal energy at any bifrequency that is
  quadratically phase coupled (QPC)
• Bispectrum and bicoherence may be applied to the
  case of non-linear interactions of a continuous
  spectrum of propagating waves in one dimension

Application Bicoherence measurements have been
carried out for EEG signals monitoring in sleep,
Wakefulness and epileptic seizure.
61
Bicoherence cont
Bispectrum is a third-order moment spectra of a
signal, say x(t). Let, X(w) be the Fourier
transform of x(t), bispectrum is defined as
below
Bicoherence is the normalized bi-spectrum, The
mathematical form for bicoherence is given by
The properties of bicoherence are

• Bicoherence is zero for a Gaussian process.
• The magnitude of bicoherence at a bifrequency
  (w1,w2) is a good estimator of QPC at that
  bifrequency.
• The phase of bicoherence at a bi-frequency
  (w1,w2) would be zero, when there is complete QPC
  at that bi-frequency.

62
Bicoherence
63
Bicoherence
Go Back