Sensor Fingerprinting: Challenges and Trends

1
Sensor Fingerprinting Challenges and Trends

• Walter J. Scheirer
• wjs3_at_vast.uccs.edu
• VAST Lab
• University of Colorado at Colorado Springs

2
What is Sensor Fingerprinting?

• Identifying the data acquisition device that
  generated a given image
• Device Class Identification
• Specific Device Identification

3
Why would we want to do this?

• Digital Image Forensics
• Over 90 of our of computer forensics cases
  involve child pornography
• - unnamed detective from the Pennsylvania State
  Police
• The Burden of digital photography
• Is an image real, or has it been rendered by a
  computer?
• Is an image an original, or has it been altered
  by digital image processing?

?
?
?
?
?
?
?
?
4
Ashcroft v. Free Speech Coalition

• Supreme Court Ruling
• Struck down two provisions of the Child
  Pornography Prevention Act of 1996
• Synthetic images depicting what would be
  considered illegal in a genuine image are legal,
  and protected by the first amendment
• How does a forensic investigator determine what
  is real, and what is synthetic?
• http//www.law.cornell.edu/supct/html/00-795.ZO.h
  tml

5
Digital Image Forensics

• A three stage process
• Image Source Identification (Sensor
  Fingerprinting)
• Discrimination of Synthetic Images from Real
  Images
• Image Forgery Detection

6
Digital Image Forensics

• If an image is indeed real
• What can we determine about the nature of the
  source?
• Can we pinpoint the exact model of the device?
• Can we prove beyond a reasonable doubt that a
  certain device produced the image?
• Evidence!

7
Vision of the Unseen

• Sensor fingerprinting relies on underlying
  characteristics of sensors and image processing
  techniques used by digital cameras
• Artifacts, distortions, statistical properties
• These characteristics are nearly always
  imperceptible to the human eye

8
Image Acquisition Pipeline
Lens
Filter(s)
Sensor
Camera Processing
Color Filter Array
9
Sensors

• CCD CMOS
• Tremendous variation between individual sensors
• Defects caused by manufacturing or use

10
Color Filter Arrays

• RGB Bayer Pattern
• CMYK

11
Sensor Fingerprinting Source Model Identification

• Digital Cameras
• Lens
• Size of sensor
• Choice of CFA
• Demosaicing algorithm
• Color processing algorithm
• Many manufacturers use the same components

12
Image Features

• Revisiting some of the techniques of
  steganalysis Kharrazi et al. 2004
• Defines a set of 34 features inspired by
  universal steganlaysis techniques
• Color features, wavelet coefficient statistics,
  image quality metrics

M. Kharrazi, H. T. Sencar, and N. Memon, Blind
Source Camera Identi?cation, Proc. of IEEE ICIP
(2004)
13
Image Color Features

• Average pixel value
• Gray world assumption average values in RGB
  channels of an image should average to gray
• RGB pairs correlation
• Variance in correlation of RG, RB, and GB pairs
  across sensor manufacturers
• Neighbor distribution center of mass
• Calculate the number of pixel neighbors for each
  pixel value
• Distribution gives an indication of the
  sensitivity of the camera to different intensity
  levels

14
Image Color Features

• RGB Pairs Energy Ratio
• Wavelet Domain Statistics
• Decompose each color band using separable quadric
  mirror filters
• Calculate the mean for each sub-band

G2
G2
B2
E2
E1
E1
R2
R2
B2
H. Farid and S. Lyu, Detecting hidden messages
using higher-order statistics and support vector
machines, 5th International Workshop on
Information Hiding. 2002
15
Image Quality Metrics

• Pixel difference based measures
• Mean square error, mean absolute error, modified
  infinity norm
• Correlation based measures
• Normalized cross correlation, Czekanowski
  correlation
• Spectral distance based measure
• Spectral phase and magnitude errors

I. Avcibas, N. Memon, and B. Sankur,
Steganalysis using image quality metrics. IEEE
transactions on Image Processing, January 2003.
16
Classification

• Construct feature vector out of the calculated
  features for a given image
• Build training sets for each class of camera
• Use machine learning for classification of images
  with unknown sources

Camera 1
Camera 2
SVM Hyperplane
17
Results of Kharrazi et al.

• Standard two class SVM

Serious Quantization effects observed, driving up
classification performance
Images recompressed to a JPEG quality level of
75
18
Results of Kharrazi et al.

• Multi-class SVM

See also M.J. Tsai and G.H. Wu, Using Image
Features to Identify Camera Sources, Proc. of
IEEE ICASSP (2006).
19
CFA and Demosaicing Artifacts

• Choice of CFA
• Demosaicing

Bayer Pattern (RGB)
CMYK
20
Image Features from CFA - Cell Phone Cameras

• Çeliktutan et al. 2005
• Motivation proprietary interpolation algorithms
  leave correlations across adjacent bit planes of
  the images
• Binary Similarity Measures (also used in
  steganalysis)
• Define a stencil function

1 if xc 0 xn 0
2 if xc 0 xn 1
n c
?
(k,b)
3 if xc 1 xn 0
4 if xc 1 xn 1
O. Celiktutan, I. Avcibas, B. Sankur and N.
Memon, Source Cell-Phone Identi?cation, Proc. of
ADCOM (2005).
21
Image Features - Cell Phone Cameras

• Sum over its four neighbors, and over
  all MxN pixels
• Normalize the agreement scores
• Define binary Kullback Leibler distance (feature
  1)

n c
(k,b)
?
?(k,b) / ? (k,b)
b k
p
k
7 n
p
4
m1 - ? p log
7 n
8 n
p
n 1
22
Image Features - Cell Phone Cameras

• Define neighborhood weighting mask (feature 2)
• S ? xi 2i

7
i0
Score Function
Weighting Pattern of the Neighbors
511
m2 ? S - S
7 n
8 n
Absolute Histogram Difference
n0
23
Image Features - Cell Phone Cameras

• Czenakowski distance - feature 3
• Classification
• KNN and multi-class SVM

Scatter plot of three features for three
different cameras from Çeliktutan et al. 2005
24
Results of Çeliktutan et al. 2005

• KNN classification

Overall performance 98.7
Overall performance 81.3
25
Results of Çeliktutan et al. 2005

• Multi-class SVM classification
• Overall accuracy 62.3
• Random guessing 11.1

26
The Expectation/Maximization Algorithm

• Popescu 2005
• Motivating assumption rows and columns of
  interpolated images are likely to be correlated
  with their neighbors
• Two steps
• Expectation the probability of each sample
  belonging to each model is estimated
• Maximization the specific form of the
  correlations between samples is estimated
• Both steps are iterated till convergence

A. Popescu, Statistical Tools for Digital Image
Forensics, Ph.D. Dissertation, Department of
Computer Science, Darthmouth College (2005).
27
The Expectation/Maximization Algorithm

• Assume that each sample belongs to one of two
  models
• M1, if a sample is linearly correlated with its
  neighbors
• M2, if a sample is not correlated with its
  neighbors

N
f(x,y) ? ?u,v f(xu, yv) n(x,y)
u,v-N
28
The Expectation/Maximization Algorithm

• Expectation Step
• The probability of each sample belonging to M1 is
  estimated using Bayes rule

Prf(x,y) ? M1 f(x,y)
Pr f(x,y) f(x,y) ? M1 Prf(x,y) ? M1
2 i1
?
Pr f(x,y) f(x,y) ? Mi Pr f(x,y) ? Mi
29
The Expectation/Maximization Algorithm

• The probability of observing a sample knowing it
  was generated by a model M1

Prf(x,y) f(x,y) ? M1
)

(

2
N
1
1
f(x,y) - ? ?u,v f(xu, yv)
exp
-
2?2
? 2?
u,v-N
30
The Expectation/Maximization Algorithm

• The Maximization step
• Generate a new estimate of ? using weighted least
  squares
• Execute both steps until a stable ? is achieved.
  Final result maximizes the likelihood of observed
  samples

)
(
2
N
E(?) ? w(x,y) f(x,y) - ? ?u,vf(xu, yv)
x,y
u,v-N
31
Demosaicing Algorithms
p
Image
F(p)
bilinear
bi-cubic
smooth line
Popescu 2005
32
Demosaicing Algorithms
Image
p
F(p)
Median 3x3
Median 5x5
gradiant
Popescu 2005
33
Demoasicing Algorithms
Image
p
F(p)
adaptive color plane
Variable number of gradiants
No CFA interpolation
Popescu 2005
34
The Expectation/Maximization Algorithm

• Results of Popescu 2005
• Average accuracy over all pairs of algorithms
  tested was 97
• Minimum testing accuracy was 87 (3x3 median
  filter vs. variable number of gradients)

Estimated interpolation coef. from 100 images CFA
for 8 different algorithms projected on a 2D
space
35
EM Algorithm for Camera Detection

• Bayram et al. 2005
• Applies EM algorithm to identify 3 different
  cameras, classification via SVM

predicted
A c t u a l
3x3 Interpolation kernel
4x4 Interpolation kernel
5x5 Interpolation kernel
S. Bayram, H. Sencar and N. Memon, Source
Camera Identification Based on CFA
interpolation, Proc. of the IEEE ICIP (2005)
36
EM Algorithm for Camera Detection

• Bayram et al. 2005
• Multiclass SVM

predicted
A c t u a l
5x5 interpolation kernel
37
Enhancements to the EM Approach

• Bayram et al. 2006
• Better detection of interpolation artifacts in
  smooth images
• Low-order interpolation introduces periodicity in
  the variance of the second derivative of an
  interpolated signal

S. Bayram, H.T. Sencar and N. Memon,
Improvements on Source Camera-Model
Identiciation Based on CFA Interpolation, Proc.
of WG 11.9 Int. Conf. on Digital Forensics
(2006).
38
Enhancements to the EM Approach

• Results of Bayram et al. 2006

predicted
A c t u a l
5x5 interpolation kernel
Periodicity in the second order derivative
Combined set of features
39
Enhancements to the EM Approach

• Long et al. 2006
• Use modeling error, instead of interpolation
  filter coefficients
• Swaminathan et al. 2006
• Assumes a CFA pattern, discriminating between
  the interpolated and un-interpolated pixel
  locations and values
• Estimate interpolation filter coefficients
  corresponding to the pattern
• Compute error between an image of newly
  interpolated interpolated pixels, and the
  actual image

40
Lens Distortions

• Compensation for radial distortion induces unique
  artifacts in the images
• Choi et al. introduces a second order radial
  symmetric distortion model
• Model parameters are used as classification
  features
• Accuracy 91

Radial distortion
Rectified Image
41
Sensor Fingerprinting Individual Source
Identification

• Need more detail beyond what weve looked at so
  far with Source Model Identification
• Hardware and component imperfections, defects,
  and faults
• Effects of manufacturing process, environment,
  operating conditions
• Aberrations produced by a lens, noisy sensor,
  dust on the lens
• Artifacts may be temporal!

42
Imaging Sensor Imperfections

• Early work Kurosawa et al. 1999
• Detect fixed pattern noise caused by dark current
  in digital video cameras
• Dark current the rate that electrons accumulate
  in each pixel due to thermal action

Image credit http//www.diaginc.com/techforum/ima
gecorrections.shtml
Accumulation of dark current on CCD (dark frame
brightened 37 times for viewing)
K. Kurosawa, K. Kuroki and N. Saitoh, CCD
Fingerprint Method, Proc. of the IEEE ICIP
(1999)
43
Imaging Sensor Imperfections

• Kurosawa et al. 1999

DCR-VX1000
80642
30821
72567
49967
Serial Numbers
44
Imaging Sensor Imperfections

• Geradts et al. 2001
• Detect hot pixels, cold/dead pixels, pixel
  traps, and cluster defects

Same Camera Model, Averaged Blank Images
Camera A
Camera B
Z.J. Geradts, J. Bijhold, M. Kieft, K. Kurusawa,
K. Kuroki, and N Saitoh, Methods for
Identification of Images Acquired with Digital
Cameras, Proc. of SPIE, vol. 4232 (2001)
45
Imaging Sensor Imperfections

• Effects of temperature on pixel response
• Pixel defects in a real image

0º C
20º C
40º C
46
Imaging Sensor Imperfections

• No quantitative analysis of previous two methods
• Lukas et al. 2006 Formal quantification and
  analysis of sensor noise for identification

Pattern noise
Fixed pattern noise
Photo-response non-uniformity noise
Pixel non-uniformity
Low-frequency defects
J. Lukas, J. Fridrich and M. Goljan, Digital
Camera Identification From Sensor Pattern Noise,
IEEE Trans. On Inf. Forensics and Security, vol.
1, no. 2, pp. 205-214.
47
Pixel Non-Uniformity Noise

• The signal r exhibits properties of a white noise
  signal with an attenuated high frequency band
• Attenuation is likely due to the low-pass
  character of the CFA interpolation algorithm
• PNU noise cannot be found in saturated pixels
  (255)

Magnitude of Fourier transform of one row in an
image obtained as average over 118 images of a
flat seen
48
PNU Camera Identification Algorithm

• Establish a camera reference pattern Pc, which is
  an approximation to the PNU noise
• An approximation of PNU noise P(k)
• (P1 P2 PN) / N
• Optimization suppress scene content by applying
  denoising filter F, and averaging noise residuals
  n(k)
• n(k) P(k) - F(P(k))

49
PNU Camera Identification Algorithm

• Calculate the correlation ?C between the noise
  residual n p - F(p) and the camera reference
  pattern PC
• ?C(p)

(n - n) ? (PC - PC)
n - n ? PC - PC
50
PNU Identification Results of Lukas et al. 2006
Distribution of the correlation of the reference
pattern from Nikon D100 with noise residual from
8x300 images from all other cameras
Distribution of the correlation of the reference
pattern from Canon S40 with noise residual from
300 Olympus C765 images
51
PNU Identification Results of Lukas et al. 2006
Decision threshold t and FRR for FAR 10-3
52
PNU Identification Results of Lukas et al. 2006
Nikon D100 with noise residual from approximately
6x300 JPEG compressed images with quality factor
90 from 6 other cameras
Correlation of noise residuals from 84 Cannon G2
1600x1200 JPEG images with 6 reference patterns
53
Application of PNU Camera Identification

• Sutcu et al. 2007
• Fuse pattern noise properties with demosaicing
  characteristics

yes
Decision image was taken by this camera
yes
Classifier based on interpolation coefficients
Pattern noise match
no
no
Decision image was not taken by this camera
Y. Sutcu, S. Bayram, H.T. Sencar and N. Memon,
Improvements on Sensor Noise Based Camera
Identification, Proc. of IEEE ICME (2007).
54
Application of PNU Camera Identification