Titelfolie

1
Deconvolution with the Huygens Remote Manager
Aaron Ponti Facility for Advanced Imaging and
Microscopy
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
2
This course bases heavily on the SVI-wiki
at http//support.svi.nl/wiki/ Another very nice
site is http//www.micro.magnet.fsu.edu/primer/
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
2
3
Summary

• Image formation (basics)?
• Acquisition pitfalls
• Deconvolution primer
• HRM demo

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
3
4
PART I Image formation (basics)
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
4
5
Summary

• Image formation (basics)?
• The Huygens-Fresnel principle
• The role of the objective
• Resolution
• Image formation model
• Acquisition pitfalls
• Deconvolution primer
• HRM demo

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
5
6
Image formation (basics) The Huygens-Fresnel
principle
Image formation (basics)?
A. The Huygens-Fresnel principle Simplified model
to describe the behavior of waves in space that
explains phenomena like refraction, diffraction
and interference. In particular, it applies to
image formation in light microscopy.
Aristotle (384-322 BC) Augustin-Jean
Fresnel (1788-1827) Thomas Young (1773-1829)
Pythagoras (580-500BC) James Clerk
Maxwell (1831-1879) Philipp Lenard (1862-1947)
Albert Einstein (1879-1955)
Arthur H. Compton (1892-1962)
Louis-Victor de Broglie (1892-1987)
What is light? http//micro.magnet.fsu.edu/primer/
lightandcolor/particleorwave.html
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
6
7
Image formation (basics) The Huygens-Fresnel
principle
Huygens
Each point in space reached by the wave front
behaves as a new, secondary source
Spherical wave front
Huygens wavelet
Single, mathematical point source emitting light
in all directions
Fresnel
The amplitude of the wave at any given point in
space is given by the superposition
(interference) of the amplitudes of all secondary
wavelets at that point.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
7
8
Image formation (basics) The Huygens-Fresnel
principle
If we put an obstacle in the way of the spherical
wave front, we avoid the excitation of some of
the secondary sources ? the spherical wave front
is broken

• If the obstacle is an infinite wall ? no
  radiation will cross
• If we make a small hole in the wall, only the
  space in the hole will be excited and emit on the
  other side in a particular wave front that
  depends on the shape of the hole
• Thomas Youngs experiment that seemed to prove
  once and for all that light is a wave

This simple idea helps understand a lot of
phenomena like diffraction a person in a room
hears the noise coming from another room as if it
were originating from the doorway.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
8
9
Image formation (basics) The role of the
objective
Image formation (basics)?
B. The role of the objective The role of the
objective is to refocus the light emitted by a
light source to some screen or detector.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
9
10
Image formation (basics) The role of the
objective
Focusing, convergent lens
Huygens wavelet
Cone of captured light a large fraction of
the wave front is not captured by the lens and
the light source cannot be refocused to a point
Screen or detector, where all wavelets interfere
Airy disk, NOT a point!
Interference pattern
The lens delays the light more in the center and
less toward the periphery and inverts
(mirrors) the shape of the spherical wave front
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
10
11
Image formation (basics) The role of the
objective
The interference behavior in the image plane
(Airy disk) also extends along the optical axis
and gives rise to the (3D) Point Spread Function
(PSF).
Lateral (XZ) view
Image plane
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
12
Image formation (basics) Resolution
Image formation (basics)?

• C. Resolution
• The amount of light that the lens can collect is
  measured by the Numerical Aperture (NA).
• NA and wavelength l define the size of the Airy
  disk (PSF)
• The radius r of the Airy disk (the distance
  between the central maximum and the first
  minimum) at the focal plane is
• r 0.61 l / NA
• Rayleigh criterion
• The Rayleigh criterion establishes a standard to
  characterize the spatial resolution of an optical
  device the minimum resolvable detail, or how
  much can two points be close to each other before
  they become indistinguishable.

larger NA, higher resolution
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
12
13
Image formation (basics) Resolution
Fully resolved
Rayleigh
Unresolved
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
14
Image formation (basics) Image formation model
Image formation (basics)?

• Image formation model
• Fluorescent microscopes are incoherent1 imaging
  systems ? the image formation process is linear
  and described by linear system theory Linear
  means that two different light signals (or of any
  other nature, in general) coming from two
  different points of the object do not interfere
  with each other. The resulting image of two
  emitting points is equal to the addition of the
  images that would arise by measuring the two
  points separately.
• This is mathematically represented by a
  Convolution equation.

1 For a specimen illuminated by a large-angle
cone of light, or for self-luminous objects, the
light rays forming adjacent Airy patterns are
incoherent and do not interfere with each other.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
14
15
Image formation (basics) Image formation model
Image formation (basics)
Blurring (convolution)
Acquired image
Noise
Sampling!
Real signal
Convolution operator
3D PSF
Noise model
The imaging in the fluorescent microscope is
completely described by its PSF.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
15
16
Image formation (basics) Image formation model
Image formation (basics)
XZ view
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
16
17
Image formation (basics) Image formation model
Once the exposure is over, the charge at each
sensor is measured, and the measurement is
converted into a digital value. This measurement
process is called readout. Readout noise since
the accumulated signal has to be amplified to be
read, and there is no such thing as a perfect
amplifier, the amplifier adds a bit of noise,
similar to static in a radio signal, to the
charge it is amplifying.
A simplified sensor (e.g. a CCD pixel)
The sensor converts photons into electrons(1) and
accumulates them during some (exposure) time
until readout
Dark noise is an accumulation of heat-generated
electrons (dark current) in the sensor
Photons are emitted, they reach the sensor with a
rate that follows a Poisson distribution Photon
noise
Fluorescently labeled sample
(1) With given Quantum Efficiency
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
17
18
Image formation (basics) Image formation model
The signal to noise ratio
noise dominated by photonic component
noise dominated by readout and dark components
EM-CCD
IDEAL
S/N
CCD
Number of photons
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
18
19
PART II Acquisition pitfalls
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
19
20
Summary

• Image formation (basics)?
• Acquisition pitfalls
• Refractive index mismatch
• Clipping
• Undersampling
• Bleaching
• Illumination instability
• Mechanical instability
• Thermal effects
• Deconvolution primer
• HRM demo

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
20
21
Acquisition pitfalls Refractive index mismatch
Acquisition pitfalls

• A. Refractive index mismatch
• Geometrical aberration the Fishtank effect
• Spherical aberration
• Total internal reflection

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
21
22
Acquisition pitfalls Refractive index mismatch
Geometrical aberration
Acquisition pitfalls
Refractive index mismatch Geometrical
aberration The objective moves along the
optical axis (z) for a certain user-defined
distance, but the focus shifts inside the sample
by a different step size. Therefore objects will
appear shortened (as in a fish tank when the
objects in the water are viewed from the air) or
elongated in the microscope, depending on the
ratios of the indices.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
22
23
Acquisition pitfalls Refractive index mismatch
Spherical aberration
Acquisition pitfalls

• Refractive index mismatch Spherical aberration
  /1
• Can also be given by problems with the optical
  setup
• Spherical aberration (SA) is an optical effect
  occurring when the oblique rays entering a lens
  are focused in a different location than the
  central rays. The distance in this focal shift is
  dependent on the depth of the focus in the
  specimen.

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
23
24
Acquisition pitfalls Refractive index mismatch
Spherical aberration
Acquisition pitfalls

• Refractive index mismatch Spherical aberration
  /2
• When light crosses the boundary between materials
  with different refractive indices, it bends
  across the boundary surface differently depending
  on the angle of incidence (refraction) oblique
  rays are bent more than the central rays, and
  therefore the focusing is spoiled.
• SA is especially harmful for wide-field
  deconvolution.

With SA, the PSF becomes asymmetric at depths of
a few microns.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
24
25
Acquisition pitfalls Refractive index mismatch
Spherical aberration
Acquisition pitfalls
Refractive index mismatch Spherical aberration
/3 Workarounds keep the Z-range of the data as
small as possible Solution use a lens with an
immersion medium with a refractive index that
matches that of the preparation.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
25
26
Acquisition pitfalls Refractive index mismatch
Total internal reflection (TIR)
Acquisition pitfalls

• Refractive index mismatch Total internal
  reflection (TIR)
• When the lens numerical aperture is larger than
  the medium refractive index, TIR will occur,
  causing excitation light to be bounced back into
  the lens and limiting the effective numerical
  aperture (and therefore the resolution).
• The TIR increases with increasing NA.

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
26
27
Acquisition pitfalls Clipping
Acquisition pitfalls

• Clipping
• Clipping during acquisition occurs when the
  dynamic range of the input signal exceeds that of
  the analog-to-digital converter (ADC).

Saturated signal
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
27
28
Acquisition pitfalls Clipping
Acquisition pitfalls

• Clipping
• Clipping during acquisition occurs when the
  dynamic range of the input signal exceeds that of
  the analog-to-digital converter (ADC).

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
28
29
Acquisition pitfalls Clipping
Acquisition pitfalls

• Clipping
• Clipping during acquisition occurs when the
  dynamic range of the input signal exceeds that of
  the analog-to-digital converter (ADC).

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
29
30
Acquisition pitfalls Clipping
Acquisition pitfalls

• Clipping
• An example of artifacts of deconvolution of
  clipped images are apparently hollow structures
  (that in reality are NOT hollow at all)!

Saturated histogram ? clipping
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
30
31
Acquisition pitfalls Undersampling
Acquisition pitfalls

• Undersampling
• Sampling is the process of converting a signal
  (e.g. a function of continuous time or space )
  into a numeric sequence (a function of discrete
  time or space).
• The sampling density is the number of recorded
  samples per unit volume.
• ? The larger the sample size, the lower the
  sampling density.
• ? The size of the sample is the size of the voxel
  in the acquired image.

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
31
32
Acquisition pitfalls Undersampling
The Nyquist-Shannon sampling theorem establishes
that when sampling a signal (converting from an
analog signal to a digital signal), the sampling
frequency must be greater than twice the band
width of the input signal (Nyquist rate) in order
to be able to reconstruct the original perfectly
from the sampled version. The band width
is the difference between the maximum and the
minimum frequency transferred by the optical
system. In microscopy, the band width of the
signal is given by the extension of the PSF.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
32
33
Acquisition pitfalls Undersampling
Emission wavelength excitation wavelength 500
nm
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
33
34

• Consequences of undersampling
• Signal information is lost (the acquired images
  have lower resolution than what is seen under the
  microscope)

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
34
35

• Consequences of undersampling
• Aliasing can occur

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
35
36
Acquisition pitfalls Undersampling
Example of quality of deconvolution vs. axial
sampling distance
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
36
37
Acquisition pitfalls Undersampling (and
oversampling)
Acquisition pitfalls

• What about oversampling?
• Acquiring at a frequency larger than the optimal
  does not bring any more image information
• It can lead to longer acquisition times
• It can increase bleaching
• File size and computation time increase

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
37
38
Acquisition pitfalls Bleaching
Acquisition pitfalls

• D. Bleaching
• Bleaching (also photobleaching) is the
  progressive fading of the emission intensity of
  the sample under study.
• High intensity excitation light often leads to
  irreversible decomposition of the fluorescent
  molecules that is seen in practice as fading of
  the emitted fluorescent light.
• The more illumination is required, the more
  bleaching will occur (e.g. in widefield 3D images
  and generally in time series).
• There are chemical agents that can reduce the
  impact of bleaching.
• Deconvolution allows you to use less light to
  excite you fluorophore and to get the missing
  contrast algorithmically!

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
38
39
Acquisition pitfalls Bleaching
Acquisition pitfalls

• E. Illumination instability
• Some widefield microscopes are equipped with
  unstable (flickering) arc lamps.

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
39
40
Acquisition pitfalls Bleaching
Acquisition pitfalls

• F. Mechanical instability
• Mechanical instability can take many shapes, for
  example
• The Z stage moves irregularly or with sudden
  jumps. Near-fatal for confocal or WF
  deconvolution.
• The specimen moves. If in WF data the object can
  clearly be seen moving when slicing along over a
  few micron in Z this will cause problems for the
  deconvolution. Best cause of action, apart from
  speeding up acquisition, is limiting the Z-range
  of the data as much as possible. Confocal data of
  moving specimen causes less problems.

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
40
41
Acquisition pitfalls Bleaching
Acquisition pitfalls

• G. Thermal effects
• Thermal effects are known to affect calibration
  of the Z-stage, especially if piezo actuators
  without feedback control are used. In particular
  harmful for WF data.
• In time series the effect can be seen as a drift
  of the Z-position, or even a periodic movement
  induced by periodic switching on and off of an
  air-conditioning system.

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
41
42
PART III Deconvolution primer
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
42
43
Summary

• Image formation (basics)?
• Acquisition pitfalls
• Deconvolution primer
• HRM demo

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
43
44
Deconvolution primer
Deconvolution primer

• Some terminology
• A better term for what we mean here is image
  restoration
• Image restoration refers to the recovery of a
  signal from degraded observations
• Image restoration is different from image
  enhancement in that the latter is designed to
  emphasize features of the image that make the
  image more pleasing to the observer, but not
  necessarily to produce realistic data from a
  scientific point of view
• Deconvolution is just one of the tasks of image
  restoration.
• It tries to counteract the physical process of
  convolution.
• Deconvolution inverse( convolution )

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
44
45
Deconvolution primer
Deconvolution primer
Convolution Theorem
Convolution in direct space multiplication in
frequency (Fourier) space
Fourier transform
Inverse Fourier transform
Inverse filtering
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
45
46
Deconvolution primer
Deconvolution primer
Cookie cutter
h
f
Fourier transform F
F
H
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
46
47
Deconvolution primer
f
Deconvolution primer
Cookie cutter
f
F
?
F -1(G/H)
G
g
F (f)
h
H
F -1(G)
F (h)
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
47
48
Deconvolution primer
Deconvolution primer
Cookie cutter
Missing frequencies
XZ
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
48
49
Deconvolution primer
Deconvolution primer
Deconvolving trains
Sub-resolution train Noise-free convolution and
deconvolution
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
49
50
Deconvolution primer
Deconvolution primer
Deconvolving trains
Sub-resolution train Noise-free convolution and
deconvolution
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
50
51
Deconvolution primer
Deconvolution primer
Deconvolving trains
Sub-resolution train Noise-free convolution and
deconvolution
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
51
52
Deconvolution primer
Deconvolution primer
Convolution Theorem
Extreme noise amplification!
Artifacts!
H 0 at many places!
Inverse filtering will never allow us to recover
the true object function f.
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
52
53
Deconvolution primer
Deconvolution primer

• Deconvolution (image restoration) algorithms
• Nearest neighbors (subracts from each plane a
  weighted average of the nearest planes not
  really a deconvolution...)
• No neighbors (subtracts from each plane a blurred
  version of itself not really a
  deconvolution...)
• Wiener filter (an inverse filter that tries to
  minimize the impact of deconvolved noise at
  frequencies which have a poor SNR)
• Iterative constrained algorithms (iteratively
  refines an estimate of the object by convolving
  it with the known PSF and comparing it with the
  acquired image the weighted (by a relaxation
  factor) difference is added to the estimate as
  guess for the next round it is an iterative
  inverse filter)
• Blind deconvolution (an iterative constrained
  algorithm that tries to estimate both the
  original object and the PSF simultaneously from
  the degraded image)
• Maximum likelihood estimation algorithms
  (iteratively optimizes the likelihood of an
  estimate of the object given the measured image
  and the PSF the noise should be
  Poisson-distributed)
• Wavelet-based deconvolution (future?)

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
53
54
Deconvolution primer
Deconvolution primer

• What HRM (Huygens) does
• Increases resolution (especially in the axial
  direction)
• Removes noise
• Increases contrast
• Corrects for some of the acquisition pitfalls
• Geometrical distortion CAREFUL!
• Spherical aberration CAREFUL!
• Bleaching
• Illumination instability
• Thermal effects

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
54
55
Deconvolution primer
Deconvolution primer

• How does HRM (Huygens) do it
• Uses either a theoretical PSF calculated from
  optical parameters (wavelength, NA, voxel
  size,... ) or an experimental PSF obtained by
  distilling images of sub-resolution beads.
• Uses one of the following algorithms for image
  restoration
• Classic maximum likelihood estimation (good for
  almost any type of microscopy image, and
  well-suited for low-signal images and to restore
  point-, line-, and plane-like features)
• Quick maximum likelihood estimation (much faster
  than classic and with almost the same quality
  optimal for time series)
• Iterative constrained Tikhonov-Miller (fast and
  particularly good for low-noise wide field
  images)
• Quick Tikhonov Miller (is an inverse filtering
  method that can give noise amplification used in
  very specific circumstances only)

Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
55
56
Deconvolution primer
Deconvolution primer
Deconvolution example
Actin filaments, 2-Photon microscope
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
56
57
PART IV HRM demo
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
57
58
Some important places
HRM http//huygens.fmi.ch http//hrm.fmi.ch
Source folder Q\huygens_src \\huygens\huygens2_d
ata\username\huygens_src Destination
folder Q\huygens_dst \\huygens\huygens2_data\use
rname\huygens_dst
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
58
59
http//huygens.fmi.ch
Deconvolution with the Huygens Remote Manager
(Aaron Ponti)
59