CRUSH

1
CRUSH
OPTIONS
IMAGES
2
The Submillimeter Challenge
_at_350 um Atmospheric Background is 107 times
brighter than faint galaxies. Analogous to
Observing a 16 Magnitude star at daytime!
3
The Atmospheric Power Spectrum (5 January 2003)
Strong 1/f characteristic
Turnover to white noise regime at 10 Hz
Need Fast Sampling of Background (SHARC-2 has
samples every 36ms)
4
Differencing of Signals
100 Jy source
5
Residual Sky Noise
Lissajous Sweep
Chopped
Vesta 5 Jy (2 min)
Simulated 4 Hz chopper with 40 throw under
better than average conditions
Blank Sky (10 min)
Chopped Image has Limiting noise typically 2-3
times higher than 'state of the art'
reduction. Not including deconvolution noise!!!
6
The Need for Speed (Sweeping)

• Moving Several pixels onto the same sky position
  within the limiting instrumental time scale
  'calibrates' pixels against one another.
• The more pixels that can be thus related, the
  more robust the 'calibration' measurement.
• 'Calibrated' pixels can observe several positions
  on the sky within the limiting time scale,
  leading to high fidelity maps.

Faster

• Better pixel-to-pixel calibration
• Larger and higher fidelity maps

• Crossing Sweep Patterns
• Non-periodic source crossing
• Move Primary to avoid changing illumination
  pattern

Smarter and Better
7
Scanning Strategies without a Chopping Secondary
For large map making. Obtains uniform coverage
over an area much larger than the array
For compact and point sources Maximizes time
coverage over a small area.
8
From Chopped Data to Discreet Modeling
Singular Value Decomposition
Mathematically Rigorous Maximum Entropy Solution.
Difficulties with SVD

• Computationally costly. (Large Matrices to
  invert)
• Non-linearities. (Gain fitting).
• Degeneracies, Singularities and Constraints
• Time dependent noise

Parallel SVD Effort at Goddard Space Flight Centre
9
Model 1
Iterative Reduction

• Series of Maximum Likelihood Estimators.
• Brightest First
• Convergence via Iterating

Model 2
Advantages
Model 3

• Intuitive
• Fast. (Linear with Data Size)
• Able to Deal with Non-Linearities
• Easily Configurable / Changeable

Model N
Final Source Model
10
Instrument Specific Models
Vesta 8293 5 January 2003 Excellent Conditions
Static Residual Pixel Offsets 2000 Jy
Static Residual Pixel Offsets 2000 Jy
Source Model 5 Jy

Electronic Row Drifts 1 Jy
Detector 1/f Drift Model 1 Jy
Acceleration Response 0.2 Jy
11
The Format of the SHARC-2 Array
32 x 12 pixels. Nearly Nyquist Sampled.
12
Maximum Likelihood Estimators
Alternatively, in terms of the Residuals
As if residuals only contained given model...
13
CRUSH Models with Maximum Likelihood Estimators
Typical Time Scale scan 10 min scan scan 1
chop 1 frame 1 second 30 seconds 1
second 0.1-1 second scan all time scales all
time scales
Typical Spacial Scale pixel array pixel pixel pixe
l array row pixel array 4x4 to
8x6 pixel pixel pixel
Typical Flux 1,000 10,000 Jy 10-1000 Jy 10
Jy 1-10 Jy 1 Jy 1 Jy 10 mJy 10 mJy
Model Name Residual Pixel Offsets Correlated
Background Noise Gain Modelling Pixel
Weights Chopping Residuals. Temperature
Gradients. Electronic Row Drifts Detector 1/f
Drifts Time Weights Residuals Spikes Regional
Correlations Acceleration Response Temporal
Features Spectral Features

14
Simulated Raw Data (1/f correlated)
SHARC 1.5 Uranus Raw (Dowell)
Preliminary Simulations for the Iterative
Reduction Method (Feb 2001)
Partial Cleaning (50 iterations)
SHARC 1.5 Uranus Reduced (Dowell)
Reduction Goal Simulated source with only white
noise this is the best any analysis could
achieve
Deep Cleaning (200 iterations)
SHARC 1.5 single row of bolometers. Edge pixels
used for estimating correlated noise.
15
Simulations with CRUSH
Tom Tyranowski
100 mJy Ring surrounding compact star in 1
hour 10' x 10' Billiard Ball Scan. Imperfect
cleaning of faint large scale structures
100 mJy Compact Lissajous Sweep in 1 hour Clean
background on small scales
Source Fluxes Recovered within 1
16
Correlated Noise And Gain Fitting
17
Non-Linear Response
where,
Small signal gain
Large signal gain
18
Weighting
Weights Separated into a product of pixel-only
weights and time-only weights.
Where B is the number of Active Bolometers, and P
is the number of Parameters fitted in the time
interval T'
Where P is the number of fitted parameters in the
time interval T
19
Identifying Anomalous Pixel Behaviour

• One of the Most challenging aspects of deep
  reductions
• Removal of Residual Spikes
• Flagging of Pixels with Unreasonable Gain Fits
• Looking for Statistically Significant Temporal
  Features...
• ... and Spectral Features

About 300 of 384 pixels are good enough
20
The Lockman Hole z 2-3
SHARC-2

• Follow-up of SCUBA galaxies
• 4 Fields, 3-4 Hours of Grade I Each
• 1-sigma of 5mJy depth.
• 3 of 4 SCUBA galaxies detected
• Additional 7 objects detected above noise

21
Map Noise Characteristics Detection Confidence

• Perfectly Gaussian Noise
• 2 Times wider than expected statistically from
  independent pixel noise! Detectors are NOT
  independent!
• Positive tail Clearly indicating the presence of
  source flux.

22
Residual Pixel-to-Pixel Covariances (Learning
from the Data Themself...)
Gaussian with ca. 35 FWHM
23
Principal Options to CRUSH

• Reduction Options
• Scan Specific Options
• Model Specific Options
• Source Map
• Gain Modelling
• Frame and Pixel Weighting
• Row Model
• Detector Drift Model
• 2-D Gradient Model
• Residual Spike Removal
• Temporal and Spectral Feature Identification
• Chopping Residual Model
• Acceleration Response Model

24
Options to CRUSH
crush options -help
A concise listing of ALL available options to
crush, including their current settings.
http//www.submm.caltech.edu/sharc/crush/glossary
.html
A detailed explanation of the options, with
corresponding configuration keys, and valid
arguments. Cross-referenced and easily
searchable. The true CRUSH User's Guide.
25
Principal Reduction Options
Brightness Related
-faint -deep
LOAD_CONFIG faint.cfg LOAD_CONFIG deep.cfg
lt 1 Jy lt 100 mJy. Similar to -faint -compact,
but more aggressive.
Size Related
Do not worry about filtering extended structures
on the scale of the array. Instead clean
aggressively...
EXTENDED_STRUCTURE false
-compact
Miscalleneous
-rounds -point -configxxx -altaz
ITERATIONS LOAD_CONFIG point.cfg LOAD_CONFIG
xxx ALTAZ true
The number of iterations before final solution.
Perform pointing after reduction. Load setting
for confguration file xxx. Reduce maps in Alt/Az
coordinates instead of the default RA/DEC.
26
Scan Specific Options
Options affect all scans that are subsequently
listed.
Preconditioning of Scan data
-average -chopped
The number of 36 millisecond frames to integrate
data over. E.g. -average3. If a chopping
secondary was used.
AVERAGE_FRAMES CHOPPED_OBSERVATION true
Adjustments
Scale the scan data by this factor. E.g.
-scale1.23 13852 13853 Adjust the pointing.
E.g. -FAZO-103.0 -FZAO12.7 9712 Change the
tau value to use. E.g -tau225GHz0.046 15224
-scale -FAZO -FZAO -tauidvalue
Location
Specify the directory where the raw data resides.
E.g. -path/home/nobody/data/SHARC2
RAW_DATA_PATH
-path
27
Generic Options
Activation Iteration
-Ixxx
XXX_TURN
Specify when a given model is to be activated.
Several models have 'auto' setting available.
When specified each model will evaluate a number
of possible criteria, based on which it activates
or delays. One can also explicitly activate a
model at the given iteration.
Characteristic Time Constant
-xxxT
XXX_T
Several of the models have time constants
assigned to them. Most commonly they control the
time resolution of the given model. The longer
the time constant, the less often a parameter is
fitted for the given model. Large
Robust Small Aggressive
28
Options to Source Map
Pipeline Auto Configure Related
-Ifidel -Iextended
MAP_FAITHFUL_TURN EXTENDED_STRUCTURE_TURN
Do not allow self-activating models to activate
until the given iteration. Also, if negative
clipping is enabled, clip images until the
specified iteration. E.g. -Ifidel3 If
EXTENDED_STRUCTURE is set true, do not self
activate models that may remove extended
structure until the specified number of source
generations is obtained. E.g. -Iextended4
Map Appearance Related
Clip noisy map edges. Use exposure time relative
to peak exposure to define clipping criterion.
E.g. -minExp0.1 Convolve map to beam size
to get rid of unwanted high spacial frequency
structure. E.g. -convolve4.0
MAP_RELATIVE_EXPOSURE CONVOLVING_BEAM_FWHM

-minExp -convolve
29
Gain Fitting and Gain Adjustment Options
Gain Related
-gainRounds -gainGoal
GAIN_ITERATIONS GAIN_CONVERGENCE_GOAL
The number of initial gain iterations. Like the
full pipeline iterations, except only gain and
models preceding gain are solved for. The
desired gain convergence if GAIN_ITERATIONS is
set to 'auto'. The fainter the source, the better
the convergence that is required.
Fast Line-of-Sight Opacity Adjustment
Allow/Disallow for fast (real-time) adjustment of
the extinction around the mean extinction value
(explicitly defined or obtained from Mai-Tau)
base on the background power variations.
TAU_ADJUSTtrue/false
-tauAdjust -notauAdjust
30
Weighting and Flagging
Weight Related
Set the time scales over which pixel weights are
to be determined. Max should be set such that it
is smaller than beam crossing time.
-pWeightTmin-max
PIXEL_WEIGHT_T
Flagging Related
-minDOF -spikeLevel -spikeRatio -maxTemp
oral -maxSpectral
The minimu degrees of freedom that must remain
per pixel for the pixel to remain active. Remove
residual spikes above the specified significance
level. Flag pixels that have a higher fraction
of spikes to data point. Flag pixels with
temporal features above the spec'd significance
level. Flag pixels with spectral features above
the spec'd significance level.
MIN_DEGREES_OF_FREEDOM DESPIKE_LEVEL REJECT_S
PIKE_FRACTION MAX_TEMPORAL_FEATURE MAX_SPECTRA
L_FEATURE
31
Output Related Options
Gain Related
-outpath -name -precess -resolution

REDUCED_MAP_PATH REDUCED_MAP_EPOCH MAP
_RESOLUTION
The output directory in which to place files that
are automatically names as SourceName.Scan1.
... .ScanN.fits. E.g. -outpath/home/nobody
/reduced The name of the output file including
absolute path. E.g. -name/home/nobody/reduced/
mysource.fits The coordinate epoch in which the
reduced map is written. CRUSH does precession!!!
E.g. -epoch1950.0 The map resolution. The
setting 'auto' defaults to 1/3 Cassegrain
detector pixel size, 1.62, but one can set this
to any desired value really.

32
CRUSH Tips

• Reduce as much data at once as you possibly can.
  Coadd later. Increase the memory allocated to
  the Java VM.
• Reduce Data together that were taken with
  different Position Angles and Scanning Angles.
• If default reduction is 'funny'.
• Try different brightness. (-bright, -faint,
  -deep)
• Adjust gain convergence criterion.
• Try -compact if emission is not on the scale of
  the array.
• Adjust specific model time-scales.
• Try to identify problematic scans. Act on them!
• Trade Extended flux for Flatter Baselines.
• Check on Pointing.
• Use Mai-Tau.

33
Observing Tips

• Choose Appropriate Scanning Pattern
• Size (smaller patterns yield cleaner data)
• Coverage (Lissajous vs Billiard Ball)
• Speed (Faster Better)
• Orientation (not along rows!!!)
• Feasibility.
• Chopping / Not Chopping...
• Point Often on Nearby object (esp. in ZA)
• Calibrate Often on known calibrator
• Check on Beam Quality every night
• Use DSOS (Dish Surface Optimization System)
• Keep Detailed Logs

34
The Data Structure
Free FITS viewing software FV
35
Primary Image Flux Distribution
Measurement Flux As would be seen by detector
if there was no atmospheric absorbtion
Natural units are Voltage (nV, V)
Response to 1 Jy source
Pseudo flux units (Jy/beam, Jy/sr, Jy/arcsec2)
36
Aperture Flux
For some Aperture
With default crush pixelization (1/3 Cassegrain
Pixel Size)
37
Peak Flux (Flux Inside a Beam)
Where, the typical 350um values for SHARC-2 are,
38
Smoothing with a Beam
Define Smoothing as
Where, the a typical smoothing beam will be a
Gaussian of the form
39
Second Image RMS
Measurement Uncertainty Uncertainty of the
Detector Measurement
Map Uncertainty The Uncertainty of the Flux
value on the map.
For default pixelization...
...and Gaussian Beam
40
Flux Uncertainty inside an Aperture
With default crush pixelization (1/3 Cassegrain
Pixel Size)
41
Excess Noise and Non-Independent Pixels
Correlated Pixel Noise
Some Linear Quantity A
The Uncertainty on A
42
Excess Noise Determination
From Map Chi-Squared (No source)
From Map NoiseDistribution
Gaussian Fit upper limit
43
Crush Suite of Utilities
imagetool Aiming to be a comprehensive image
manipulation tool. Scaling, rms scaling, map
units, regridding, smoothing, filtering and
clipping. Under development... show (via
imagetool) Display tool that allows quick view of
images with some limited capabilities such as PSF
fitting, image toggle, units on the fly
etc. coadd Produce maps out of multiple
images. jiggle Shift maps on the fly to
determine alignment covarsee A visualization
tool for pixel-to-pixel covariance
matrixes. histogram Map signal-to-noise
Histrogram generator. Useful for determining
excess noise quick diagnostic
deconvolve Obtain super-resolution.
Undocumented...