A%20Perceptual%20Metric%20for

1
(No Transcript)
2
A Perceptual Metric for Production
Testing Yangli Hector Yee Anna Newman RD
Staff PDI/DreamWorks
3
Perceptual Metric for Production Testing

• The Big Picture
• Complex pipelines and tight production schedules
  demand stable, fast rendering tools
• Procedural testing is tricky, requires its own
  specialized tools.
• Perceptual metrics increase the accuracy and
  efficiency of our testing

4
Perceptual Metric for Production Testing

• Testing Reduces Production Costs
• Validate consistency of software behavior reduce
  wasted production effort, re-renders, etc.
• Evaluate new configurations minimize
  productivity loss after deployment
• Monitor render performance (timing) avoid
  degradations
• RD POC for artist route reproduce defects,
  etc.

5
Perceptual Metric for Production Testing

• Testing at PDI/DreamWorks
• 4 week release cycle
• Test new software versions with abstract tests
  (example fur shader testcases)
• Test new versions in pipeline with reference
  frames from movie
• Track performance (timing) of rendering across
  releases

6
Perceptual Metric for Production Testing

• Evaluating Test Results
• Compare new renders with reference renders
• Differences at the Pixel level failure
• Pixel changes have many sources
• Architecture differences (OS, gfx hardware, etc)
• Rounding
• Randomization and sampling
• Changes to functionality (bug fix, new
  functionality)
• Unexpected changes in rendering behavior that
  need to be resolved

7
Perceptual Metric for Production Testing

• False Positives
• Time consuming to manually evaluate failures
• Easy missing leave, gross skin opacity, light
  mask, etc
• Harder fur shift, noise changes, etc.
• Some pixel changes visually insignificant false
  positives
• Need tool to focus evaluation and debugging on
  actual defects, not false positives

8
Easy to Evaluate
9
Easy to Evaluate
10
Easy to Evaluate
11
Harder to Evaluate
12
Very Hard to Evaluate
gcc 2.96
icc 8.0.66
13
Error Metric - Physical
Perceptual Metric for Production Testing
Reference
Physical Difference
Test
14
Perceptual Metric for Production Testing
Perceptual Metrics More Accurate Efficient
Testing

• Most false positives can be eliminated
• Ability to tune level of allowable difference
• Metrics consider the type of difference

15
Perceptual Metric for Production Testing

• Perceptual Metric
• Detect visible differences
• Fast execution
• Minimize false positives
• Easy to understand

Quickly implemented
16
Perceptual Metric for Production Testing

• Spatial frequency sensitivity
• Luminance sensitivity
• Color sensitivity
• Additional hacks
• Extend
• Ramasubramanian, et al., 1999
• Yee, et al., 2000

17
Frequency Dependent Loss of Sensitivity
Perceptual Metric for Production Testing
100
CSF
10
log contrast sensitivity


1
Campbell-Robson Contrast Sensitivity Chart
.1
1
10
log spatial frequency (cpd)
CSF Contrast Sensitivity Function
18
Threshold Elevation
Perceptual Metric for Production Testing
100
log threshold factor
10
1
.1
1
10
log spatial frequency (cpd)
19
Spatial Frequency Estimation
Perceptual Metric for Production Testing
Laplacian Pyramid - Burt Adelson, 1983
Bandpass Image
Gaussian Low Pass Filter
20
Luminance Dependent Loss of Sensitivity
Perceptual Metric for Production Testing
TVI Threshold vs. Intensity
Flash Target
21
Perceptual Metric for Production Testing

• Color Sensitivity
• RGB -gt LAB
• Euclidian Distance

DE
(L2, A2, B2)
(L1, A1, B1)
22
Perceptual Metric for Production Testing

• DL, DE gt F CSF Luminance factor

DL - Luminance difference DE LAB Euclidian
difference F Decrease in sensitivity
23
Perceptual Metric for Production Testing

• Observer Parameters
• Field of view affects CSF
• Hollywood
• Front row 85 degrees
• Back row 27 degrees
• Orlando
• Front row 84 degrees
• Back row 44 degrees
• Larger FOV ? stronger error metric

24
Perceptual Metric for Production Testing
Effect of FOV
Reference
Test
25
Physical Difference
Perceptual Metric for Production Testing
26
FOV 85 degrees Theatre front row
Perceptual Metric for Production Testing
27
FOV 50 Degrees Theatre back row
Perceptual Metric for Production Testing
28
FOV 10 Degrees
Perceptual Metric for Production Testing
29
FOV 6 Degrees Palm sized
Perceptual Metric for Production Testing
30
Perceptual Metric for Production Testing

• Hack
• Ignore the first N pixel failures

31
Where it breaks
Perceptual Metric for Production Testing
Perceptual Difference
Reference Test
No spatial frequency to cover errors
32
Future
Perceptual Metric for Production Testing
Time dependent Contrast Sensitivity
0.15 deg/s
3 deg/s
11 deg/s
100
Retinal Velocity
10
Sensitivity
1
1
10
0.1
Spatial Frequency (cpd)
Graph adapted from Kelly, D. H., 1979, J. Opt.
Soc. Am., 691340-1349.
33
Questions and Answers
Acknowledgements Steve BakerWill BiltonMark
JackelsBill Seneshen
Corrin MilewskiPeter MillerJacques RenduJohn
Serene