Practical Implementation of High Dynamic Range Rendering

1
Practical Implementation of High Dynamic Range
Rendering

• Masaki Kawase
• BUNKASHA GAMES
• BUNKASHA PUBLISHING CO.,LTD
• http//www.bunkasha-games.com
• http//www.daionet.gr.jp/masa/

2
Agenda

• What can be done with HDR?
• Dynamic Range
• Implementation on DX8 hardware
• Implementation on DX9 hardware
• Multiple Gaussian Filters
• HDR in games
• References

3
What can be done with HDR?

• Dazzling light
• Dazzling reflection
• Fresnel reflection
• bright reflection in the normal direction
• Exposure control effects
• Realistic depth-of-field effects
• Realistic motion blur

4
Dazzling Light
5
Dazzling Reflection
6
HDR Fresnel
Bright reflection off low-reflectance surfaces
7
Exposure Control
8
HDR Depth-of-Field
Future perspective
9
HDR Motion Blur
Future perspective
10
Dynamic Range

• The ratio of the greatest value to the smallest
  value that can be represented
• Displayable image
• 28 Low dynamic range (LDR)
• Frame buffer of absolute luminance
• Render the scene in absolute luminance space
• gt232 represents all luminances directly
• Frame buffer of relative luminance
• Apply exposure scaling during rendering
• gt21516 dark regions are not important

11
HDR Frame Buffers

• For glare generation
• When rendering with relative luminances
• Ideally, more than 21516
• In games
• 21213 (4,00010,000) is acceptable

12
HDR Environment Maps

• Very important for representing
• Realistic specular reflection
• Dazzling specular reflection
• Specular reflectance of nonmetals
• Reflectance in the normal direction is
  typicallyless than 4
• Bright light remains bright after such low
  reflection
• To maintain dazzles after reflection of 1-4
• Dynamic range of more than 10,000 or 20,000 is
  necessary

13
Implementation on DX8 Hardware

• We have no choices
• Pixel Shader 1.x
• Integer operations only
• HDR buffer formats
• Low-precision buffers only
• Use the alpha channel as luminance information
• Fake it to achieve believable appearance
• Accurate calculation is not feasible

14
Fake HDR Pixel Shader
ps_1_1 tex t0 tex t1 tex t2 mad r0.rgb, v0,
t2, v1 // Scale the primary diffuse color by
// shadow/light map, and
add the result of //
other per-vertex lighting mul t0.a, v1.a, t0.a
// Scale the specular reflectance
// by gloss map mul r0.rgb, t0,
r0 // Modulate diffuse color with decal
texture mul r0.a, t0.a, t1.a // r0.a
specular reflectance
envmap luminance mul t1.rgb, t1, c0
// Modulate envmap with specular color mul
r1.a, r0.a, t1.a // Envmap brightness
parameter // r1.a
specular reflectance
envmap luminance gloss map lrp r0.rgb,
t0.a, t1, r0 // Reflect the envmap by specular
reflectance mul r1.a, r1.a, c0.a // Envmap
brightness parameter
// r1.a specular reflectance
envmap luminance gloss map
Clamp(gloss
2, 0, 1) lrp r0.rgb, r1.a, t1, r0 // Output
color // Interpolate
the envmap color and the
// result of LDR computation, based on
// the envmap brightness
parameter (r1.a) lrp r0.a, r1.a, t1.a, r0.a //
Output luminance information

• Glossy reflection material

// v0.rgb Diffuse color of primary light //
v1.rgb Color for other lights/ambient //
pre-scaled by (exposure 0.5) // // v1.a
Specular reflectance (Fresnel) // // t0.rgb
Decal texture (for diffuse) // t0.a Gloss map
(for specular) // t1.rgb Envmap color // t1.a
Envmap luminance // t2.rgb Shadow/light
map // // c0.rgb Specular color // c0.a
Clamp(gloss 2, 0, 1)
15
Generating Displayable Image

• Extract high-luminance regions
• Threshold 0.4-0.5
• Generate glare
• Reference
• Kawase, Masaki, Frame Buffer Postprocessing
  Effects in DOUBLE-S.T.E.A.L (Wreckless)
• Generate a displayable image
• Calculate the luminance from the frame buffer
• Add the result of glare generation to the
  luminance

16
Notes on DX8 Implementation

• Accurate calculation is not feasible
• How to make it believable by faking
• Based on appearance rather than theory

17
Implementation on DX9 Hardware

• There are currently many limitations
• Choose implementations accordingly
• Pixel Shader
• Pixel Shader 2.0 or later
• Pixel Shader 1.x
• Buffer formats for HDR
• High-precision integer/float buffers
• Low-precision integer buffers

18
Issues with High-Precision Buffers

• Memory usage
• At least twice as much memory as the conventional
  full-color buffer is needed
• Limitations
• Alpha blending cannot be used
• Texture filtering cannot be used with
  floating-point formats
• Some systems dont support them
• The situation is not good

19
Use Low-Precision Buffers

• Make use of low-precision buffers
• A8R8G8B8 / A2R10G10B10 etc.
• Low memory consumption
• Alpha blending can be used

20
Compression with Tone Mapping

• Render directly to displayable format
• Nonlinear color compression
• Effectively wide dynamic range
• Reference
• Reinhard, Erik, Mike Stark, Peter Shirley, and
  Jim Ferwerda, Photographic Tone Reproduction for
  Digital Images
• The alpha channel is not used
• Can be used for any other purpose

21
Environment Map Formats

• Relatively low resolution
• Alpha channel/blending is not very important
• Use the 16-bit integer format if enough memory
  storage is available
• Treat it as having an interval of 0, 256 or 0,
  512
• Texture filtering can be used
• In the future
• Do it all with A16B16G16R16F

22
Low-Precision Environment Maps

• Use them when
• High-precision buffers are not supported, or
• Memory storage is limited
• If the fill-rate of your system is relatively low
• Use the same format as used in DX8 fake HDR
• If the fill-rate is high enough
• Nonlinear color compression
• Similar to tone mapping
• Store exponents into the alpha channel
• More accurate operations are possible
• Using it just as a scale factor is not enough
• Even the DX8 fake HDR has a much bigger impact

23
Color Compression

• Similar to tone mapping
• Encode when rendering to an environment map
• Offset luminance curve controlling factor
  (2-4)
• A bigger offset means
• High-luminance regions have higher resolutions
• Low-luminance regions have Lower resolutions
• Decode when rendering to a frame buffer
• From the environment map fetched
• d a small value to avoid divide-by-zero

24
Color Compression

• Use carefully
• Mach banding may become noticeable on reflections
  of large area light sources
• e.g. Light sky

25
E8R8G8B8

• Store a common exponent for RGB into the alpha
  channel
• Use a base of 1.04 to 1.08
• 
  offset 64-128
• Base1.04 means dynamic range of 23,000
  (1.04256)
• A bigger base value means
• Higher dynamic range
• Lower resolution (Mach banding becomes noticeable)

26
E8R8G8B8 Encoding (HLSL)

• // an b
• define LOG(a, b) ( log((b)) / log((a)) )
• define EXP_BASE (1.06)
• define EXP_OFFSET (128.0)
• // Pixel Shader (6 instruction slots)
• // rgb already exposure-scaled
• float4 EncodeHDR_RGB_RGBE8(in float3 rgb)
• // Compute a common exponent
• float fLen dot(rgb.rgb, 1.0)
• float fExp LOG(EXP_BASE, fLen)
• float4 ret
• ret.a (fExp EXP_OFFSET) / 256
• ret.rgb rgb / fLen
• return ret

// More accurate encoding define EXP_BASE
(1.04) define EXP_OFFSET (64.0) // Pixel
Shader (13 instruction slots) float4
EncodeHDR_RGB_RGBE8(in float3 rgb) float4 ret
// Compute a common exponent // based on the
brightest color channel float fLen max(rgb.r,
rgb.g) fLen max(fLen, rgb.b) float fExp
floor( LOG(EXP_BASE, fLen) ) float4 ret
ret.a clamp( (fExp EXP_OFFSET) / 256, 0.0,
1.0 ) ret.rgb rgb / pow(EXP_BASE, ret.a
256 - EXP_OFFSET) return ret
27
E8R8G8B8 Decoding

• // Pixel Shader (5 instruction slots)
• float3 DecodeHDR_RGBE8_RGB(in float4 rgbe)
• float fExp rgbe.a 256 - EXP_OFFSET
• float fScale pow(EXP_BASE, fExp)
• return (rgbe.rgb fScaler)

// If R16F texture format is available, // you
can use texture to convert alpha to scale
factor float3 DecodeHDR_RGBE8_RGB(in float4
rgbe) // samp1D_Exp 1D float texture of
256x1 // pow(EXP_BASE, uCoord 256 -
EXP_OFFSET) float fScale tex1D(samp1D_Exp,
rgbe.a).r return (rgbe.rgb fScale)

• Encoding/decoding should be done using
  partial-precision instructions
• Rounding errors inherent in the texture format
  are much bigger

28
Rendering with Tone Mapping

• Glossy reflection material

float4 PS_GlossReflect(PS_INPUT_GlossReflect vIn)
COLOR0 float4 vDecalMap tex2D(samp2D_Decal,
vIn.tcDecal) float3 vLightMap
tex2D(samp2D_LightMap, vIn.tcLightMap) float3
vDiffuse vIn.cPrimaryDiffuse vLightMap
vIn.cOtherDiffuse vDiffuse vDecalMap //
HDR-decoding of environment map float3 vSpecular
DecodeHDR_RGBE8_RGB(
texCUBE(sampCUBE_EnvMap, vIn.tcReflect) )
float3 vRoughSpecular texCUBE(sampCUBE_DullEn
vMap, vIn.tcReflect) float fReflectance
tex2D( samp2D_Fresnel, vIn.tcFresnel ).a
fReflectance vDecalMap.a vSpecular
lerp(vSpecular, vRoughSpecular, fShininess)
float3 vLum lerp(vDiffuse, vSpecular,
fReflectance) // HDR tone-mapping
encoding float4 vOut vOut.rgb vLum / (vLum
1.0) vOut.a 0.0 return vOut
struct PS_INPUT_GlossReflect float2 tcDecal
TEXCOORD0 float3 tcReflect
TEXCOORD1 float2 tcLightMap
TEXCOORD2 float2 tcFresnel
TEXCOORD3 // Exposure-scaled lighting
results // Use TEXCOORD to avoid clamping
float3 cPrimaryDiffuse TEXCOORD6 float3
cOtherDiffuse TEXCOORD7
29
Generating Displayable Image

• Extract high-luminance regions
• Threshold 0.5-0.8
• Divide by (1 - Threshold) to normalize
• Generate glare
• Use an integer buffer to apply texture filtering
• Hopefully, a float buffer with filtering
• Generate a displayable image
• Add the glare to the frame buffer

30
Notes on DX9 Implementation

• High-precision buffers
• Consumes a lot of memory
• No blending capability
• Low-precision buffers
• Pixel shaders are expensive
• Consider fake techniques like DX8
• High performance
• Low memory consumption
• Very effective

31
Multiple Gaussian Filters

• Bloom generation
• A single Gaussian filter does not give very good
  results
• Small effective radius
• Not sharp enough around the light position
• Composite multiple Gaussian filters
• Use Gaussian filters of different radii
• Larger but sharper glare becomes possible

32
Multiple Gaussian Filters
33
Multiple Gaussian Filters
Original image
34
Multiple Gaussian Filters

• A filter of large radius is very expensive
• Make use of downscaled buffers
• A large radius means a strong low-pass filter
• Apply a blur filter to a low-res version of the
  image and magnify it by bilinear filtering ? The
  error is unnoticeable
• Change the image resolution rather than the
  filter radius
• 1/4 x 1/4 (1/16 the cost)
• 1/8 x 1/8 (1/64 the cost)
• 1/16 x 1/16 (1/256 the cost)
• 1/32 x 1/32 (1/1024 the cost)
• Even a large filter of several hundred pixels
  square can be applied very quickly

35
Applying Gaussian Filters to Downscaled Buffers
1/4 x 1/4 (256x192 pixels)
1/8 x 1/8 (128x96 pixels)
1/16 x 1/16 (64x48 pixels)
1/32 x 1/32 (32x48 pixels)
1/64 x 1/64 (16x12 pixels)
36
Bilinear Filtering and Composition

• Magnify them using bilinear filtering and
  composite the results
• The error is almost unrecognizable

37
Good Enough!
38
Notes on Filter

• Use high-precision formats for low-res buffers
• Dont take too many samples
• Very expensive, especially for high-res images

39
HDR in Games

• Should be appealing rather than accurate
• Accuracy is not important for players
• Even an inaccurate scene can be appealing
• Cost and performance
• The key is to understand the effects of HDR
• Devise a fake technique that is fast enough and
  produces believable results
• Accurate HDR rendering is still hard in games
• Use HDR only for effects that give a large impact
• Use sprites if you want to generate glare only
  for the sun, which is much faster and gives high
  quality results

40
HDR in Games

• Integer formats have very limited dynamic range
• Render in relative luminance space
• Apply exposure scaling during rendering
• High precision is not needed for the dark regions
  after exposure scaling
• As those regions remain dark in the final image
• Carefully choose the range to maximize effective
  precision

41
HDR in Games

• Future perspective
• All operations can be done in float
• Effective use of HDR
• Depth-of-field with the shape of aperture stop
• Motion blur with high luminances not clamped

42
References

• Reinhard, Erik, Mike Stark, Peter Shirley, and
  Jim Ferwerda, Photographic Tone Reproduction for
  Digital Images
• Mitchell, Jason L., Real-Time 3D Scene
  Post-Processing
• DirectX 9.0 SDK Summer 2003 Update, HDRLighting
  Sample
• Debevec, Paul E., Paul Debevec Home Page
• Kawase, Masaki, Frame Buffer Postprocessing
  Effects in DOUBLE-S.T.E.A.L (Wreckless)