The Future of Graphics Hardware: Programmability

1
The Future of Graphics Hardware Programmability
David B. Kirk Chief Scientist
2
The Near Future
A Revolution in Graphics Hardware

• We now have full hardware OpenGL 1.3 and DirectX
  8 pipelines
• Moving from graphics accelerators to GPUs
• Next multiple programmable processors

3
Programmability Changes the World

• graphics hardware pipelines are becoming
  massively programmable
• will fundamentally change graphics
• allows hyper-realistic characters, special
  effects, and lighting and shading

4
Why are Movies and Special Effects Exciting and
Interesting?

• Suspension of Disbelief
• Something amazing is happening
• But, you believe it, because it is real
• Realistic and detailed characters
• Motion, and emotion
• Realistic and recognizable materials
• Chrome looks like chrome
• Skin looks like skin
• Action!

5
3D Graphics is about

• Animated films (Final Fantasy, Toy Story, etc.)
• Special Effects in live action movies (The
  Matrix)
• Interactive Entertainment (games! -)
• Computer Models of real world objects
• Or, objects that havent been invented yet
• Making reality more fantastic
• Making fantasies seem real

6
Live action sfx slide
Programmability Bridges the Gap between Movies
and Interactivity
7
The DirectX7 Graphics Pipeline (GeForce/GeForce2)
vertex transform and lighting
T L
setup
rasterizer
texture
per
-
pixel texture
blending
fb
anti-alias
8
3D Movie Special Effects Come to PC and Console
Graphics

• Lots of Geometry ? 1,000,000 polygons/frame!
• Geforce does this hardware Transform Lighting
• GeForce3 makes the pipeline programmable
• Lots of Lighting and Shading
• Geforce (year 2000)
• Hardwired vertex lighting
• Little shader programs run for every pixel
• Taking Shading to the next level (GeForce3)
• Powerful vertex programs run for every vertex
• Powerful shader programs run for every pixel

9
The DirectX8 Graphics Pipeline (GeForce3)
curved
surfaces
vertex
programmable
shaders
per
-
vertex processing
setup
rasterizer
tex
-
addr
shadows
ops
tex
3d
texture
programmable
blending
per
-
pixel shading
fb
antialias
10
Developers Have Been Asking For

• Complete control of the transformation and
  lighting hardware
• Complex vertex operations performed in hardware
• Custom vertex lighting
• Custom skinning and blending
• Custom texgen
• Custom texture matrix operations
• ltyour request goes heregt

11
Custom Substitute for Standard TL
Vertex Input

• Constant Memory

16 entries
Registers
ProgrammableVertex Processor
addr
addr
A0
data
data
128 instructions
12 entries
Vertex Output
96 entries
13 entries
12
What does it do?

• Per vertex calculation
• Processing of
• Colors true color, pseudo color
• 3D coordinates - procedural geometry, blending,
  morphing, deformations
• Texture coordinates texgens, set up for pixel
  shaders, tangent space bumpmap setup
• Fog elevation based, volume based
• Point size
• Vertex program accepts one input vertex,
  generates one output vertex

13
Aggressive use of Vertex Programs is SAFE and
IMPORTANT

• Its safe to use vertex programs pervasively
• Many hardware platforms have them
• Mainstream GPUs will have them this fall
• CPUs can emulate vertex shaders adequately, so
  CPU fallback is OK
• Its important to design content for vertex and
  pixel shaders
• Adopt vertex and pixel programming, and author
  content from the top-down
• Its much MUCH easier to scale down and fallback
  than to scale up

14
Programmable Shaders make possible materials l
ighting reflections shadows
15
Evolution of Hardware Shading

• Hardware Rasterizers and perspective-correct
  texture mapping (Voodoo / RIVA 128)
• Single Pass Multitexture (Voodoo2 / TNT / TNT2)
• Register Combiners a generalization of
  multitexture (GeForce 256)
• Per-pixel Shading (Geforce2 GTS)
• Programmable Hardware Pixel Shading (GeForce3)

16
Single Texture Programming Model
Source
Texture Blender
Texture 0
Result
17
Register Combiner Programming Model
Texture Combiner
Source(s)
Texture 0
Registers
Texture 1
Result(s)
18
Pixel Shading Pipeline
Triangle Rasterizer
8 Combiner Stages
ROP Frame buffer
Specular / fog Combiner
4 Pixel Shader Stages
19
Pixel Shaders
A pixel shader converts a set of texture
coordinates (s, t, r, q) into a color ( ARGB ),
using a shader program.

• Pixel shaders use
• Floating point math
• Texture lookups
• Results of previous pixel shaders

20
Simple Dependent Textures

• The results of one shading program can be
  interpreted as the texture coordinates for a
  subsequent texture lookup.
• AR ? ( s, t )
• GB ? ( s, t )
• Texture lookups become arbitrary
  functions.

Triangle Rasterizer
4 Pixel Shader Stages
21
Register Combiners / Texture Blending

• Strict superset of framebuffer alpha blending
  capabilities
• abcd
• Register-based programming
• All textures and colors available for each and
  every texture blending stage
• 8 Stages
• Signed color arithmetic

22
A processor model for Per-pixel Shading

• Computation primitives
• Texture addressing
• Cube maps
• Volume textures
• Comparison muxery
• Register combiners
• Vector math (dot3, reflection, etc)
• Hardware shading is now
• Programmable
• Extensible

23
Bigger Opportunities than TL and Multi-texture

• Can do vertex transformation, n-bone skinning,
  texture coord generation, custom fog, lighting,
  shadows, lightmaps, etc
• PLUS, A complex rendering technique can be
  "factored" into components executed on CPU,
  vertex shader, and pixel shader
• The true power of programmable vertex and pixel
  processing lies in the programmers ability to
  map more complex and varied algorithms onto the
  hardware

24
Instead of...
CPU

• CPU does
• Game code
• AI
• Physics
• Scene management
• GPU does
• TL
• Rasterization
• Texturing / Shading
• Drawing

Triangles Textures
Triangles Textures
GPU
Pixels
25
Think in terms of...
CPU

• Higher level algorithms are mapped across both
  CPU GPU
• CPU still does
• Game code, AI, Physics, Scene management
• GPU still does
• TL, Rasterization, Texturing / Shading, Drawing
• And, MORE

Partial Results
Partial Results
Data
GPU
Pixels
26
New Things to do with Vertex Programs and Pixel
Shaders

• (Automatic!!!) Shadow Volume Generation for
  Stencil Shadows
• Order-independent Transparency (Depth Peeling)
• Motion Blur / Depth of Field

27
(No Transcript)
28
Order-Independent Transparency Good Bad.
29
Order-Independent Transparency (Depth Peeling)

• The algorithm uses an implicit sort to extract
  multiple depth layers
• First pass render finds front-most fragment
  color/depth
• Each successive pass render finds (extracts) the
  fragment color/depth for the next-nearest
  fragment on a per pixel basis
• Use dual depth buffers to compare previous
  nearest fragment with current
• Second depth buffer used for comparison (read
  only) from texture

30
Layer 0
Layer 1
Layer 2
Layer 3
31
(No Transcript)
32
Using Vertex Programs/Pixel Shaders to do Motion
Blur Depth of Field
33
Future Graphics Pipeline
host interface
smart memory interface
per
-
primitive and
primitives
per
-
vertex programs
setup
rasterizer
tex
-
addr
shadows
ops
3d
tex
texture
per
-
pixel shading
blending
frame buffer
anti-alias
34
Acknowledgements

• Thanks to John Carmack, Sim Dietrich, Matthew
  Papakipos, Simon Green, Cem Cebonoyan, Erik
  Lindholm, Doug Rogers, Cass Everitt, Rui Bastos,
  Mark Kilgard, Greg James, Matthias Wloka, (and
  others I forgot!) for contributing ideas, slides,
  demos, and images.
• Demo/Example source code and more whitepapers can
  be found at
• http//www.nvidia.com/developer

35
Questions?