LowLevel Plumbing for Media Integration

1
Low-Level PlumbingforMedia Integration

• Turner Whitted
• Microsoft Research

2
Outline

• Part I Implementation chronology
• Then, Now, Then again
• aka Wheel Of Reincarnation
• Part II Architectural musing
• Perceptual/content requirements
• Data paths, data types
• Not about programming models

3
Starting points ES Frame Buffer

• NO fixed function units
• mCode for basic logic
• mController programmable by designer only
• Treated as peripheral

Ref Kajiya, J.T., Sutherland, I.E., and Cheadle,
E.C., "A Random-Access Video Frame Buffer,"
Proceedings of the Conference on Computer
Graphics, Pattern Recognition, and Data
Structure, UCLA Extension, Los Angeles,
California, May 14-16, 1975
4
Starting points Ikonas RDS3000

• Single, 32-bit wide datapath based on 2901
  bit-sliced DSP
• Programmed in C (Garys Ikonas Assemler)
• NO fixed function units
• Bound later memory mapped to single application

Ref N. England, A graphics system architecture
for interactive application-specific display
functions, IEEE CGA, pp. 60-70, Jan 1986.
5
Starting points Pixar CHAP

• SIMD processors
• Loops, conditional execution
• Focus on parallel programming issues

Ref Adam Levinthal and Thomas Porter, Chap A
SIMD Graphics Processor, Proceedings of SIGGRAPH
84, (18) 3, July 1984, pp. 77 82.
6
The pipeline

• Mostly Fixed structure, programmable nodes
• Vector graphics legacy

7
Structural evolution

• Texture engine with remnants of line drawing DNA

8
Performance

• Ikonas RDS3000 (1980)
• 20 MB/s processor to memory
• 20 MIPS equiv.
• Pixar CHAP (1984)
• 240 MB/s processor to memory (P-bus)
• 64 MIPS peak
• ATI 9700 (2002)
• 20800 MB/s chip to memory
• 8400 MIPS

9
API abstractions (OpenGL, DX)

• Fixed structure pipeline accessed through API
• API tracks hardware through several generations
  while maintaining consistency
• Integrated with mainstream computing
• Easy to program
• Single largest key to commercial success of
  graphics systems and applications

10
Complete the circle

• FPGAmemory
• NO fixed function
• Programmable in C (Verilog actually)
• Good for simple prototyping
• No API
• No device drivers
• No SDK

11
Part II Going forward

• Motivation
• Weve tacked every imaginable feature onto what
  was initially a line drawing pipeline
• Its time to start over
• The window of opportunity is wide open

12
Going forward with graphics processors

• Alternatives

3D Raster Graphics
13
Integrated media(partial illustration)
Concentric mosaics
Image centric
Geometry centric
Sprites with depth
View-dependent geometry
View-dependent texture
Light field
Lumigraph
Fixed geometry
Polygon rendering texture mapping
Interpolation
Warping
14
Geometry Image
3D geometry
completely regular sampling
geometry image257 x 257 12 bits/channel
Ref X. Gu, S. Gortler, H. Hoppe, Geometry
images, ACM Transactions on Graphics 21(3)
355-361 (2002)
15
From first principles function
U-VWL Content
Line drawing
Display Processor
Display Device
Video
Still photos
Animated 3D shapes
Text
Ultra-Vast Wasteland
16
3D text experiment

• Extend IBR/volume rendering to text
• Superior image reconstruction
• Higher image quality than mip-mapped texture

Olynyk, Mitchell, Snyder, MSR
17
Generic physical blocks

• General purpose front end
• Fixed function back end

Read
Mapper
Cache
Common
Memory
Recon-
Write
Struction/ Filtering
Cache
Ref T. Whitted, Overview of IBRHardware and
Software Issues, ICIP 2000.
18
From first principles implementation

• Dont count on quantum GPUs soon stick to CMOS
  digital logic
• Count on CAD more than feature size
• Heat is the enemy
• The economy of commodity DRAM is hard to beat
• But there is huge performance pressure on DRAM
• Designers are restricted only by a lack of
  experience

Content
19
Design challenges

• Essence of the problem
• We dont have a function to implement
• We must design specifically for unknown methods
• Brute force is prohibited
• Feeds and speeds What do we know?
• Regulated by content
• But we rarely turn content conventions into
  quantitative measures
• Limited by perception
• Which we dont fully understand
• Function of representation
• If we dont know the representation, we dont
  know the flows
• In my groups research we limit flow to HW
  sanity and then work backwards

20
Summary

• Graphics hardware has nearly completed the
  circuit back to its starting point
• Flexible, powerful, programmable
• Media processing requirements extend beyond the
  classic 3D pipeline
• Unusual window of opportunity
• to match architecture with a broader range of
  applications and content

21
(No Transcript)