Real-time Video Effects Using Programmable Graphics Cards

1
Real-time Video Effects Using Programmable
Graphics Cards

• Master of Science Thesis
• Klas Skogmar
• klas_at_skogmar.com

2
Introduction

• Graphics cards have much computing power but are
  only used by 3D applications
• Video and image editing programs often needs to
  perform per pixel operations that are ideal for
  graphics cards

3
Problems

• How can graphics cards be used for displaying and
  altering video?
• What kinds of effects are suitable for a 3D
  environment?
• Is it possible to speed up effects by using the
  graphics card?
• How are the transfer rates affecting the system?
• How much disc space is required and how does this
  affect the system?

4
Programmable graphics cards

• Graphics cards can do massive parallel
  calculations
• 3D developers required more flexibility
• 3D card producers introduced Vertex shaders and
  Pixel shaders
• Makes it possible to program graphics cards in a
  similar way to processors

5
Effects on graphics cards

• Color correction
• Masking
• Color keying
• Compositing
• Transitions
• Painting
• 3D effects

6
High resolution challenges

• HDTV standard 1080p requires throughput of 156
  Mb/sec uncompressed. Current broadcasts takes
  approximately 30 Mb/sec uncompressed

• Challenges
• Disc space
• Transfer speed
• Other problems (e.g. dynamic range)

7
Working practice

• Real-time
• Expensive
• Scanning film
• Many formats
• High resolution
• Proxies and AOI
• Specialized hardware

8
Available technologies

• Dedicated hardware
• OpenGL (Imaging subset)
• DirectX (Direct3D and DirectShow)
• Quicktime
• ATIs and Nvidias

9
DirectShow

• Microsofts media development environment
• Built up of filters each with its own task
• Filters are assembled in a filter graph
• Is supported by many video and image formats

10
My program

• Built using MFC
• Uses a DirectShow filter
• Shows how DirectShow can be used

11
Performance

• My program was slow
• ATI has developed a program that is fast (uses
  VMR-9)
• Serious Magics Texture Download Benchmark
• Matt Craigheads program

12
Performance

• Results
• ATIs program shows the potential adding
  effects do not give visible frame drops
• The other programs shows the transfer speed from
  the graphics card to the processor it is not
  enough, but driver updates enhances the
  performance

13
Conclusions

• It is already a technique that can be used!
• Drivers has to be enhanced to allow the
  modifications to be written back to memory
• Graphics cards are more flexible than dedicated
  hardware
• Effects have to be rewritten
• Many effects are more suitable to do using
  graphics cards

14
Conclusions

• The transfer to and from disc can be a problem
• There is a need for modules (e.g. DirectShow
  filters), that makes it easy to use programmable
  graphics cards