Dal A Multimedia Software Library

1
Dalí A Multimedia Software Library

• Wei Tsang Ooi
• Brian Smith, Hsi Haye Chan, Matthew Chiu, Sugata
  Mukhopadhyay, Dan Rubinovitz, Jiesang Song, Steve
  Weiss.

2
Motivations
3
Current Software Solution
4
MPEG to RealVideo
OpenFile(f) GetNextFrame()
BlackBox
Real Video Encoder
RGB
MPEG
Real
5
MPEG to RealVideo
OpenFile(f) GetNextFrame()
YUV to RGB
RGB to YUV
RGB
YUV
YUV
Decoder
Encoder
compressed data
Real
MPEG
6
MPEG to RealVideo
YUV to RGB
RGB to YUV
YUV
Decoder
Encoder
compressed data
Real
MPEG
7
Decode I-Frame only
OpenFile(f) GetNextFrame()
BlackBox
?
RGB
MPEG
8
Decode I-Frame only
OpenFile(f) GetNextFrame()
?
YUV to RGB
RGB
YUV
Decoder
compressed data
MPEG
9
Decode I-Frame only
?
YUV to RGB
RGB
Decoder
I-frames
I Frame filter
compressed data
MPEG
10
Others Motivations..
BlackBox

• Read from network ?
• Tune I/O ?
• Run on handheld ?

11
Current Software Solution
12
Dalí is in between..
BlackBox
Dalí
C Source
13
Example
14
Example

• Process a subset of the gray scale version of
  each I-frame

15
MPEG structures
16
What we want to do

• parse sequence header

17
What we want to do

• parse sequence header
• while not eof do- find parse picture header

18
What we want to do

• parse sequence header
• while not eof do- find parse picture header-
  if it is an I-frame then - parse the picture
  into DCT domain - IDCT into spatial domain

19
What we want to do

• parse sequence header
• while not eof do- find parse picture header-
  if it is an I-frame then - parse the picture
  into DCT domain - IDCT into spatial domain -
  take the gray scale version of the frame and
  process it

20
Abstractions
21
Abstractions

• For input/output
• For representing elements in MPEG
• For storing images in DCT domain
• For storing images in spatial domain

22
BitStream BitParser
BitStream
File
23
MPEG Headers
MpegGopHdr
MpegSeqHdr
MpegPicHdr
Picture Data

• Parse
• Encode

• Find
• Skip
• Dump

24
Find
MpegGopHdr
MpegSeqHdr
MpegPicHdr
Picture Data

• Parse
• Encode

• Find
• Skip
• Dump

25
Skip
MpegGopHdr
MpegSeqHdr
MpegPicHdr
Picture Data

• Parse
• Encode

• Find
• Skip
• Dump

26
Dump
MpegGopHdr
MpegSeqHdr
MpegPicHdr
Picture Data

• Parse
• Encode

• Find
• Skip
• Dump

MpegGopHdr
27
Parse
MpegGopHdr
MpegSeqHdr
MpegPicHdr
Picture Data

• Parse
• Encode

• Find
• Skip
• Dump

width 320height 240pic rate 30 fps
etc
28
Encode
MpegSeqHdr

• Parse
• Encode

• Find
• Skip
• Dump

width 320height 240pic rate 30 fps
etc
29
ScImage
ScBlock - short dc char
index63 int ac63
represents 8x8 pixels
30
ByteImage
unsigned char 0..255
represents a pixel
31
The Code
32
Example
MpegSeqHdr seqHdr MpegSeqHdrNew () MpegPicHdr
picHdr MpegPicHdrNew () BitStream bs
BitStreamNew (65536)
BitStream
33
Example
BitParser bp BitParserNew () BitParserAttach
(bp, bs)
BitStream
BitParser
34
Example
BitStreamReadFromFile (bs, f, 0)
BitStream
File
BitParser
35
Example
MpegSeqHdrFind (bp)
MpegSeqHdr
MpegGopHdr
MpegPicHdr
BitParser
36
Example
MpegSeqHdrParse (bp, seqHdr) w (seqHdr-gtwidth
7)/8 h (seqHdr-gtheight 7)/8
MpegSeqHdr
MpegGopHdr
MpegPicHdr
BitParser
37
Example
ScImage y ScNew (w, h) ScImage u ScNew
(w/2, h/2) ScImage v ScNew (w/2, h/2)
y
u
v
38
Example
status MpegPicHdrFind (bp) while (status !
NOT_FOUND) MpegPicHdrParse (bp, picHdr)
status MpegPicHdrFind (bp)
39
Example
status MpegPicHdrFind (bp) while (status !
NOT_FOUND) MpegPicHdrParse (bp, picHdr)
if (picHdr-gttype I_FRAME)
status MpegPicHdrFind (bp)
40
Example
status MpegPicHdrFind (bp) while (status !
NOT_FOUND) MpegPicHdrParse (bp, picHdr)
if (picHdr-gttype I_FRAME)
MpegPicIParse ( bp,seqHdr,picHdr,
scy,scu,scv)
status MpegPicHdrFind (bp)
41
Example
status MpegPicHdrFind (bp) while (status !
NOT_FOUND) MpegPicHdrParse (bp, picHdr)
if (picHdr-gttype I_FRAME)
MpegPicIParse ( bp,seqHdr,picHdr,
scy,scu,scv)
BitStreamFill(bp, f) status
MpegPicHdrFind (bp)
42
Refill the BitStream
BitParser
43
Refill the BitStream
BitParser
44
Refill the BitStream
File
BitParser
45
Memory Mapped BitStream
bs BitStreamMMapReadNew(tennis.mpg)
File
46
Operate on a subset
47
Physical ByteImage
width 8 height 6x 0y 0 parentWidth
8 isVirtual 0firstByte
48
Virtual ByteImage
width 5 height 4x 1y 1 parentWidth
8 isVirtual 1firstByte
49
Example
ScImage virt ScClip (y,6,2,20,20)
6
20 blocks
50
Example
ScImage virt ScClip (y,6,2,20,20) ByteImage
byte ByteNew (160, 160) ScToByte (virt, byte)
6
20 blocks
51
Implementation
52
Implementation

• C library
• Tcl bindings available

53
Organization

• Code are organized into packages
• Basic
• ByteImage, ScImage, etc.
• MPEG
• MPEG-1 Video/Audio/System
• Vision
• Useful vision routines

54
Other Packages

• AVI
• WAVE
• JPEG
• GIF

55
Availability

• http//www.cs.cornell.edu/dali
• open source
• binaries for linux/win32/sunos
• examples
• documentations

56
The End