CrystalMobile: multimedia framework for mobile platforms

1
CrystalMobile multimedia framework for mobile
platforms
www.crystalmobile.com
Kim A. BondarenkoCrystal Reality LLC, CSA
kim_at_crystalreality.com
Computer Technology DepartmentSaint-Petersburg
State University ofInformation Technology,
Mechanics and Optics,
Russia
2
Introduction

• Application of video technologies on mobile
  platforms is very wide.
• - entertainment and adult industry
• - information communications
• - collecting of private videos
• - etc.

3
Problems

• Mobile platforms have not enough memory space for
  full-quality videos and GPRS channels have no
  required bandwidth capacity.
• CPUs are too slow for application of complex
  algorithms.

4
Why to compress video?

• Uncompressed video occupies huge amounts of data.
  For example, for storing of one minute quality
  video with 320x240 resolution with 15 frames per
  second it is needed 320x240x3x15x60 200 MBytes
  of data.
• Powerful CPU platforms allowed the implementation
  of complex algorithms giving 100-500 compression
  ratio.

5
Common ideas of video compression

• Superfluous information.
• Neglect of the insignificant details for
  human-eye.

6
Superfluous details that are considered in CM
Video

• The low space correlation of picture. In any
  image, near lying pixels have little difference
  between them.
• The low time correlation of picture. In any video
  clip there are a lot of close frames with little
  difference between them.

7
Low space correlation
Normal Picture
Half resolution
8
Low time correlation

• Normal Picture

Difference between two following frames
9
The details that are insignificant for human-eye

• The human eye almost does not analyze the color,
  but very good differentiate brightness of the
  image.
• Insignificant noisy movements of image parts are
  mostly invisible for human-eye.
• The disturbances on noisy image part are weakly
  differentiated.

10
Chroma vs color
1/8 of color
1/8 of chroma
11
Insignificant movements

• Normal picture

The picture with random distortion
12
The disturbances on noise

• Normal Picture

Disturbances on noise (red)
13
YUV Colorspace

• During the translation into YUV-color every pixel
  (RGB vector) is multiplied by 3x3 matrix.
• As the result, the Y brightness channel and two
  U and V color compositions are determined.
• CME uses confidant transformations with integers
  it gives significant performance improvements
  on most platforms.
• Meanwhile, the resolution of U and V channels is
  divided by two along each axis. As human eye does
  not differentiate colors as good as brightness,
  its possible to reduce the detalization of color
  planes twice.

14
Low time correlation of the video

• B, I and P frames.
• The definition of context change and the division
  of video streams into blocks (sandwiches). The
  encoding is done by sandwiches of 1-3 frames.

15
B, I and P frames
16
Frame processing. Macroblocks.

• Every frame in sandwich is divided into 16x16
  blocks (macroblock).
• Every macroblock consists of 4 brightness blocks
  of 8x8 and two color blocks 8x8, since the color
  resolution of the frame is divided by two.
• Most processing methods operate on macroblocks.

17
Motion compensation

• Motion compensation is intended for prediction of
  the motion on the picture. It uses low time
  correlation between close frames.
• CME Video provides motion compensation of 16x16
  blocks with bi-linear interpolation.

18
Motion compensation
Frame from the stream
Motion vectors
19
I-frames

• Encoder analyses whole frame to find images
  cannot be predicted from the history. These
  frames should be stored as independent data
  blocks, I-frames.
• Positioning is precise to I-frames, so I-frames
  periodicity should be at least one I-frame for 30
  seconds.

20
Wavelets (DWT)

• The images are passed the 2D wavelet
  transformation. Every frame is passed by
  bicubic/bilinear wavelet transformation. The main
  advantage of the format is about wavelet
  transformation is applied to separated images in
  the picture.
• The smoothing is done by bicubic/bilinear
  interpolation, there are no effects of DCT. The
  pictures on the next slide show the difference
  between the losses using DCT (left) and bicubic
  wavelet composition(right) using the same bitrate.

21
DCT vs Wavelets

• DCT

Wavelets
22
Quantization

• The sandwich is passed through the process of
  quantization every point is rapidly divided by
  the definite number from vector of quantization.
  Every element of vector of quantization is
  correspondent to some frequency of wavelet
  composition. On this step, the most loss is
  occurring

23
Storing the coefficients

• Each frame is passed to zero-quantors extraction
  by applying quadro-tree processing. After
  quantization most number of small coefficients is
  zeroed, that is why quadro-tree processing is
  very efficient on this step its the main part
  of compression
• After group encoding, 16 data blocks of each
  frame are compressed using Huffman method.

24
The current status of CME Engine

• Video codec of CME Engine is done.
  High-performance audio codec is under
  development.
• The current implementation uses fixed-point ANSI
  C without any proprietary libraries.

25
Symbian 6.1 platform

• CME Engine perfectly runs on Symbian 6.1 platform
  (Nokia Series 60 phones). There is a room for
  tuning video parameters, but overall playback is
  good.
• Release version of player software for Symbian
  platform is ready and was offered to public on
  01.10.2003.

26
Player is working on Nokia3650

• Encoding on PC
• Realtime playback on Nokia Series60 family
• High quality of the video

27
Results comparisons

• CM Video has better quality than H.263 and MPEG1
  with the same bit rate.
• CM Video technical parameters
• - 176x144 10fps on Nokia3650/7650
• - 1 Mb per minute bitstream for good quality.

28
Standard video formats players

• MPEG1. Unusable for mobiles.
• H.263 Video Recorder for Nokia from Hantro Oy and
  Emuzed.
• Real Video. Real Player for Nokia from Real
  Networks.

29
Crystal Reality LLC www.crystalreality.com

• Founded in March 2003
• 900.000 downloads of Crystal Player Professional
• Very strong user community in Europe, Russia and
  USA
• Crystal Mobile Engine was developed during July
  2003 Sept 2003
• 3 fulltime and 2 part time developers are now
  employed
• Develops mobile technologies for the future