iPACKMAN

1
iPACKMAN

• High-Quality, Low Complexity Texture Compression
  for Mobile Phones

Jacob StromEricsson
Tomas Akenine-MollerLund University
2
Outline

• Motivation
• Design Goals
• Basic Idea
• PACKMAN the old version
• Differential Coding
• Compression
• Results

3
Why 3D Graphicson a Mobile Phone?

• Man-Machine Interfaces

• Screen Savers

• Games

• Maps, Messaging, Browsing and more

4
Why is 3D Graphics Hardon a Mobile Phone?

• Limited resources
• Small amount of memory
• Little memory bandwidth
• Little chip area for special purpose
• Powered by batteries

5
Texture Compression Helps

• Small amount of memory
• More texture data can fit in the limited amount
  of memory
• Little memory bandwidth
• More texturing possible for same amount of
  bandwidth
• Little chip area for special purpose
• A texture cache using compressed data can be made
  smaller
• Powered by batteries
• Reduced bandwidth means lower energy consumption
• from PACKMAN to iPACKMAN complexity

6
Previous Work
7
Previous Work
8
Previous Work
9
Previous Work
10
Design Goals

• Low Decompression Complexity
• 8 parallel units needed for one trilinear
  operation per clock
• mobile devices have very little surface area to
  spare
• High Image Quality
• Should be on par with, or better than, industry
  standard DXTC at the same bit rate
• Should be system friendly
• You want to be able to store compressed data in
  the cache, and that means that the decompression
  needs to be simple and fast.
• No indirect data such as a color palette that
  increases latency
• No data from adjacent blocks should be needed
• For systems without a texture cache, a block size
  of 32 bits would be preferable, matching the size
  of the bus.

11
Basic Idea PACKMAN

• The Human Visual System is more sensitive to
  luminance than to chrominance
• In video and still image coding, chrominance
  information is most often subsampled in the x-
  and y- direction (MPEG, JPEG, H263, H264 etc).
  Loosely speaking, chrominance is defined per 2x2
  block.
• PACKMAN has basically only one color per 2x4
  block. The rest is luminance information
• Code each 2x4 block using 32 bits

12
Basic Idea PACKMAN

• Use only 12 bits to specify a base color for a
  2x4 block

12-bit base color
13
Basic Idea PACKMAN

• Use only 12 bits to specify a base color for a
  2x4 block
• Modify the luminance for each pixel in the block

12-bit base color
per-pixel luminance
14
Basic Idea PACKMAN

• Use only 12 bits to specify a base color for a
  2x4 block
• Modify the luminance for each pixel in the block

12-bit base color
per-pixel luminance
resulting image
15
Luminance modification

• Only one value per pixel needed to specify
  luminance

16
Luminance modification

• Only one value per pixel needed to specify
  luminance

R 17 G 34 B 204
Base Color
17
Luminance modification

• Only one value per pixel needed to specify
  luminance

R 17 G 34 B 204
110 110 110
Base Color
Add same value
18
Luminance modification

• Only one value per pixel needed to specify
  luminance

R 17 G 34 B 204
110 110 110
127 144 255 (after clamping)
Resulting Color
Base Color
Add same value
19
How to Specify Luminance

• Two bits per pixel are used to specify the
  luminance. Modifier is one out of four values.
• Problem Small values -8, -2, 2, 8

20
How to Specify Luminance

• Two bits per pixel are used to specify the
  luminance. Modifier is one out of four values.
• Problem Small values -8, -2, 2, 8
• smooth transitions OK

21
How to Specify Luminance

• Two bits per pixel are used to specify the
  luminance. Modifier is one out of four values.
• Problem Small values -8, -2, 2, 8
• smooth transitions OK
• sharp edges bad

22
How to Specify Luminance

• Two bits per pixel are used to specify the
  luminance. Modifier is one out of four values.
• Problem Small values -8, -2, 2, 8
• smooth transitions OK
• sharp edges bad
• Big values -255, -127, 127, 255
• sharp edges OK

23
How to Specify Luminance

• Two bits per pixel are used to specify the
  luminance. Modifier is one out of four values.
• Problem Small values -8, -2, 2, 8
• smooth transitions OK
• sharp edges bad
• Big values -255, -127, 127, 255
• sharp edges OK
• smooth transitions bad

24
How to Specify Luminance

• Two bits per pixel are used to specify the
  luminance. Modifier is one out of four values.
• Problem Small values -8, -2, 2, 8
• smooth transitions OK
• sharp edges bad
• Big values -255, -127, 127, 255
• sharp edges OK
• smooth transitions bad
• Solution Codebook of tables, one/block.

25
Modifier Codebook

• We created the codebook from random numbers by
  minimizing the error for a set of images.

26
Modifier Codebook

• We created the codebook from random numbers by
  minimizing the error for a set of images.

27
Modifier Codebook

• We created the codebook from random numbers by
  minimizing the error for a set of images.
• Simulated Annealing
• Modified version of LBG-algorithm
• Symmetry was enforced to reduce on-chip memory

28
Modifier Codebook

• We created the codebook from random numbers by
  minimizing the error for a set of images.
• Simulated Annealing
• Modified version of LBG-algorithm
• Symmetry was enforced to reduce on-chip memory

29
Modifier Codebook

• We created the codebook from random numbers by
  minimizing the error for a set of images.
• Simulated Annealing
• Modified version of LBG-algorithm
• Symmetry was enforced to reduce on-chip memory
• This way only half the table needed to be stored
  on chip.

30
Modifier Codebook

• We created the codebook from random numbers by
  minimizing the error for a set of images.
• Simulated Annealing
• Modified version of LBG-algorithm
• Symmetry was enforced to reduce on-chip memory
• This way only half the table needed to be stored
  on chip.

31
Modifier Codebook

• We created the codebook from random numbers by
  minimizing the error for a set of images.
• Simulated Annealing
• Modified version of LBG-algorithm
• Symmetry was enforced to reduce on-chip memory
• This way only half the table needed to be stored
  on chip.

• The same table is used for all textures can be
  hardwired into the logic.

32
Bit outline

• First 12 bits is RGB444 which gives the base
  color for the entire block.

153 153 85
9
9
5
12 bit RGB444
9956b59f
33
Bit outline

• Next 4 bits selects a table from a set of 16
  tables

9
9
5
6
12 bit RGB444
34
Bit outline

• Next 4 bits selects a table from a set of 16
  tables.

9
9
5
6
12 bit RGB444
35
Bit outline

• The next 2 bits modifies the first pixel
  according to the table

9
9
5
6
10
12 bit RGB444
36
Bit outline

• The next 2 bits modifies the first pixel
  according to the table and so on.

9
9
5
6
11
10
12 bit RGB444
37
Bit outline

• The next 2 bits modifies the first pixel
  according to the table and so on.

9
9
5
6
11
10
01
12 bit RGB444
38
Bit outline

• The next 2 bits modifies the first pixel
  according to the table and so on.

9
9
5
6
11
10
01
01
12 bit RGB444
39
Bit outline

• The next 2 bits modifies the first pixel
  according to the table and so on.

9
9
5
6
11
10
01
01
10
12 bit RGB444
40
Bit outline

• The next 2 bits modifies the first pixel
  according to the table and so on.

9
9
5
6
11
10
01
01
10
01
12 bit RGB444
41
Bit outline

• The next 2 bits modifies the first pixel
  according to the table and so on.

9
9
5
6
11
10
01
01
10
01
11
12 bit RGB444
42
Bit outline

• The next 2 bits modifies the first pixel
  according to the table and so on.

9
9
5
6
11
10
01
01
10
01
11
11
12 bit RGB444
43
Simple Decompression

• The correct texel is selected

44
Simple Decompression

• The correct texel is selected
• The modifier value is looked up

45
Simple Decompression

• The correct texel is selected
• The modifier value is looked up
• The base color is extended to 24 bits

46
Simple Decompression

• The correct texel is selected
• The modifier value is looked up
• The base color is extended to 24 bits
• The modifier value is added

47
Simple Decompression

• The correct texel is selected
• The modifier value is looked up
• The base color is extended to 24 bits
• The modifier value is added
• The result is clamped

48
PACKMAN Flaws

• PACKMAN was of very low complexity, but
• 2 dB worse than DXTC in terms of Peak Signal to
  Noise Ratio (PSNR)
• Suffered from chrominance banding / block
  artifacts due to low color resolution (RGB444)

original
PACKMAN
49
PACKMAN Flaws

• PACKMAN was of very low complexity, but
• 2 dB worse than DXTC in terms of Peak Signal to
  Noise Ratio (PSNR)
• Suffered from chrominance banding / block
  artifacts due to low color resolution (RGB444)

original
PACKMAN
50
Enhancing the Chrominancewould increasing the
block size help?

• By coding 4x4 blocks instead of 2x4 blocks,
  spatial redundancy could be better exploited.
• The small block size of 32 bits would be lost,
  but that was only beneficial in systems without a
  texture cache, so it was not a big loss.

51
Neighboring Base ColorsQuite Similar
max(R1-R2, G1-G2, B1-B2)(in RGB555)
88 within interval -4, 3
52
Differential Encoding
R
table
G
B
11
10
01
01
10
01
11
11
53
Differential Encoding
R
table
G
B
54
Differential Encoding
left block

• Instead of coding the left block with RGB444

left block
R
table
G
B
55
Differential Encoding
left block
right block

• Instead of coding the left block with RGB444 and
  the right with RGB444

left block
right block
R
R
table
table
G
B
G
B
56
Differential Encoding
left block
right block

• Instead of coding the left block with RGB444 and
  the right with RGB444
• We code the left with RGB555

left block
right block
R
R
table
table
G
B
G
B
R
G
B
table bits
57
Differential Encoding
left block
right block

• Instead of coding the left block with RGB444 and
  the right with RGB444
• We code the left with RGB555 and the right with
  dR dG dB 333.

left block
right block
R
R
table
table
G
B
G
B
R
dR
G
dG
B
dB
table bits
58
Differential Encoding
left block
right block

• However, in 10 of the cases, the left and right
  blocks will differ too much in color for
  differential coding.

R
dR
G
dG
B
dB
table bits
59
Differential Encoding
left block
right block

• However, in 10 of the cases, the left and right
  blocks will differ too much in color for
  differential coding.

R
dR
G
dG
B
dB
table bits
60
Differential Encoding
left block
right block

• However, in 10 of the cases, the left and right
  blocks will differ too much in color for
  differential coding.
• For these blocks, we fall black to individual
  coding.

R
dR
G
dG
B
dB
table bits
G
B
R
G
B
R
table bits
61
Differential Encoding
left block
right block

• However, in 10 of the cases, the left and right
  blocks will differ too much in color for
  differential coding.
• For these blocks, we fall black to individual
  coding.
• We thus need an extra bit to signal if we are in
  differential mode or not.

555differential or 444
R
dR
G
dG
B
dB
table bits
G
B
R
G
B
R
table bits
62
Differential Encoding
left block
right block

• However, in 10 of the cases, the left and right
  blocks will differ too much in color for
  differential coding.
• For these blocks, we fall black to individual
  coding.
• We thus need an extra bit to signal if we are in
  differential mode or not.
• We must take that bit from somewhere.

555differential or 444
R
dR
G
dG
B
dB
table bits
G
B
R
G
B
R
table bits
63
Differential Encoding

• By shrinking the codebook from 16 entries to 8,
  we can save one bit on each of the table code
  words.

R
dR
G
dG
B
dB
table bits
G
B
R
G
B
R
table bits
64
Differential Encoding

• By shrinking the codebook from 16 entries to 8,
  we can save one bit on each of the table code
  words.

R
dR
G
dG
B
dB
table bits
G
B
R
G
B
R
table bits
65
Differential Encoding

• By shrinking the codebook from 16 entries to 8,
  we can save one bit on each of the table code
  words.

R
dR
G
dG
B
dB
table bits
G
B
R
G
B
R
table bits
66
Differential Encoding

• By shrinking the codebook from 16 entries to 8,
  we can save one bit on each of the table code
  words.
• That creates room for an extra bit.

flipped or non flipped
R
dR
G
dG
B
dB
table bits
G
B
R
G
B
R
table bits
67
Differential Encoding

• By shrinking the codebook from 16 entries to 8,
  we can save one bit on each of the table code
  words.
• That creates room for an extra bit.
• The new bit determines if the 2x4 blocks are
  vertically or horizontally oriented.

flipped or non flipped
R
dR
G
dG
B
dB
table bits
G
B
R
G
B
R
table bits
68
Results improved PACKMANor iPACKMAN for short

• Much less chrominance banding
• Jumps 2.5 dB in PSNR overall

PACKMAN
iPACKMAN
69
Compression
70
Simple Compression

• Find the base color for the two blocks
• See if their RGB555 representations can be coded
  differentially, else use RGB444 representations
• Exhaustively try all tables and all per-pixel
  modifiers
• Very fast around 60 milliseconds for a 128 x
  128 texture

71
Finding the Base Color

• We start with the average color of the block.
• Going from RGB888 to RGB555, we go from many
  points in RGB space

G
R
72
Finding the Base Color

• We start with the average color of the block.
• Going from RGB888 to RGB555, we go from many
  points in RGB space
• to a few.

G
R
73
Finding the Base Color

• We start with the average color of the block.
• Going from RGB888 to RGB555, we go from many
  points in RGB space
• to a few.
• Ordinary quantization just chooses the closest
  one.

G
R
74
Finding the Base Color

• We start with the average color of the block.
• Going from RGB888 to RGB555, we go from many
  points in RGB space
• to a few.
• Ordinary quantization just chooses the closest
  one.

G
R
75
Finding the Base Color

• However, the per-pixel luminance modification can
  compensate for errors in the direction (1,1,1)
• We can reach (almost) all points on the dotted
  lines

G
R
76
Finding the Base Color

• However, the per-pixel luminance modification can
  compensate for errors in the direction (1,1,1)
• We can reach (almost) all points on the dotted
  lines

G
R
77
Finding the Base Color

• However, the per-pixel luminance modification can
  compensate for errors in the direction (1,1,1)
• We can reach (almost) all points on the dotted
  lines
• We should also try the point closest to this line.

G
R
78
Finding the Base Color

• However, the per-pixel luminance modification can
  compensate for errors in the direction (1,1,1)
• We can reach (almost) all points on the dotted
  lines
• We should also try the point closest to this
  line.
• This is fast 120 ms per 128x128 texture.

G
R
79
More Thorough Compression

• A more thorough compression can try all the
  neighboring colors /- 3 steps from the average
  color.
• Medium-speed, around 20 seconds per 128x128
  texture

G
R
80
Even slower Compression

• Our slowest version will search 5 steps out from
  the average color.
• About 7 minutes per 128x128 block
• Exhaustive compression is also possible (as
  proven by Eric Fausett)

G
R
81
Compression

• The difference between the fastest and best
  compression methods is about 1.3 dB

PSNR
1.3 dB
Slow (7 min)
Exhaustive
Average (60 ms)
Fast (120 ms)
Medium (20s)
82
Decompression
83
Simple HW for decompression
84
Simple HW for decompression
6 adders
85
Simple HW for decompression
6 adders a few MUXes
86
Simple HW for decompression
6 adders a few MUXes 1 LUT
87
Results
88
Quality Measure

• One common measure is the Root Mean Square Error
  (RMSE) measure.

89
Quality Measure

• One common measure is the Root Mean Square Error
  (RMSE) measure.

• A variant of this is Peak Signal to Noise ratio
  (PSNR)

90
Quality Measure

• One common measure is the Root Mean Square Error
  (RMSE) measure.

• A variant of this is Peak Signal to Noise ratio
  (PSNR)

• Usually, 0.25 dB difference is a visible
  difference.

91
ResultsWe have compared against the following
systems

92
ResultsWe have compared against the following
systems

93
ResultsWe have compared against the following
systems

The 7 images used were Lena
Lorikeet Kodim1-5
94
ResultsWe have compared against the following
systems

95
Results for iPACKMANWe have compared against the
following systems

96
Strengths

• The strengths can most easily be seen in areas
  with fine variations in luminance.

original
97
Strengths

• The strengths can most easily be seen in areas
  with fine variations in luminance.

original
S3TC/DXTC
98
Strengths

• The strengths can most easily be seen in areas
  with fine variations in luminance.

original
S3TC/DXTC
iPACKMAN
99
Strengths

• The strengths can most easily be seen in areas
  with fine variations in luminance.

original
S3TC/DXTC
iPACKMAN
100
Strengths

• The strengths can most easily be seen in areas
  with fine variations in luminance.

original
S3TC/DXTC
iPACKMAN
101
Strengths

• The strengths can most easily be seen in areas
  with fine variations in luminance.

original
S3TC/DXTC
iPACKMAN
102
Weaknesses

• When there are more than two colors of different
  chrominance in a 2x4 block, iPACKMAN has
  problems.
• Such artifacts are especially visible when the
  two colors have similar luminance.

103
Weaknesses

• When there are more than two colors of different
  chrominance in a 2x4 block, iPACKMAN has
  problems.
• Such artifacts are especially visible when the
  two colors have similar luminance.

original
104
Weaknesses

• When there are more than two colors of different
  chrominance in a 2x4 block, iPACKMAN has
  problems.
• Such artifacts are especially visible when the
  two colors have similar luminance.

original
S3TC/DXTC
105
Weaknesses

• When there are more than two colors of different
  chrominance in a 2x4 block, iPACKMAN has
  problems.
• Such artifacts are especially visible when the
  two colors have similar luminance.

original
S3TC/DXTC
iPACKMAN
106
Error Metric

• When selecting one way to compress a block over
  another, an error metric is used to tell which is
  better.
• An obvious error metric to use is to sum the
  squared error over the block
• e2 (R-R)2 (G-G)2 (B-B)2
• However, since the eye is more sensitive to
  errors in green than in red and blue, it makes
  sense to add more weight to the green component
• e2 wR2 (R-R)2 wG2 (G-G)2 wB2 (B-B)2

107
Error Metric

• Original

108
Error Metric

• Original

109
Error Metric

• Original
• Non-perceptualError Metric

110
Error Metric

• Original
• Non-perceptualError Metric
• PerceptualError Metric

111
Error Metric

• Original
• Non-perceptualError Metric
• PerceptualError Metric

112
Error Metric

• Original
• Non-perceptualError Metric
• PerceptualError Metric

113
Error Metric

• Original
• Non-perceptualError Metric
• PerceptualError Metric

114
Error Metric

• Original
• Non-perceptualError Metric
• PerceptualError Metric

115
Normal Maps

• We have tried to compress tangent space normal
  maps with iPACKMAN, with mixed results.
• For high-frequency normal maps, such as the one
  to the right, it works fine.
• However, for slowly varying normal maps, banding
  becomes a problem.
• The result is that iPACKMAN should not be used
  for normal maps, except in high-frequency cases.

original
iPACKMAN
116
Alpha (RGBA) Textures

• Currently, iPACKMAN cannot handle RGBA textures.
• In DXT2-5, the alpha (A-component) is treated
  separately from the RGB components

117
Alpha (RGBA) Textures

• Currently, iPACKMAN cannot handle RGBA textures.
• In DXT2-5, the alpha (A-component) is treated
  separately from the RGB components

64 bits RGB
64 bits A
64 bits RGB
64 bits A

• A similar method could be applicable for iPACKMAN
• It is also possible to cram all four components
  (R, G, B and A) into 64 bits, but this will
  compromise image quality.

118
Adoption

• The Khronos organization has adopted iPACKMAN
  under the name Ericsson Texture Compression
  (ETC) through an OES extension for OpenGL ES.
• Compression software source code will soon be
  available for download.
• Independent hardware vendors have started
  implementing ETC.

119
Summary

• We have presented a High-Quality, Low Complexity
  texture compression system.
• Quality is slightly better (0.4 dB) than
  state-of-the-art compression technology (DXTC) at
  the same bit rate (4 bits per pixel).
• Khronos has adopted the technology through an OES
  extension to their popular OpenGL ES API.
• Future work includes improving on the weak
  points, as well as handle alpha and normal data
  better.

120
Thank you
121
(No Transcript)
122
PACKMAN Flaws

• PACKMAN was of very low complexity, but
• 2 dB worse than S3TC in terms of Peak Signal to
  Noise Ratio (PSNR)
• Suffered from chrominance banding / block
  artifacts due to low color resolution (RGB444)

original
PACKMAN
123
Results improved PACKMANor iPACKMAN for short

• Much less chrominance banding
• Jumps 2.5 dB in PSNR overall

PACKMAN
iPACKMAN