Introduction to Coding Theory

1
Introduction to Coding Theory

• Rong-Jaye Chen

2
Outline

• 1 Introduction
• 2 Basic assumptions
• 3 Correcting and detecting error patterns
• 4 Information rate
• 5 The effects of error correction and detection
• 6 Finding the most likely codeword transmitted
• 7 Some basic algebra
• 8 Weight and distance
• 9 Maximum likelihood decoding
• 10 Reliability of MLD
• 11 Error-detecting codes
• 12 Error-correcting codes

3
Introduction to Coding Theory

• 1 Introduction
• Coding theory
• The study of methods for efficient and accurate
  transfer of information
• Detecting and correcting transmission errors
• Information transmission system

n-digit
n-digit
k-digit
k-digit
4
Introduction to Coding Theory

• 2 Basic assumptions
• Definitions
• Digit0 or 1(binary digit)
• Worda sequence of digits
• Example0110101
• Binary codea set of words
• Example1. 00,01,10,11 , 2. 0,01,001
• Block code a code having all its words of the
  same length
• Example 00,01,10,11, 2 is its length
• Codewords words belonging to a given code
• C Size of a code C(codewords in C)

5
Introduction to Coding Theory

• Assumptions about channel

2. Identifying the beginning of 1st word
3. The probability of any digit being affected
in transmission is the same as the other one.
6
Introduction to Coding Theory

• Binary symmetric channel

p
In many books, p denotes crossover
probability. Here crossover probability(error
prob.) is 1-p
7
Introduction to Coding Theory

• 3 Correcting and detecting error patterns

Any received word should be corrected to a
codeword that requires as few changes as possible.
Cannot detect any errors !!!
parity-check digit
8
Introduction to Coding Theory

• 4 Information rate
• Definition information rate of code C
• Examples

9
Introduction to Coding Theory

• 5 The effects of error correction and detection
• 1. No error detection and correction

Let C0,1110000000000, , 11111111111
Transmission rate107 digits/sec
Reliability p1-10-8
Then Pr(a word is transmitted incorrectly)
1-p11 ?11x10-8
11x10-8(wrong words/words)x107/11(words/sec)0.1
wrong words/sec
1 wrong word / 10 sec 6 wrong words / min 360
wrong words / hr 8640 wrong words / day
10
Introduction to Coding Theory

• 2. parity-check digit added(Code length becomes
  12 )
• Any single error can be detected !
• (3, 5, 7, ..errors can be detected too !)
• Pr(at least 2 errors in a word)1-p12-12 x
  p11(1-p)?66x10-16
• So 66x10-16 x 107/12 ? 5.5 x 10-9 wrong
  words/sec

one word error every 2000 days!
The cost we pay is to reduce a little information
rate retransmission(after error detection!)
11
Introduction to Coding Theory
3. 3-repetition code Any single error can be
corrected ! Code length becomes 33 and
information rate becomes 1/3

• Taskdesign codes with
• reasonable information rates
• low encoding and decoding costs
• some error-correcting capabilities

12
Introduction to Coding Theory

• 6 finding the most likely codeword transmitted
• Example

p reliability d digits incorrectly
transmitted n code length
Code length 5
13
Introduction to Coding Theory

• Theorem 1.6.3
• Suppose we have a BSC with ½ lt p lt 1. Let
  and be codewords and a word, each of length
  . Suppose that and disagree in
  positions and and disagree in positions.
  Then

14
Introduction to Coding Theory

• Example

15
Introduction to Coding Theory

• 7 Some basic algebra

16
Introduction to Coding Theory

• Kn is a vector space

words of length n
scalar
17
Introduction to Coding Theory

• 8 Weight and distance
• Hamming weight
• the number of times the digit 1 occurs in
• Example
• Hamming distance
• the number of positions in which and
  disagree
• Example

18
Introduction to Coding Theory

• Some facts

19
Introduction to Coding Theory

• 9 Maximum likelihood decoding

wvu

• CMLDComplete Maximum Likelihood Decoding
• If only one word v in C closer to w , decode
  it to v
• If several words closest to w, select arbitrarily
  one of them
• IMLDIncomplete Maximum Likelihood Decoding
• If only one word v in C closer to w, decode it
  to v
• If several words closest to w, ask for
  retransmission

Source string x
codeword
Error pattern
20
Introduction to Coding Theory
The most likely codeword sent is the one with the
error pattern of smallest weight
ExampleConstruct IMLD. M3 , C0000, 1010,
0111
21
Introduction to Coding Theory

• 10 Reliability of MLD
• The probability that if v is sent over a BSC of
  probability p then IMLD correctly concludes that
  v was sent

The higher the probability is, the more correctly
the word can be decoded!
22
Introduction to Coding Theory

• 11 Error-detecting codes

Example
Cant detect
Can detect
23
Introduction to Coding Theory

• the distance of the code C
• the smallest of d(v,w) in C
• Theorem 1.11.14
• A code C of distance d will at least detect all
  non-zero error patterns of weight less than or
  equal to d-1. Moreover, there is at least one
  error pattern of weight d which C will not
  detect.
• t error-detecting code
• It detects all error patterns of weight at most t
  and does not detect at least one error pattern of
  weight t1
• A code with distance d is a d-1 error-detecting
  code.

24
Introduction to Coding Theory

• 12 Error-correcting codes
• Theorem 1.12.9
• A code of distance d will correct all error
  patterns of weight less than or equal to
  . Moreover, there is at least one error
  pattern of weight 1 which C will
  not correct.
• t error-correcting code
• It corrects all error patterns of weight at most
  t and does not correct at least one error pattern
  of weight t1
• A code of distance d is a
  error-correcting code.

25
Introduction to Coding Theory
C corrects error patterns 000,100,010,001