1. Communicating Efficiently
Coding Theory

2. The Problem

3. The Problem
Sending data from one place to another (through a channel) may yield errors.
How do you detect such errors?
How do you correct the errors?

4. Mind Test
“It tkaes a geart dael of cuoarge to satnd up to yuor enimees, but a geart dael mroe to satnd up to yuor freidns.”
-Dumbledore

5. Your Brain is a Code-cracking Machine.
For emaxlpe, it deson’t mttaer in waht oredr the ltteers in a wrod aepapr, the olny iprmoatnt tihng is taht the frist and lsat ltteer are in the rghit pcale. The rset can be a toatl mses and you can sitll raed it wouthit pobelrm.

6. Your Brain is a Code-cracking Machine.
S1M1L4RLY, Y0UR M1ND 15 R34D1NG 7H15 4U70M471C4LLY W17H0U7 3V3N 7H1NK1NG 4B0U7 17.

7. Sending and Receiving
SENDER
CHANNEL
RECEIVER

8. The Basic Problem of Coding Theory
Messages are transmitted over a communication channel which is subject to noise. But noise can distort messages resulting in errors.

9. The Basic Problem of Coding Theory
GOALS of Coding Theory:
Error Detection
Error Correction
HI, BOB!
HI, COB!
HOW DO WE ACHIEVE THESE GOALS EFFICIENTLY?

10. Example
Messages:
YES → NO → 11
YES
??? 00 10
COMMUNICATION FAILURE

11. What is coding theory?

12. What is coding theory?
CODING THEORY deals with the design of error-correcting codes for the reliable transmission of information across noisy channels.

13. Imperfect Transmission of Data
Data is usually transferred in the form of a series of 0s and 1s. However, transmission of data is not perfect.

14. Some Causes
An electric surge, cross-contamination from another data stream, or human error can easily change some of the 0s to 1s and vice versa.

15. Noise in Communication
Information media, such as communication systems and storage devices of data, are not absolutely reliable in practice because of noise or other forms of introduced interference.

16. Applications

17. Applications of Coding Theory
Transmission of pictures from distant space
Quality of sound in CDs
Establishment of computer networks
Communication through telephone lines
Messaging through wireless communication

18. Applications of Coding Theory
When photographs are transmitted to Earth from deep space, error-control codes are used to guard against the noise caused by atmospheric interruptions.

19. Applications of Coding Theory
Compact discs (CDs) use error- control codes so that a CD player can read data from a CD even if it has been corrupted causing imperfections on the CD.

20. Tasks of Coding Theory

21. Tasks of Coding Theory
The goal of coding theory is to improve the reliability of digital communication by devising methods that enable the receiver to decide whether there have been errors during the transmission (error detection), and if there are, to possibly recover the original message (error correction).

22. Terms to Understand

23. Source Coding
Source coding involves changing the message source, such as a data terminal or the human voice, to a suitable code for transmission through the channel.
The source encoder transforms the source output into a sequence of symbols which we call a “message”.
An example of source coding is the ASCII code.

24. Terminologies Codeword
a string of 0's and 1's representing an actual message
Stop → 000
Go → 101
Wait → 110

25. Terminologies Stop → 000 Go → 101 Wait → 110
000, 101, 110 are codewords
100, 111, 001, 010, 011 are NOT codewords
The LENGTH OF A CODEWORD is the number of binary digits it has.

26. Terminologies Code - the collection or set of all codewords EXAMPLE:
Stop → 000
Go → 101
Wait → 110
The code is C = {000, 101, 110}.

27. Terminologies “message“ becomes a “codeword“ STOP 000 Encode STOP
SOURCE
ENCODER
000

28. Terminologies Decode “received word“ is reverted back to a “message“
STOP
000
SOURCE
DECODER
STOP

29. 7-bit ASCII CODE A B E S I T O W U R
The word is “SUT“ using the ASCII table shown above. But if you are aware that this word does not make sense then perhaps there is an error.
Encode the message “SAT“ using the 7-bit ASCII code table
shown above.
Decode

30. 7-bit ASCII CODE Decode 1000010 1000101 1010011 1010100 A 1000 001 B E S I T O W U R
The word is “SUT“ using the ASCII table shown above. But if you are aware that this word does not make sense then perhaps there is an error.
Decode

31. Simple Codes

32. Parity Check Codes
The simplest form of error detection is parity, where a single bit is appended to a bit string.
A bit string has odd parity if the number of 1s in the string is odd.
A bit string has even parity if the number of 1s in the string is even.
Even parity is more common, but both are used.

33. Parity Check Digit
Include a parity check digit to make an even parity code.
Messages: YES → 000? NO → 111?
YES
???
0000
0100

34. Parity Check Digit
Include a parity check digit to make an even parity code.
Messages: YES → NO → 1111
YES
???
0000
0100

35. EVEN Parity Codewords
Append ONE BIT to each word to form a binary word of EVEN PARITY. ? ? ?

36. EVEN Parity Codewords
Append ONE BIT to each word to form a binary word of EVEB PARITY.

37. Odd Parity Codewords
Append ONE BIT to each word to form a binary word of ODD PARITY. ? ? ?

38. Odd Parity Codewords
Append ONE BIT to each word to form a binary word of ODD PARITY.

39. Even Parity Code (An Example)
Three Messages
Stop →
Go →
Wait →
This is an even parity code.
C = { , , }

40. Even Parity Code (Detecting an Error)
Three Messages
Stop →
Go →
Wait →
Suppose the binary word received is
r =
Will this make any sense?
What do you think is the correct message sent?

41. Odd Parity Code (An Example)
Write all codewords in an odd parity code of length 4.
Identify all binary words of length 3.
How many are they?
Add a parity check digit to each binary word to create all codewords of length 4.

42. Odd Parity Code of Length 41

43. Repetition Codes
The simplest possible error-correcting code is the repetition code.
For instance, if we wanted to send the message 1010, we could repeat each letter a certain number of times and send, say r = 3 times and we obtain,

44. Example Encode the bit string 011001 by repeating each bit twice.
Solution:
2. Encode the bit string 011 by repeating each
bit thrice.

45. Majority decoding
Consider a code where each bit is repeated 3 times. Suppose we want to transmit the following bit string: If no error is made during transmission, the receiver gets

46. Majority decoding Now assume that the receiver gets
Has there been any error in this transmitted word?
How do you correct it?

47. Majority decoding The received word 000 111 110 000 111 000 010.
will be decoded as

48. Repetition Code (An Example)
Write all codewords in a repetition code of length 9 with r = 3.
Identify all binary words of length 3.
How many are they?
Repeat every bit in each binary word 3 times to form the corresponding codeword.

49. Repetition Code of Length 9 (r = 3)1

50. Error Detection and Error Correction

51. Hamming Distance
The Hamming distance between two words is the number of differences between corresponding bits.
If x and y are two bits (binary strings) then their hamming distance is denoted by d(x,y).
What is d(0010, 1100)?
What is d(101011, )?

52. Nearest Neighbor Decoding
Suppose that when a codeword x from a code C is sent, the bit string y is received.
If the transmission was error-free, then y would be the same as x. But if there are transmission errors, then y would not be the same as x. How can we correct errors? That is, how can we recover x?

53. Nearest Neighbor Decoding
One approach would be to compute the Hamming distance between y and each of the codewords in C.
Then, to decode y, we take the codeword of minimum Hamming distance from y, if such a codeword is unique.

54. Nearest Neighbor Decoding
If the distance between the closest codewords in C is large enough and if sufficiently few errors were made in transmission, this codeword should be x. This type of decoding is called nearest neighbor decoding.

55. Example
Given the code C = { , , }, decode the received word
y =
Decode this received word using the nearest-neighbour decoding process.

56. Example
Given the code C = { , , }, decode the received word
y =
We find the hamming distance between the received word y and each codeword x in C.

57. Example
Given the code C = { , , }, decode the received word
y =
d(y, ) = ?
d(y, ) = ?
d(y, ) = ?

58. Example By using the nearest neighbor decoding principle we obtain
d = 6 6 2
Example
By using the nearest neighbor decoding principle we obtain
D(y) =
since this codeword is the nearest word to the received word y.

59. Minimum Distance of a Code
The minimum distance of a code is the smallest distance between any two distinct codewords in the code.

60. Minimum Distance of a Code
Example:
Given C = {111, 110, 011},
we have
d(111, 110) = 1, d(111, 011) = 1, d(110, 011) = 2.
Therefore,
the minimum distance of C is
d(C) = 1.

61. Detection and Correction Capability of a Code
Let C be a set of codewords and let
2e + 1 be its minimum Hamming distance. Then
It is possible to detect up to 2e errors.
It is possible to correct up to e errors.

62. Error Detection and Error Correction
Suppose a code has minimum distance equal to
2e + 1 = 15.
How many errors can this code detect?
How many errors can this code correct?

63. Error Detection and Error Correction
Suppose a code has minimum distance equal to
2e + 1 = 20.
How many errors can this code detect?
How many errors can this code correct?

64. Detection and Correction Capability of a Code
Minimum Distance
Error Detection/ Correction 1
No detection possible 2
1-error detecting 3
2-error detecting/
1-error correcting 4
3-error detecting/ 5
4-error detecting/
2-error correcting