Basic Concepts of Information Theory Entropy for Two-dimensional Discrete Finite Probability Schemes. Conditional Entropy. Communication Network. Noise.

Slides:



Advertisements
Similar presentations
Random Variables ECE460 Spring, 2012.
Advertisements

Information Theory EE322 Al-Sanie.
Probability Theory Part 1: Basic Concepts. Sample Space - Events  Sample Point The outcome of a random experiment  Sample Space S The set of all possible.
Entropy Rates of a Stochastic Process
Operations Research: Applications and Algorithms
Chapter 6 Information Theory
Introduction to stochastic process
Background Knowledge Brief Review on Counting,Counting, Probability,Probability, Statistics,Statistics, I. TheoryI. Theory.
Fundamental limits in Information Theory Chapter 10 :
Information Theory Rong Jin. Outline  Information  Entropy  Mutual information  Noisy channel model.
Visual Recognition Tutorial1 Random variables, distributions, and probability density functions Discrete Random Variables Continuous Random Variables.
Review of Probability and Random Processes
Information Theory and Security
Lecture II-2: Probability Review
X= {x 0, x 1,….,x J-1 } Y= {y 0, y 1, ….,y K-1 } Channel Finite set of input (X= {x 0, x 1,….,x J-1 }), and output (Y= {y 0, y 1,….,y K-1 }) alphabet.
Chapter6 Jointly Distributed Random Variables
1 Statistical NLP: Lecture 5 Mathematical Foundations II: Information Theory.
§1 Entropy and mutual information
Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 2nd Mathematical Preparations (1): Probability and statistics Kazuyuki Tanaka Graduate.
INFORMATION THEORY BYK.SWARAJA ASSOCIATE PROFESSOR MREC.
: Appendix A: Mathematical Foundations 1 Montri Karnjanadecha ac.th/~montri Principles of.
§4 Continuous source and Gaussian channel
Chapter 8 Probability Section R Review. 2 Barnett/Ziegler/Byleen Finite Mathematics 12e Review for Chapter 8 Important Terms, Symbols, Concepts  8.1.
Discrete Mathematical Structures (Counting Principles)
Chapter 11: The Data Survey Supplemental Material Jussi Ahola Laboratory of Computer and Information Science.
Channel Capacity.
Discrete Math 6-1 Copyright © Genetic Computer School 2007 Lesson 6 Probability.
COMMUNICATION NETWORK. NOISE CHARACTERISTICS OF A CHANNEL 1.
Communication System A communication system can be represented as in Figure. A message W, drawn from the index set {1, 2,..., M}, results in the signal.
DIGITAL COMMUNICATIONS Linear Block Codes
CHAPTER 5 SIGNAL SPACE ANALYSIS
Basic Principles (continuation) 1. A Quantitative Measure of Information As we already have realized, when a statistical experiment has n eqiuprobable.
Basic Concepts of Information Theory Entropy for Two-dimensional Discrete Finite Probability Schemes. Conditional Entropy. Communication Network. Noise.
Review of Probability. Important Topics 1 Random Variables and Probability Distributions 2 Expected Values, Mean, and Variance 3 Two Random Variables.
1 Lecture 7 System Models Attributes of a man-made system. Concerns in the design of a distributed system Communication channels Entropy and mutual information.
The Channel and Mutual Information
Basic Concepts of Information Theory A measure of uncertainty. Entropy. 1.
F Y (y) = F (+ , y) = = P{Y  y} 3.2 Marginal distribution F X (x) = F (x, +  ) = = P{X  x} Marginal distribution function for bivariate Define –P57.
Copyright © Cengage Learning. All rights reserved. 5 Joint Probability Distributions and Random Samples.
Channel Coding Theorem (The most famous in IT) Channel Capacity; Problem: finding the maximum number of distinguishable signals for n uses of a communication.
Mutual Information, Joint Entropy & Conditional Entropy
1 Review of Probability and Random Processes. 2 Importance of Random Processes Random variables and processes talk about quantities and signals which.
Physics Fluctuomatics (Tohoku University) 1 Physical Fluctuomatics 2nd Probability and its fundamental properties Kazuyuki Tanaka Graduate School of Information.
Probability Distribution. Probability Distributions: Overview To understand probability distributions, it is important to understand variables and random.
UNIT I. Entropy and Uncertainty Entropy is the irreducible complexity below which a signal cannot be compressed. Entropy is the irreducible complexity.
(C) 2000, The University of Michigan 1 Language and Information Handout #2 September 21, 2000.
Chapter 4: Information Theory. Learning Objectives LO 4.1 – Understand discrete and continuous messages, message sources, amount of information and its.
PROBABILITY AND COMPUTING RANDOMIZED ALGORITHMS AND PROBABILISTIC ANALYSIS CHAPTER 1 IWAMA and ITO Lab. M1 Sakaidani Hikaru 1.
Basics of Multivariate Probability
Virtual University of Pakistan
Linear Algebra Review.
Introduction to Information theory
Graduate School of Information Sciences, Tohoku University
Appendix A: Probability Theory
Outline Introduction Signal, random variable, random process and spectra Analog modulation Analog to digital conversion Digital transmission through baseband.
Graduate School of Information Sciences, Tohoku University
Hiroki Sayama NECSI Summer School 2008 Week 3: Methods for the Study of Complex Systems Information Theory p I(p)
Information Theory Michael J. Watts
Subject Name: Information Theory Coding Subject Code: 10EC55
A Brief Introduction to Information Theory
Dependencies in Structures of Decision Tables
ASV Chapters 1 - Sample Spaces and Probabilities
Problem 1.
Chapter 2. Random Variables
Information Theoretical Analysis of Digital Watermarking
Discrete Random Variables and Probability Distributions
Chapter 3-2 Discrete Random Variables
Watermarking with Side Information
Continuous Random Variables: Basics
Presentation transcript:

Basic Concepts of Information Theory Entropy for Two-dimensional Discrete Finite Probability Schemes. Conditional Entropy. Communication Network. Noise Characteristics of a Communication Channel. 1

Entropy. Basic Properties Continuity: if the probabilities of the occurrence of events are slightly changed, the entropy is slightly changed accordingly. Symmetry: Extremal Property : when all the events are equally likely, the average uncertainty has the largest value: 2

Entropy. Basic Properties Additivity. Let is the entropy associated with a complete set of events E 1, E 2, …, E n. Let the event E n is divided into k disjoint subsets: Thus and where 3

Entropy. Basic Properties In general, is continuous in p i for all 4

ENTROPY FOR TWO-DIMENSIONAL DISCRETE FINITE PROBABILITY SCHEMES 5

Entropy for Two-dimensional Discrete Finite Probability Schemes The two-dimensional probability scheme provides the simplest mathematical model for a communication system with a transmitter and a receiver. Consider two finite discrete sample spaces Ω 1 (transmitter space) Ω 2 (receiver space) and their product space Ω. 6

Entropy for Two-dimensional Discrete Finite Probability Schemes In Ω 1 and Ω 2 we select complete sets of events Each event may occur in conjunction with any event. Thus for the product space Ω= Ω 1 Ω 2 we obtain the following complete set of events: 7

Entropy for Two-dimensional Discrete Finite Probability Schemes We may consider the following three complete sets of probability schemes Each one of them is, by assumption, a finite complete probability scheme like 8

Entropy for Two-dimensional Discrete Finite Probability Schemes The joint probability matrix for the random variables X and Y associated with spaces Ω 1 and Ω 2 : Respectively, 9

Entropy for Two-dimensional Discrete Finite Probability Schemes Complete Probability SchemeEntropy 10

Entropy for Two-dimensional Discrete Finite Probability Schemes If all marginal probabilities and are known then the marginal entropies can be expressed according to the entropy definition: 11

Conditional Entropies Let now an event F i may occur not independently, but in conjunction with 12

Conditional Entropies Consider the following complete probability scheme: Hence 13

Conditional Entropies Taking this conditional entropy for all admissible y j, we obtain a measure of average conditional entropy of the system: Respectively, 14

Conditional Entropies Since Then finally conditional entropies can be written as 15

Five Entropies Pertaining to Joint Distribution Thus we have considered: Two conditional entropies H(X|Y), H(Y|X) Two marginal entropies H(X), H(Y) The joint entropy H(X,Y) 16

COMMUNICATION NETWORK. NOISE CHARACTERISTICS OF A CHANNEL 17

Communication Network Consider a source of communication with a given alphabet. The source is linked to the receiver via a channel. The system may be described by a joint probability matrix: by giving the probability of the joint occurrence of two symbols, one at the input and another at the output. 18

Communication Network x i – a symbol, which was sent; y j - a symbol, which was received The joint probability matrix: 19

Communication Network: Probability Schemes There are following five probability schemes of interest in a product space of the random variables X and Y : [P{X,Y}] – joint probability matrix [P{X}] – marginal probability matrix of X [P{Y}] – marginal probability matrix of Y [P{X|Y}] – conditional probability matrix of X | Y [P{Y|X}] – conditional probability matrix of Y | X 20

Communication Network: Entropies There is the following interpretation of the five entropies corresponding to the mentioned five probability schemes: H(X,Y) – average information per pairs of transmitted and received characters (the entropy of the system as a whole); H(X) – average information per character of the source (the entropy of the source) H(Y) – average information per character at the destination (the entropy at the receiver) H(Y|X) – a specific character x k being transmitted and one of the permissible y j may be received (a measure of information about the receiver, where it is known what was transmitted) H(X|Y) – a specific character y j being received ; this may be a result of transmission of one of the x k with a given probability (a measure of information about the source, where it is known what was received) 21

Communication Network: Entropies’ Meaning H(X) and H(Y) give indications of the probabilistic nature of the transmitter and receiver, respectively. H(X,Y) gives the probabilistic nature of the communication channel as a whole H(Y|X) gives an indication of the noise (errors) in the channel H(X|Y) gives a measure of equivocation (how well one can recover the input content from the output) 22

Communication Network: Derivation of the Noise Characteristics In general, the joint probability matrix is not given for the communication system. It is customary to specify the noise characteristics of a channel and the source alphabet probabilities. From these data the joint and the output probability matrices can be derived. 23

Communication Network: Derivation of the Noise Characteristics Let us suppose that we have derived the joint probability matrix: 24

Communication Network: Derivation of the Noise Characteristics In other words : where: 25

Communication Network: Derivation of the Noise Characteristics If [P{X}] is not diagonal, but a row matrix ( n -dimensional vector) then where [P{Y}] is also a row matrix ( m -dimensional vector) designating the probabilities of the output alphabet. 26

Communication Network: Derivation of the Noise Characteristics Two discrete channels of our particular interest: Discrete noise-free channel (an ideal channel) Discrete channel with independent input- output (errors in the channel occur, thus noise is presented) 27