1. Test of Independence through Mutual Information
Wednesday, September 21, 2016
Jee Vang Ph.D.
This lecture focuses on possible ways variables can relate to each other. The lecture is based on slides prepared by Dr. Lin and modified by Dr. Alemi. It is also based on the treatment of the topic by Alan Agresti in the book on Categorical Data Analysis.

2. Something Else Mutual Information
There are many different methods for assessing independence and mutual information is often used as a way to do so.

3. Mutual Information
The mutual information between two variables, X and Y, is defined as weighted sum of the log of probability of both events divided by the product of the probability of each event.

4. Mutual Information
Note how mutual information reduces from statements about probability to the counts of times events occur.

5. Mutual Information # times both x & y occur
In this formula, N with index x and y indicates the number of times both x and y occur together.

6. Mutual Information # times both x & y occur # times y occurs
N with index x indicates number of times x occurs. N with index y indicates number of times y occurs.
# times x occurs

7. [0, +Infinity] Threshold ? Range
For mutual information, a value closer to zero implies independence, and values farther away from zero implies dependence. The range of values for mutual information is from zero to plus infinity. If we are to use mutual information to determine independence or dependence, typically, a threshold is specified a priori as a cutoff to make such a determination. For example, we can say a mutual information smaller than is interpreted as independence, and a value larger than is interpreted as dependence. This threshold value must be tinkered with to gauge what is appropriate, and differs from dataset to dataset.
Threshold ?

8. Notation for Example Notation 1 Notation 2 X = true, Y = true
X = t, Y = t
X = true, Y = false
X = t, Y = f
X = false, Y = true
X = f, Y = t
X = false, Y = false
X = f, Y = f
Now, imagine two variables, X and Y, are both binary variables, taking on the values { true, false }. Let’s represent these values as { t, f } to be more concise. There are four unique combinations of the values shown in the table below. Note the equivalent notations.

9. Components of Mutual Information
If we compute the mutual information between X and Y, we need to compute these four values.

10. Calculate the Count Using SQL
TRUE
select count(*) as total from data where n1=TRUE and n3=TRUE
3791
FALSE
1276
6833
8100 -
select count(*) as total from data
20000
select count(*) as total from data where n1=TRUE
5067
select count(*) as total from data where n1=FALSE
14933
select count(*) as total from data where n3=TRUE
10624
select count(*) as total from data where n3=FALSE
9376
If we store the data into a table called “data”, we can issue the following SQL queries to get the count we need to compute the quantities above.

11. Mutual Information
I(N1;N3) = =
Now we can calculate the mutual information between N1 and N3. Keep in mind that higher numbers indicate more dependence.

12. Conditional Mutual Information
The conditional mutual information between two variables, X and Y given Z, and denoted as I(X,Y|Z), is defined based on conditional probabilities .

13. Conditional Mutual Information
As before, the conditional mutual information can be calculated from counts as well. In this calculation Z can be a set of events or variables.

14. Conditional Mutual Information
[0, +Infinity]
Like mutual information, the value of conditional mutual information is in the range 0 to +infinity, where values closer to zero implies conditional independence and values farther from zero implies conditional dependence. Again, we also have to specify a cutoff value a priori to separate conditional independence from conditional dependence.
Threshold ?

15. Division by Zero
Division by zero can happen. This situation can happen when number of cooccurence of z with x or y is zero. In these situations the inner sum is set to zero and contribution of this level of z and x and y are ignored.

16. Mutual Information
This lecture has shown how mutual and conditional mutual information can be carried out to get insights into the relationship among variables.