Test of Independence through Mutual Information Wednesday, September 21, 2016 Jee Vang Ph.D. This lecture focuses on possible ways variables can relate to each other. The lecture is based on slides prepared by Dr. Lin and modified by Dr. Alemi. It is also based on the treatment of the topic by Alan Agresti in the book on Categorical Data Analysis.

Something Else Mutual Information There are many different methods for assessing independence and mutual information is often used as a way to do so.

Mutual Information The mutual information between two variables, X and Y, is defined as weighted sum of the log of probability of both events divided by the product of the probability of each event.

Mutual Information Note how mutual information reduces from statements about probability to the counts of times events occur.

Mutual Information # times both x & y occur In this formula, N with index x and y indicates the number of times both x and y occur together.

Mutual Information # times both x & y occur # times y occurs N with index x indicates number of times x occurs. N with index y indicates number of times y occurs. # times x occurs

[0, +Infinity] Threshold ? Range For mutual information, a value closer to zero implies independence, and values farther away from zero implies dependence. The range of values for mutual information is from zero to plus infinity. If we are to use mutual information to determine independence or dependence, typically, a threshold is specified a priori as a cutoff to make such a determination. For example, we can say a mutual information smaller than 0.001 is interpreted as independence, and a value larger than 0.001 is interpreted as dependence. This threshold value must be tinkered with to gauge what is appropriate, and differs from dataset to dataset. Threshold ?

Notation for Example Notation 1 Notation 2 X = true, Y = true X = t, Y = t X = true, Y = false X = t, Y = f X = false, Y = true X = f, Y = t X = false, Y = false X = f, Y = f Now, imagine two variables, X and Y, are both binary variables, taking on the values { true, false }. Let’s represent these values as { t, f } to be more concise. There are four unique combinations of the values shown in the table below. Note the equivalent notations.

Components of Mutual Information If we compute the mutual information between X and Y, we need to compute these four values.

Calculate the Count Using SQL TRUE select count(*) as total from data where n1=TRUE and n3=TRUE 3791 FALSE 1276 6833 8100 - select count(*) as total from data 20000 select count(*) as total from data where n1=TRUE 5067 select count(*) as total from data where n1=FALSE 14933 select count(*) as total from data where n3=TRUE 10624 select count(*) as total from data where n3=FALSE 9376 If we store the data into a table called “data”, we can issue the following SQL queries to get the count we need to compute the quantities above.

Mutual Information I(N1;N3) = 0.0282 + -0.0172 + -0.0221 + 0.0257 = 0.0145 Now we can calculate the mutual information between N1 and N3. Keep in mind that higher numbers indicate more dependence.

Conditional Mutual Information The conditional mutual information between two variables, X and Y given Z, and denoted as I(X,Y|Z), is defined based on conditional probabilities .

Conditional Mutual Information As before, the conditional mutual information can be calculated from counts as well. In this calculation Z can be a set of events or variables.

Conditional Mutual Information [0, +Infinity] Like mutual information, the value of conditional mutual information is in the range 0 to +infinity, where values closer to zero implies conditional independence and values farther from zero implies conditional dependence. Again, we also have to specify a cutoff value a priori to separate conditional independence from conditional dependence. Threshold ?

Division by Zero Division by zero can happen. This situation can happen when number of cooccurence of z with x or y is zero. In these situations the inner sum is set to zero and contribution of this level of z and x and y are ignored.

Mutual Information This lecture has shown how mutual and conditional mutual information can be carried out to get insights into the relationship among variables.