Type I & Type II Errors, Power of a Statistical Test, & Effect Size four of the most confusing topics in introductory statistics packaged in a way that will hopefully make them clear Nicole Radziwill, James Madison / Feel free to use & share with citation!

Type I & Type II Errors When you perform a statistical test, you’re only taking ONE SAMPLE from a population – and there are tons of different samples you could potentially be collecting. You have to think about your sample in the context of ALL the potential samples you could have collected… fortunately this is made easy thanks to the sampling distributions of proportions and means. Type I Error, also called , is the likelihood that you incorrectly reject the null hypothesis Type II Error, also called , is the likelihood that you fail to detect the effect that your alternative hypothesis is trying to uncover

The Pregnancy Test Example Reality What the Pregnancy Test Said H 0 True (Really Are Pregnant) H 0 False (Really NOT pregnant) POS (Reject H 0 ) NEG (Fail to Reject H 0 ) H 0 : You are not pregnant. H A : You ARE pregnant. There are four things that can happen when you take the pregnancy test: The test can be ACCURATE, and say that: 1) you’re pregnant when you actually are, OR 2) you’re not pregnant when you actually AREN’T Or it can be INACCURATE, and say that: 3) you’re pregnant when you’re NOT (a FALSE ALARM), or 4) you’re not pregnant when you actually ARE…. (a FAILURE TO DETECT the effect) H 0 True (Really NOT pregnant) H 0 False (Really ARE pregnant)

The Pregnancy Test Example Reality What the Pregnancy Test Said H 0 True (Really Are Pregnant) H 0 False (Really NOT pregnant) POS (Reject H 0 ) NEG (Fail to Reject H 0 ) H 0 : You are not pregnant. H A : You ARE pregnant. There are four things that can happen when you take the pregnancy test: The test can be ACCURATE, and say that: 1) you’re pregnant when you actually are, OR 2) you’re not pregnant when you actually AREN’T Or it can be INACCURATE, and say that: 3) you’re pregnant when you’re NOT (a FALSE ALARM), or 4) you’re not pregnant when you actually ARE…. (a FAILURE TO DETECT the effect)  = probability of this test raising a FALSE ALARM  = probability of this test FAILING TO DETECT your pregnancy! H 0 True (Really NOT pregnant) H 0 False (Really ARE pregnant)

The Pregnancy Test Example Reality What the Pregnancy Test Said H 0 True (Really Are Pregnant) H 0 False (Really NOT pregnant) POS (Reject H 0 ) NEG (Fail to Reject H 0 ) H 0 : You are not pregnant. H A : You ARE pregnant. If Type II Error is the probability that a statistical test performed on this particular sample FAILS TO DETECT THE EFFECT (pregnancy), then what is the probability that a test using this sample will SUCCESSFULLY DETECT THE EFFECT? P(successfully detecting the effect) = 1 – P(NOT detecting the effect) = 1 -  This (1 -  ) is called The POWER OF THE TEST!  = probability of this test raising a FALSE ALARM  = probability of this test FAILING TO DETECT your pregnancy! H 0 True (Really NOT pregnant) H 0 False (Really ARE pregnant)

Consequences of Pregnancy Test Errors Type I Error – The test raised a false alarm and got you either very worried or excited. You may have already run to the store to buy baby supplies, incurring unnecessary costs. You may have spent days or weeks panicking until you realized that the test was faulty. Type II Error - The test failed to detect your pregnancy and you didn’t stop drinking or smoking, therefore potentially harming a life. You didn’t seek prenatal medical attention. In a case like this, the test designers need to think about how to minimize BOTH Type I and Type II Errors. There are psychological and cost ramifications if either kind happens.

Getting Picked Up By the Cops Reality What the Cop Decided H 0 True (Really NOT Guilty; it was a SKUNK) H 0 False (Really ARE Guilty) I’m Taking You to Jail (Reject H 0 ) I’m Letting You Go (Fail to Reject H 0 ) The scenario: A cop pulls you over and says he smells pot in your car. He is trying to figure out whether or not to take you to jail. The cop needs to do a little hypothesis test in his head to figure out whether to bring you in. H 0 : You are not guilty. H A : You ARE guilty. You have been smoking pot in your car. There are four things that can happen here: The cop can be ACCURATE, and say that: 1) you’re guilty when you actually are, OR 2) you’re not guilty and you haven’t been smoking pot Or he can be INACCURATE, and say that: 3) you’re guilty when you’re NOT (a FALSE ARREST), or 4) you’re not guilty when you actually ARE…. (a FAILURE TO DETECT the effect and a GET OUT OF JAIL FREE card)

Getting Picked Up By the Cops Reality What the Cop Decided H 0 True (Really NOT Guilty; it was a SKUNK) H 0 False (Really ARE Guilty) I’m Taking You to Jail (Reject H 0 ) I’m Letting You Go (Fail to Reject H 0 ) The scenario: A cop pulls you over and says he smells pot in your car. He is trying to figure out whether or not to take you to jail. The cop needs to do a little hypothesis test in his head to figure out whether to bring you in. H 0 : You are not guilty. H A : You ARE guilty. You have been smoking pot in your car. There are four things that can happen here: The cop can be ACCURATE, and say that: 1) you’re guilty when you actually are, OR 2) you’re not guilty and you haven’t been smoking pot Or he can be INACCURATE, and say that: 3) you’re guilty when you’re NOT (a FALSE ARREST), or 4) you’re not guilty when you actually ARE…. (a FAILURE TO DETECT the effect and a GET OUT OF JAIL FREE card)  = probability of this test leading to FALSE ARREST  = probability of this cop FAILING TO DETECT your pot smoking!

Consequences of Cop Errors Type I Error – The test raised a false alarm and got you sent to jail even though you didn’t deserve it. It cost you time and maybe even cost you money – bailing yourself out, or fighting court fees!! Type II Error - The cop let you go when he had reason to send you to jail! Probably good for you, bad for the cop (who may be trying to make his arrest quota for the month). Possibly bad for society, but other factors would have to be considered. Probably best to focus on keeping the Type I Error as low as possible in these cases – it’s more problematic to have a lot of false alarms than to let a few pot smokers off free here and there.

Getting Picked Up By the Cops #2 Reality What the Cop Decided H 0 True (Really NOT Guilty; you just look sketchy) H 0 False (Really ARE Guilty) I’m Taking You to Jail (Reject H 0 ) I’m Letting You Go (Fail to Reject H 0 ) The scenario: A cop pulls you over and suspects that you have just committed a murder. He is trying to figure out whether or not to take you to jail. The cop needs to do a little hypothesis test in his head to figure out whether to bring you in. H 0 : You are not guilty. H A : You ARE guilty. You are hiding evidence and a body in your trunk. There are four things that can happen here: The cop can be ACCURATE, and say that: 1) you’re guilty when you actually are, OR 2) you’re not guilty - you haven’t just killed someone Or he can be INACCURATE, and say that: 3) you’re guilty when you’re NOT (a FALSE ARREST), or 4) you’re not guilty when you actually ARE…. (a FAILURE TO DETECT the murder and eluding the law)

Getting Picked Up By the Cops #2 Reality What the Cop Decided H 0 True (Really NOT Guilty; you just look sketchy) H 0 False (Really ARE Guilty) I’m Taking You to Jail (Reject H 0 ) I’m Letting You Go (Fail to Reject H 0 )  = probability of this test leading to FALSE ARREST  = probability of this cop FAILING TO DETECT the murder! The scenario: A cop pulls you over and suspects that you have just committed a murder. He is trying to figure out whether or not to take you to jail. The cop needs to do a little hypothesis test in his head to figure out whether to bring you in. H 0 : You are not guilty. H A : You ARE guilty. You are hiding evidence and a body in your trunk. There are four things that can happen here: The cop can be ACCURATE, and say that: 1) you’re guilty when you actually are, OR 2) you’re not guilty – you haven’t just killed someone Or he can be INACCURATE, and say that: 3) you’re guilty when you’re NOT (a FALSE ARREST), or 4) you’re not guilty when you actually ARE…. (a FAILURE TO DETECT the murder and eluding the law)

Consequences of Cop Errors (#2) Type I Error – The test raised a false alarm and got you sent to jail and into a HUGE legal mess even though you didn’t deserve it. It costs you time and will probably cost you tons of money – bailing yourself out, court fees, lawyers… a trial!! Type II Error - The cop let you go when you had a body in the trunk! Probably good for you, bad for the cop, and VERY BAD for society. I’d want to keep the Type II Error as low as possible in this case, and risk some false alarms to AVOID letting killers go. Would you?? (Notice that keeping the Type II Error low ALSO keeps the power of the test pretty high.)

Power of the Test Power = 1 -  It’s the probability of successfully detecting an effect The power of the test INCREASES as the effect size increases… So girls, if you are just ONE day late, the effect size is small – it will be difficult for that pregnancy test to give you an accurate result. You’ll have a high Type II Error at this time. If you are a week late, the effect size is bigger – the Type II Error will be lower, because the pregnancy is easier to detect. If you are a month late, the effect size is HUGE and the Type II Error will be even smaller.

Interrelationships P 0 is the proportion that you’re assuming is true – often a standard or “known” value P* is the TRUE population proportion The bigger the difference between P 0 and P*, the bigger the EFFECT SIZE (the bottom curve shifts to the right as effect size gets bigger) If the proportion you measured from your sample is BIGGER than the real proportion P*, you’ll incorrectly reject the null and incur a Type I Error. If the proportion you measured from your sample is LESS than the real proportion P*, but STILL GREATER than the values in the top curve that are way out to the left all on their own, You’ll fail to reject the null and incur a Type II Error.