Measuring causal impact 2.1

What is impact? The impact of a program is the difference in outcomes caused by the program It is the difference between what happened and what would have happened without the program But we never observe both conditions – What happened with the program – What would have happened without the program

Counterfactual The counterfactual is what would have happened if the program or policy had not been implemented We never directly observe the counterfactual – We can never see the same person with and without the program at the same time We have to mimic the counterfactual

Time Primary Outcome Impact Counterfactual Intervention Measuring Impact

Mimicking the counterfactual Because there is no exact replica of a single person we look for a group of people that on average are the same as participants would have been without the program Do participants prior to the program make a good counterfactual? Do those who choose not to participate make a good counterfactual?

Selection Programs are started in specific places and at specific times for a reason, they are selected People choose or select to participate or not participate This selection process means that participants and non participants are on average different not just because of the program Nonparticipants are not always a good counterfactual Because people select when to take up a program, participants prior to the program may not be a good counterfactual

Selection bias If we compare outcomes for those with and without the program the difference will have two parts: i) That caused by the program ii) That caused by underlying differences between participants and nonparticipants Our estimate of impact will not, on average, be equal to the true impact of the program unless ii) is zero and there is no selection bias

Example: Testing the impact of training Time Wages Skills program X X Ashenfelter Dip