III. Practical Considerations in preparing a CIE

Planning ahead Evaluation plan  evaluation process  activities  timing, budget Evaluation scheme  detailed plan  taylored to specific circumstances All evaluations CIE CIE Which questions do I need to answer for a CIE evaluation scheme???

1. Selecting interventions Justify funds Reform process Learning Strategic issues 7 factors related to the intervention(s) Amenable to CIE Types of data Data protection Data availability

Amenable to CIE Discrete, distinctive and homogenous interventions? Identifyable causal mechanism? Interference with other instruments? Clear target group? Sufficient size of treatment group? Meaningful control group? Anticipation of system-wide changes? Amenable to CIE

Data availability Types of data Data protection Treatment and control group records Result records Contextual data/control variables Data protection Microdata available? National data on individual careers to compare ESF participants with control group? Anonymized data Possibility to de-anonymize data to follow individual careers? Data availability

2. Evaluation scheme Description: objectives, strategy, intervention logic Resources Timing Identify target group Identify control group Data issues Reporting and dissemination

A. Description Logic framework Objectives Purpose Stakeholder Use of the results Specific questions Input and activities Output Results Impacts Questions Results – success Qualitative results Relevant subgroups Contextual factors When do impacts materialize? Identify results that require measurement Link them to data sources Promote understanding how the intervention operates

B: Human resources Knowledge and collaboration Staff in MA/IB CIE Methods Data collection and managmt Guiding external contractors External contractors Understanding of contexts Familiar with data sources Data handling and statistical methods Statistical data providers

B: Financial resources Budget for CIE Internal costs External costs Relation to other evaluations Costs depend on Novelty of intervention Data collection/handling

C: Timing Crucial dates and constraints When do I need my results? When do I get reasonable results? How does CIE fit with other evaluation work? When are data available and ready- to-be-used?

No employment due to training C: When to evaluate? Training Higher productivity Higher wage New job Job search No employment due to training time

D: Identify the „treated group“ Anticipating treatment – drop out of target group Do not participate in treatment Drop outs of treatment Participants WHY? Intention to treat Treat- ment Target group

D: Identify the treated group Data sources (persons/enterprises) Enumerated or sample ESF monitoring or survey Characteristics provided Individual entity identifiable Consent required by individual entity Possibility to follow up by a survey

E: Identify the control group Analytical: Obtain unbiased estimates – non randomized design Excluded from participation (unrelated to results) Matching through Location Time Eligibility Choice/awareness

E: Identify the control group Policy related: What is the alternative to the intervention? No treatment? Other treatment? Can I find appropriate definition of net effects? Practical considerations Data availability

Counterfactual design Counterfactual results F: Data requirements Collect, collate, documente micro-data for an analytical data set for… Counterfactual design Eligible target population Individual level identifier Contextual data Collect baseline data Random allocation Treatment group Control group Treatment and control group records Intervention to be evaluated Treatment as usual or no treatment Results Counterfactual results Result records for treated and control group

F: Constraints in data issues Sample size big enough? Other constraints Independence and objectivity in measuring outcomes? Sufficient information on control group to explain differences in results? Minimum detectable effects size at different sample size

F: Checklist for data issues Are sources consistent? How to access them Who will check potential sources? Can data be merged? Which sources Data issues Legal barriers? Which IT infrastructure is required? Can individuals be identified? How will data be stored and transferred safely?

G: Reporting Presentation All evaluations need to be made public!! Written evaluation report(s) Verbal presentation Technical report All evaluations need to be made public!!

IV. Windup Thinking beyond individual CIEs— There’s more to evaluation than finding the ‘credible counterfactual’ Infrastructure is critical Collaboration is essential