1. III. Practical Considerations in preparing a CIE

2. Planning ahead
Evaluation plan  evaluation process  activities  timing, budget Evaluation scheme  detailed plan  taylored to specific circumstances
All evaluations
CIE
CIE
Which questions do I need to answer for a CIE evaluation scheme???

3. 1. Selecting interventions
Justify funds
Reform process
Learning
Strategic issues
7 factors related to the intervention(s)
Amenable to CIE
Types of data
Data protection
Data availability

4. Amenable to CIE Discrete, distinctive and homogenous interventions?
Identifyable causal mechanism?
Interference with other instruments?
Clear target group?
Sufficient size of treatment group?
Meaningful control group?
Anticipation of system-wide changes?
Amenable to CIE

5. Data availability Types of data Data protection
Treatment and control group records
Result records
Contextual data/control variables
Data protection
Microdata available?
National data on individual careers to compare ESF participants with control group?
Anonymized data Possibility to de-anonymize data to follow individual careers?
Data availability

6. 2. Evaluation scheme
Description: objectives, strategy, intervention logic
Resources
Timing
Identify target group
Identify control group
Data issues
Reporting and dissemination

7. A. Description Logic framework
Objectives
Purpose
Stakeholder
Use of the results
Specific questions
Input and activities
Output
Results
Impacts
Questions
Results – success
Qualitative results
Relevant subgroups
Contextual factors
When do impacts materialize?
Identify results that require measurement
Link them to data sources
Promote understanding how the intervention operates

8. B: Human resources Knowledge and collaboration Staff in MA/IB
CIE Methods
Data collection and managmt
Guiding external contractors
External contractors
Understanding of contexts
Familiar with data sources
Data handling and statistical methods
Statistical data providers

9. B: Financial resources
Budget for CIE
Internal costs
External costs
Relation to other evaluations
Costs depend on
Novelty of intervention
Data collection/handling

10. C: Timing Crucial dates and constraints When do I need my results?
When do I get reasonable results?
How does CIE fit with other evaluation work?
When are data available and ready- to-be-used?

11. No employment due to training
C: When to evaluate?
Training
Higher productivity
Higher wage
New job
Job search
No employment due to training
time

12. D: Identify the „treated group“
Anticipating treatment – drop out of target group
Do not participate in treatment
Drop outs of treatment
Participants
WHY?
Intention to treat
Treat- ment
Target group

13. D: Identify the treated group
Data sources (persons/enterprises)
Enumerated or sample
ESF monitoring or survey
Characteristics provided
Individual entity identifiable
Consent required by individual entity
Possibility to follow up by a survey

14. E: Identify the control group
Analytical: Obtain unbiased estimates – non randomized design
Excluded from participation (unrelated to results)
Matching through
Location
Time
Eligibility
Choice/awareness

15. E: Identify the control group
Policy related:
What is the alternative to the intervention?
No treatment?
Other treatment?
Can I find appropriate definition of net effects?
Practical considerations
Data availability

16. Counterfactual design Counterfactual results
F: Data requirements
Collect, collate, documente micro-data for an analytical data set for…
Counterfactual design
Eligible target population
Individual level identifier
Contextual data
Collect baseline data
Random allocation
Treatment group
Control group
Treatment and control group records
Intervention
to be evaluated
Treatment as usual or no treatment
Results
Counterfactual results
Result records for treated and control group

17. F: Constraints in data issues
Sample size big enough?
Other constraints
Independence and objectivity in measuring outcomes?
Sufficient information on control group to explain differences in results?
Minimum detectable effects size at different sample size

18. F: Checklist for data issues
Are sources consistent?
How to access them
Who will check potential sources?
Can data be merged?
Which sources
Data issues
Legal barriers?
Which IT infrastructure is required?
Can individuals be identified?
How will data be stored and transferred safely?

19. G: Reporting Presentation All evaluations need to be made public!!
Written evaluation report(s)
Verbal presentation
Technical report
All evaluations need to be made public!!

20. IV. Windup Thinking beyond individual CIEs—
There’s more to evaluation than finding the ‘credible counterfactual’
Infrastructure is critical
Collaboration is essential