1 Module 3 Designs
2 Family Health Project: Exercise Review Discuss the Family Health Case and these questions. Consider how gender issues influence the formulation and implementation of the development of a family health project? Consider some likely questions that you might ask if you wanted to monitor and/or evaluate this family health project? What additional questions might you ask to make sure your questions are gender-sensitive? What other actions would you take?
3 Learning Objectives At the end of this session participants will understand: the importance of design in answering questions about impact design options gender issues as they relate to design
4 Type of Questions - Determine Strategy 1. For descriptive and normative question, strategy may be simple and straight forward. 2. For impact questions, how you will control for other factors so you can ascertain the effects of your program. 3. Impact questions require a different types of design. Evaluation Research Design
5 Designs for M&E Design = Strategy for answering M&E questions: Did the program cause the changes you observe? To answer this key causal question, you must be able to eliminate other explanations. This is done through design which controls for other factors so you can measure the effect of your program.
6 Design Validity Judge the quality of the design by its validity Two types of validity: 1. Internal validity Is what you observe due to the program and not other factors? 2. External validity Is what you observe true in the larger population?
7 Threats to Internal Validity History Due to a particular event that took place while data was being conducted. Maturation Skills increase because people get older. Testing Risk is that they “learned” how to do the test. Instrumentation Changes in data collection instrument, procedures or measures.
8 Threats to Internal Validity (Continued) Regression to the Mean The next set of scores are likely to change – to fall closer to the mean– regardless of treatment. Selection The group under study may be different in ways that effect the results. Attrition Different rates of dropping out of the program may affect results.
9 Threats to External Validity Biased or non-representative sample Sample is selected that is different from the larger population. Unique program Results are not likely to be the same elsewhere because of a unique program quality. Unique setting Results are not likely to be the same in a different location.
10 Types of Design Experimental Quasi-Experimental Non-Experimental
11 Types of Design Experimental Design Ideal Design for Measuring Impact Key elements: Random Assignment to groups - Program group - Control group Before and After Measurement Strongest design because all other explanations (threats to internal validity) have been ruled out May be a problem with external validity
12 Types of Design Experimental Design R O 1 X O 2 R O 1 O 2 R indicates Random assignment O is the Observation or measure X is the Program or the intervention
13 Experimental design is the strongest for answering causal questions But: it can be flawed for examining gender issues if the group that is randomly assigned does not include both men and women experimental studies of only men are not valid for conclusions about women Experimental Design Gender Issues in Answering Causality
14 Experimental Design is Rarely Used It is often not possible to use experimental design. ethics, practicality, costs Other evaluation designs Quasi-experimental Non-experimental Each successive design is weaker and more vulnerable to wrong interpretations of project impacts. Experimental Design
15 Quasi-Experimental Design O 1 X O 2 Program Group O 1 O 2 Control Group Groups No random assignment Matched pairs Non-equivalent comparison groups. Types of Design
16 Collect data from a large sample or the whole population and then statistically create comparison groups Are widowed women more likely to control resources than married women? Are men more or less likely to repay loans than women? Useful to measure program impact when it is not possible to randomly assign people to programs Eliminates threats to external validity Quasi-Experimental Design Using Statistical Controls
17 Before and After Design (lacks control group): O X O Static Group Comparison (lacks pre-measure): XO O One Shot (lacks control group and pre-measure): X O Types of Design Non-Experimental Design
18 Gender Issues in Design Men and women must be included: in the design selected in the program and comparison groups in the before and after groups Types of Design What are the examples?
19 Design in Monitoring Monitoring is the periodic assessment of the project to determine its progress towards intermediate or final goals. Validity of the monitoring indicators are design-sensitive. What are the examples?
20 Case Discussion Micro-Credit Studies Economic Study: Quasi-experimental design using statistical controls to create various comparison groups Cross-sectional survey No pre-test (baseline) data Social Study: Data collected from participants only after project was implemented One-shot design: no comparison group, no pre- measures.
21 Case Discussion Micro-Credit Case What are some flaws in each of these designs? What might be better designs?
22 Case Discussion Family Health Project Looking at the questions you developed for the Family Health Project, what designs would you select? What design did you anticipate for having them answer questions? Why would the designs be appropriate?