1. From the Conceptual Framework to the Empirical Model
Gustavo Angeles MEASURE Evaluation
University of North Carolina at Chapel Hill
Workshop on Impact Evaluation of Population, Health and Nutrition Programs
Accra, Ghana July 18-29, 2016

2. Importance of Conceptual Frameworks
A conceptual framework explicitly identifies the main relevant factors that influence the health result of interest
It also makes explicit the causal relationships between the factors, and, the pathways through which the program has an influence on the outcome
It provides the reference framework for the empirical model
It allows you to better identify endogenous explanatory factors
It also provides the reference framework for including or excluding variables in the empirical model

3. Simple Conceptual Framework
Individual / Household Characteristics
Use of FP Methods
Community Characteristics / Health Service supply

4. Simple Conceptual Framework
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation

5. Simple Conceptual Framework
What do we observe?
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation

6. Simple Conceptual Framework
What do we observe?
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation

7. Simple Conceptual Framework
What do we observe?
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation

8. Simple Conceptual Framework Building an empirical model
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Use =f(Age , Edu , SES , Rur , Prog ,

9. Simple Conceptual Framework Building an empirical model
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij=f(Ageij, Eduij, SESij, Rurj, Progj,

10. Simple Conceptual Framework Building an empirical model
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj)

11. Simple Conceptual Framework Building an empirical model
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj)
Unobserved
Observed

12. Simple Conceptual Framework Building an empirical model
Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj) ?
What is the functional form of this function?

13. Simple Conceptual Framework Building an empirical model
Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj) ?
What is the functional form of this function?
We don’t know.
But, we need a functional form to estimate the relationship between “Use” and its determinants, in particular, the Program.
So, what can we do?

14. Simple Conceptual Framework
Building an empirical model – An useful result
Look at the statistics toolkit. An interesting result:
If you have Y= f( X1, X2) with f(.) unknown, you can approximate it by:

15. Simple Conceptual Framework
Building an empirical model – An useful result
Look at the statistics toolkit. An interesting result:
If you have Y= f( X1, X2) with f(.) unknown, you can approximate it by:
Y≈ α0 + α1X1 + α2X2 + α3X1X2 + α4X12 + α5X22 + α6X12 X22
+ α7X13 + α8X23 + α9X13 X23
+ α10X14 + α11X24 + α12X14 X24
+ … (polynomial of order n continues)

16. Simple Conceptual Framework
Building an empirical model – An useful result
Look at the statistics toolkit. An interesting result:
If you have Y= f( X1, X2) with f(.) unknown, you can approximate it by:
Y≈ α0 + α1X1 + α2X2 + α3X1X2 + α4X12 + α5X22 + α6X12 X22
+ α7X13 + α8X23 + α9X13 X23
+ α10X14 + α11X24 + α12X14 X24
+ … (polynomial of order n continues)
This is the Taylor Series approximation (Taylor polynomial; 1715)
In practice, we use the linear part of the approximation:
Y= α0 + α1X1 + α2X and sometimes you add interactions and squares.

17. Simple Conceptual Framework
Going back to our model
So, we have:
Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj)
Which can be approximated by a linear relationship:
Useij = α0+ α1Ageij + α2Eduij+α3SESij+α4Rurj +α5Progj
+ (α6Riskij+ α7 Genij + α8HFj + α9Sanj)

18. Simple Conceptual Framework Examining the structure of our model
Useij = α0+ α1Ageij + α2Eduij+α3SESij+α4Rurj +α5Progj
+ (α6Riskij+ α7 Genij + α8HFj + α9Sanj)
It has a basic structure:
Useij = Observed factors + Unobserved factors
Usually, all unobserved factors are summarized in the term εij
Useij = α0 + α1Ageij + α2Eduij + α3SESij + α4Rurj + α5Progj + εij

19. Simple Conceptual Framework Examining the structure of our model
Useij = α0+ α1Ageij + α2Eduij+α3SESij+α4Rurj +α5Progj
+ (α6Riskij+ α7 Genij + α8HFj + α9Sanj)
It has a basic structure:
Useij = Observed factors + Unobserved factors
Usually, all unobserved factors are summarized in the term εij
Useij = α0 + α1Ageij + α2Eduij + α3SESij + α4Rurj + α5Progj + εij
But, it is important to know what those unobserved factors are and how they relate to the other determinants.

20. Simple Conceptual Framework Examining the structure of our model
Notice that there are unobserved factors at the individual and community level. In particular,
Unobserved = α6Riskij+ α7 Genij + α8HFj + α9Sanj
Individual-level
Unobservables
Community-level
Unobservables

21. Now, what can we say about the relationship between the explanatory factors and Use?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj
What if? ↑Edu

22. Now, what can we say about the relationship between the explanatory factors and Use?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj
What if? ↑Edu → ∆ Use

23. Now, what can we say about the relationship between the explanatory factors and Use?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj
What if? ↑Edu → ∆ Use
↑HF

24. Now, what can we say about the relationship between the explanatory factors and Use?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj
What if? ↑Edu → ∆ Use
↑HF → ∆ Use

25. Now, what can we say about the relationship between the explanatory factors and Use?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj
What if? ↑Edu → ∆ Use
↑HF → ∆ Use
↑Prog

26. Now, what can we say about the relationship between the explanatory factors and Use?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj
What if? ↑Edu → ∆ Use
↑HF → ∆ Use
↑Prog → ∆ Use

27. Now, what can we say about the relationship between the explanatory factors and Use?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Characteristics / Health Service supply
- Rurality
- Health facilities (HF)
- Program
- Sanitation
Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj
What if? ↑Edu → ∆ Use
↑HF → ∆ Use
↑Prog → ∆ Use
Then, we have direct and unique relationships between each determinant
and “Use”. Then, we can estimate the effect of Prog on Use.

28. Now, a simple modification: Conceptual Framework 2
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program

29. A simple modification: Conceptual Framework 2
What do we observe?
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Charact.
- Rurality
- Health facilities(HF)
- Sanitation
Program

30. A simple modification: Conceptual Framework 2
What is the associated empirical model?
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj

31. A simple modification: Conceptual Framework 2
What is the associated empirical model?
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj

32. A simple modification: Conceptual Framework 2
What is the associated empirical model?
Observed?
YES NO
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Observed?
YES NO
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
Now the empirical model has two equations.
Notice that equation (1) is exactly as in the previous slides for the first conceptual framework.

33. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu

34. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu → ∆ Use

35. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu → ∆ Use
↑Risk

36. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu → ∆ Use
↑Risk → ∆ Use

37. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu → ∆ Use
↑Risk → ∆ Use
↑HF

38. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model?
Individual / Household Characteristics
- Age
- Education
- Household wealth/SES
- Risk aversion
- Biological endowments
Use of FP Methods
Community Charact.
- Rurality
- Health facilities (HF)
- Sanitation
Program
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu → ∆ Use
↑Risk → ∆ Use
↑HF → ∆ Prog → ∆ Use
∆ Use
We don’t have a unique relationship between “Health Facilities (HF)” and “Use”!!
∆ Use

39. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model?
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu → ∆ Use
↑Risk → ∆ Use
↑HF → ∆ Prog → ∆ Use
∆ Use
A change in HF generates two effects:
1. Change “Prog”, which changes “Use”.
2. Change “Use”, directly.
Notice that we only observe the net effect on Use (∆ Use).
A simple analysis will relate the observed change on “Use” to the observed change in “Prog”. Clearly, it estimates incorrectly the impact of Prog on Use.
∆ Use

40. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model?
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu → ∆ Use
↑Risk → ∆ Use
↑HF → ∆ Prog → ∆ Use
∆ Use
A change in HF generates two effects:
1. Change “Prog”, which changes “Use”.
2. Change “Use”, directly.
Notice that we only observe the net effect on Use (∆ Use).
A simple analysis will relate the observed change on “Use” to the observed change in “Prog”. Clearly, it estimates incorrectly the impact of Prog on Use.
This is the problem of endogeneity. Cannot estimate α5 correctly.
Program Impact: α5
∆ Use

41. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model? Endogenetity
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu → ∆ Use
↑Risk → ∆ Use
↑HF → ∆ Prog → ∆ Use
∆ Use
This problem is caused by unobserved factors, such as “HF”, that influence both the Outcome (“Use”) and the Program (“Prog”).
Special methods are needed to disentangle the effects and separate out the effect of Prog on Use.
Methods:
Instrumental Variables
Longitudinal data models
Some other “clever” methods.
Program Impact: α5
∆ Use

42. A simple modification: Conceptual Framework 2
What can we say about the relationships in the model? Endogenetity
Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj
Progj = β0+ β1Rurj + β2HFj + β3Sanj
What if? ↑Edu → ∆ Use
↑Risk → ∆ Use
↑HF → ∆ Prog → ∆ Use
∆ Use
Question: Which kind of factors (or determinants) can be considered as suspects of being endogenous?
Program Impact: α5
∆ Use

43. Thanks.

44. This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government.