From the Conceptual Framework to the Empirical Model Gustavo Angeles MEASURE Evaluation University of North Carolina at Chapel Hill Workshop on Impact Evaluation of Population, Health and Nutrition Programs Accra, Ghana July 18-29, 2016

Importance of Conceptual Frameworks A conceptual framework explicitly identifies the main relevant factors that influence the health result of interest It also makes explicit the causal relationships between the factors, and, the pathways through which the program has an influence on the outcome It provides the reference framework for the empirical model It allows you to better identify endogenous explanatory factors It also provides the reference framework for including or excluding variables in the empirical model

Simple Conceptual Framework Individual / Household Characteristics Use of FP Methods Community Characteristics / Health Service supply

Simple Conceptual Framework Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation

Simple Conceptual Framework What do we observe? Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation

Simple Conceptual Framework What do we observe? Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation

Simple Conceptual Framework What do we observe? Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation

Simple Conceptual Framework Building an empirical model Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Use =f(Age , Edu , SES , Rur , Prog ,

Simple Conceptual Framework Building an empirical model Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij=f(Ageij, Eduij, SESij, Rurj, Progj,

Simple Conceptual Framework Building an empirical model Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj)

Simple Conceptual Framework Building an empirical model Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj) Unobserved Observed

Simple Conceptual Framework Building an empirical model Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj) ? What is the functional form of this function?

Simple Conceptual Framework Building an empirical model Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj) ? What is the functional form of this function? We don’t know. But, we need a functional form to estimate the relationship between “Use” and its determinants, in particular, the Program. So, what can we do?

Simple Conceptual Framework Building an empirical model – An useful result Look at the statistics toolkit. An interesting result: If you have Y= f( X1, X2) with f(.) unknown, you can approximate it by:

Simple Conceptual Framework Building an empirical model – An useful result Look at the statistics toolkit. An interesting result: If you have Y= f( X1, X2) with f(.) unknown, you can approximate it by: Y≈ α0 + α1X1 + α2X2 + α3X1X2 + α4X12 + α5X22 + α6X12 X22 + α7X13 + α8X23 + α9X13 X23 + α10X14 + α11X24 + α12X14 X24 + … (polynomial of order n continues)

Simple Conceptual Framework Building an empirical model – An useful result Look at the statistics toolkit. An interesting result: If you have Y= f( X1, X2) with f(.) unknown, you can approximate it by: Y≈ α0 + α1X1 + α2X2 + α3X1X2 + α4X12 + α5X22 + α6X12 X22 + α7X13 + α8X23 + α9X13 X23 + α10X14 + α11X24 + α12X14 X24 + … (polynomial of order n continues) This is the Taylor Series approximation (Taylor polynomial; 1715) In practice, we use the linear part of the approximation: Y= α0 + α1X1 + α2X2 and sometimes you add interactions and squares.

Simple Conceptual Framework Going back to our model So, we have: Useij=f(Ageij, Eduij, SESij, Rurj, Progj, Riskij, Genij,HFj,Sanj) Which can be approximated by a linear relationship: Useij = α0+ α1Ageij + α2Eduij+α3SESij+α4Rurj +α5Progj + (α6Riskij+ α7 Genij + α8HFj + α9Sanj)

Simple Conceptual Framework Examining the structure of our model Useij = α0+ α1Ageij + α2Eduij+α3SESij+α4Rurj +α5Progj + (α6Riskij+ α7 Genij + α8HFj + α9Sanj) It has a basic structure: Useij = Observed factors + Unobserved factors Usually, all unobserved factors are summarized in the term εij Useij = α0 + α1Ageij + α2Eduij + α3SESij + α4Rurj + α5Progj + εij

Simple Conceptual Framework Examining the structure of our model Useij = α0+ α1Ageij + α2Eduij+α3SESij+α4Rurj +α5Progj + (α6Riskij+ α7 Genij + α8HFj + α9Sanj) It has a basic structure: Useij = Observed factors + Unobserved factors Usually, all unobserved factors are summarized in the term εij Useij = α0 + α1Ageij + α2Eduij + α3SESij + α4Rurj + α5Progj + εij But, it is important to know what those unobserved factors are and how they relate to the other determinants.

Simple Conceptual Framework Examining the structure of our model Notice that there are unobserved factors at the individual and community level. In particular, Unobserved = α6Riskij+ α7 Genij + α8HFj + α9Sanj Individual-level Unobservables Community-level Unobservables

Now, what can we say about the relationship between the explanatory factors and Use? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj What if? ↑Edu

Now, what can we say about the relationship between the explanatory factors and Use? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj What if? ↑Edu → ∆ Use

Now, what can we say about the relationship between the explanatory factors and Use? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj What if? ↑Edu → ∆ Use ↑HF

Now, what can we say about the relationship between the explanatory factors and Use? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj What if? ↑Edu → ∆ Use ↑HF → ∆ Use

Now, what can we say about the relationship between the explanatory factors and Use? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj What if? ↑Edu → ∆ Use ↑HF → ∆ Use ↑Prog

Now, what can we say about the relationship between the explanatory factors and Use? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj What if? ↑Edu → ∆ Use ↑HF → ∆ Use ↑Prog → ∆ Use

Now, what can we say about the relationship between the explanatory factors and Use? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Characteristics / Health Service supply - Rurality - Health facilities (HF) - Program - Sanitation Useij = α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj +α6Riskij+α7Genij+α8HFj+α9Sanj What if? ↑Edu → ∆ Use ↑HF → ∆ Use ↑Prog → ∆ Use Then, we have direct and unique relationships between each determinant and “Use”. Then, we can estimate the effect of Prog on Use.

Now, a simple modification: Conceptual Framework 2 Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Charact. - Rurality - Health facilities (HF) - Sanitation Program

A simple modification: Conceptual Framework 2 What do we observe? Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Charact. - Rurality - Health facilities(HF) - Sanitation Program

A simple modification: Conceptual Framework 2 What is the associated empirical model? Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Charact. - Rurality - Health facilities (HF) - Sanitation Program Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj

A simple modification: Conceptual Framework 2 What is the associated empirical model? Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Charact. - Rurality - Health facilities (HF) - Sanitation Program Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj

A simple modification: Conceptual Framework 2 What is the associated empirical model? Observed? YES NO Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Observed? YES NO Community Charact. - Rurality - Health facilities (HF) - Sanitation Program Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj Now the empirical model has two equations. Notice that equation (1) is exactly as in the previous slides for the first conceptual framework.

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Charact. - Rurality - Health facilities (HF) - Sanitation Program Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Charact. - Rurality - Health facilities (HF) - Sanitation Program Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu → ∆ Use

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Charact. - Rurality - Health facilities (HF) - Sanitation Program Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu → ∆ Use ↑Risk

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Charact. - Rurality - Health facilities (HF) - Sanitation Program Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu → ∆ Use ↑Risk → ∆ Use

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Charact. - Rurality - Health facilities (HF) - Sanitation Program Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu → ∆ Use ↑Risk → ∆ Use ↑HF

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Individual / Household Characteristics - Age - Education - Household wealth/SES - Risk aversion - Biological endowments Use of FP Methods Community Charact. - Rurality - Health facilities (HF) - Sanitation Program Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu → ∆ Use ↑Risk → ∆ Use ↑HF → ∆ Prog → ∆ Use ∆ Use We don’t have a unique relationship between “Health Facilities (HF)” and “Use”!! ∆ Use

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu → ∆ Use ↑Risk → ∆ Use ↑HF → ∆ Prog → ∆ Use ∆ Use A change in HF generates two effects: 1. Change “Prog”, which changes “Use”. 2. Change “Use”, directly. Notice that we only observe the net effect on Use (∆ Use). A simple analysis will relate the observed change on “Use” to the observed change in “Prog”. Clearly, it estimates incorrectly the impact of Prog on Use. ∆ Use

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu → ∆ Use ↑Risk → ∆ Use ↑HF → ∆ Prog → ∆ Use ∆ Use A change in HF generates two effects: 1. Change “Prog”, which changes “Use”. 2. Change “Use”, directly. Notice that we only observe the net effect on Use (∆ Use). A simple analysis will relate the observed change on “Use” to the observed change in “Prog”. Clearly, it estimates incorrectly the impact of Prog on Use. This is the problem of endogeneity. Cannot estimate α5 correctly. Program Impact: α5 ∆ Use

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Endogenetity Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu → ∆ Use ↑Risk → ∆ Use ↑HF → ∆ Prog → ∆ Use ∆ Use This problem is caused by unobserved factors, such as “HF”, that influence both the Outcome (“Use”) and the Program (“Prog”). Special methods are needed to disentangle the effects and separate out the effect of Prog on Use. Methods: Instrumental Variables Longitudinal data models Some other “clever” methods. Program Impact: α5 ∆ Use

A simple modification: Conceptual Framework 2 What can we say about the relationships in the model? Endogenetity Useij=α0+α1Ageij+α2Eduij+α3SESij+α4Rurj+α5Progj+α6Riskij+α7Genij+α8HFj+α9Sanj Progj = β0+ β1Rurj + β2HFj + β3Sanj What if? ↑Edu → ∆ Use ↑Risk → ∆ Use ↑HF → ∆ Prog → ∆ Use ∆ Use Question: Which kind of factors (or determinants) can be considered as suspects of being endogenous? Program Impact: α5 ∆ Use

Thanks.

This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L-14-00004. MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government. www.measureevaluation.org