Modeling Political Phenomena Using Control Variables and Gauging Validity

Face Validity Face validity means the measurement of a concept is consistent with an agreed definition. It does not mean however that this is the best measurement to capture the concept.

Construct Validity construct validity: The match between the land of theory and the land of observation How effective do our operationalized variables represent the mental image of a concept into the public manifestation of that world?

Reliability We may want, or need to, test for reliability, which is to ask if our variables consistently provide the same results. To be sure we can measure/test repeatedly or even use multiple measures for the same variable. For development we can also use GDP per capita besides energy consumption per capita.

Key question of Internal Validity When we test a hypothesis and either accept it or reject it, how do we know that we made the right decision? What about alternative explanations that we did not account for? What should we do to gain confidence?

Internal Validity Are there other causes for what I am observing? If so, a study will lack internal validity if it cannot rule out plausible alternative explanations.

Internal Validity of a Study What you measured and what you saw? Your program and your observations? Observation ProgramObservations What you do What you see Is the relationship causal between... Alternativecause Alternativecause Alternativecause Alternativecause In this study Program-outcome Relationship

The Purpose of Control Variables We use control variables to account for possible alternative explanations we can think of. For example, when I examined whether democracies are generally more peaceful than autocracies I included several control variables.

Explaining Pacifistic Democracy Peace (Y) = Democracy (X1) + State Power (X2) + Development (X3) + # of Bordering States (X4) In the model above, I have more confidence that Democracy is related to peace considering I control for the other variables that may skew my test.

We need to take care that our theory is not missing other factors that may undermine the validity of our theory and tests. Our inferences will be flawed if we are actually capturing other processes through our variables. This means that the validity of our measures would be undermined.

Several possible problems arise that are related to model misspecification and spurious relationships. Thus, we need to control for confounding factors and alternative explanations!!!

Model Misspecification and Spuriousness Antecedent variable: A variable that indirectly affecting the relationship between two other variables. For example, Ivy league education increases income. However, parental wealth and legacy admissions affect Ivy league education. Thus, income of graduates from Ivy League schools may not be random.

Here Ivy League Parents is an antecedent variable Ivy League Parents Ivy League Kids high income kids Hence, admission to Ivy schools clearly not random or pure merit-based, and thus the income earned by these people.

Model Misspecification and Spuriousness Intervening Variable: These may be spuriously related to another relationship. How can states fight each other if they are not contiguous with each other? Only the strongest, with large navies, bases, etc., could do so. Hence, geographic contiguity or distance is an intervening variable. States may or may not be more peaceful, but it is hard to avoid conflict when it is on your borders.

Model Misspecification and Spuriousness Alternative Variables: We also want to control for variables that would bias our results if omitted. In this case, the X variables in a model would produce biased estimates, undermining their validity and producing error that leads to inaccurate inferences.

Here is a spurious relationship from my research IGOs + conflicts + + Powerful states Powerful states both in more IGOs and conflicts, but these two variables not directly related but a function of state power.

Classic Spurious Case Ice Cream ConsumptionCrime Summer Temperatures ??? Hence we see that despite the fact that ice cream consumption is correlated with crime, the real cause is that summer temperatures increase both ice cream consumption and crime.

Assessing your knowledge If your scientific study has taken care to make sure that your variables are measured correctly, used the appropriate control variables, and used proper tests, then what is next?

Conclusion Validity What you did and what you saw? Your program and your observations? Observation ProgramObservations What you do What you see Is there a relationship between... In this study Program-outcome Relationship

Group Work Identify the level on which variables are measured. Identify problems of construct validity, internal validity, and biased samples

External Validity Now that you are confident of what you found in your study, how well does my study or sample relate to the general population? In other words, how strong is my study able to generalize to other cases?

Research Designs and Sampling In most studies what is examined are some cases, not an entire population. For example, in Presidential Election polls not every voter is asked how they will vote but still polls can be very accurate. How does that happen?

Population vs. Sample Research in the social sciences typically uses sampling methods. We draw a sample of subjects from a greater population. We then draw an inference from the sample about the greater population. In other words, we are generalizing about a population from a subset (the sample).

Validity and Bias In order to draw an accurate inference from a sample, the sample needs to be reflective of the population from which it is drawn. If a sample is not reflective of the population, then it is biased in some manner and the greater study will lack validity.

Types of Sampling Nonrandom: snowballing, various improper selection techniques or limited data. Measurement error is greater. Random: pure chance of lottery and should reflect population the larger the sample. Measurement error decreases. Quota or stratified: Selecting on groups to form sample so as to reflect greater population. Measurement error is lower. Census: Includes entire population. No measurement error.

Mathematical Principle The larger the sample size, the more it will reflect the population estimates/values. Thus, the larger the sample, the less chance of measurement error.

External Validity of a Study Theory CauseconstructEffectconstruct What you think Cause-effect Construct Observation ProgramObservations What You Do What You See program-outcome relationship Observation ProgramObservations What You Do What you see program-outcome relationship Observation ProgramObservations What You Do What You See program-outcome relationship Observation ProgramObservations What You Do What You See program-outcome relationship Can we generalize to other persons, places, times?

External Validity of a Study The last graphic is meant to convey the principle that external validity is gained by additional observations/tests in other studies. This is why pundits will compare several election polls to see how well they compare. If they do not, then somebody is doing something wrong or different.