Causal Structure, Endogeneity, and the Missing Data Problem in Modeling the Impact of Information and Communication Technology Use on Society Tuesday, December 01, 2015 Hun Myoung Park University Information Technology Services Indiana University
HICSS-41, January 7-10, Outline ICT Use and Society Competing Perspectives Review of Traditional Approaches Nature of Problems Alternative Approaches Data and Illustrations Findings Implications
HICSS-41, ICT Use and Society Does ICT use influence society? Positive, negative, or negligible effect? Technological determinism Optimistic perspective Pessimistic perspective Skeptical perspective
HICSS-41, Optimistic Perspective ICT UseSociety Positive impact on society Transformation Theory Rheingold (1993); Grossman (1995); Morris (1999) “Getting the general public engaged”
HICSS-41, Pessimistic Perspective ICT UseSociety Negative impact on society Reinforcement theory David (1999, 2005); Norris (2001) Digital inequality (digital divide) “Engaging the engaged” rather than the disenfranchised
HICSS-41, Skeptical Perspective ICT UseSociety I CT use shaped by society R eflection of the real world N ormalization theory M argolis and Resnick (2000); Bimber (2001, 2003); Uslaner (2004) ” Politics as usual ”
HICSS-41, Conflicting Evidence, How? Conflicting empirical results depending on perspectives What is wrong? Failure to deal with the nature of problems properly How do we assess the impact of ICT use (treatment effect) more correctly?
HICSS-41, Review: T-test (ANOVA) Comparing means/proportions Scott (2006) Impact of ICT use: mean difference Simplicity and easy interpretation Two groups are assumed to have same characteristics except for the treatment
HICSS-41, Review: Linear Regression Least squares dummy variable model (LSDV) Jennings and Zeitner (2003); Uslaner (2004); Welch and Pandey (2007) Impact: dummy coefficient δ What if the dummy d are related to disturbance ε?
HICSS-41, Review: Binary Response Model Binary logit and probit model for binary dependent variables Bimber (2001, 2003) and Thomas and Streib (2003) Impact: a discrete change of d, difference in predicted probabilities Large N required
HICSS-41, Nature of Problems Measurement issues: categorical and binary DVs Limited DVs (self-selected) Ambiguous causal structure Endogeneity: d and ε are related The “missing data problem” in nonexperimental research
HICSS-41, Causal Structure ICT UseSociety ICT UseSociety Unidirectional versus bidirectional Interactive and jointly determined? Iterative and virtuous circle: Norris (2000)
HICSS-41, Endogeneity ICT use may not be exogenous Disturbance ε is related to the ICT use d violation of key OLS assumption Jointly determined in a system Instrumental variable (IV) approach?
HICSS-41, Missing Data Problem A subject is either ICT user (participant) or nonuser, not both. NOT necessarily means many missing values in data Users and nonusers may have different characteristics, which are not controlled in research (survey): self-selection bias
HICSS-41, Nonexperimental Design OBS pre Treatment OBS post_treatment OBS pre OBS post_control Treatment (?) OBS users OBS nonusers Randomized control group pre-post test design Non-randomized post test only design Is ICT use a real treatment?
HICSS-41, Propensity Score Matching 1 Rosenbaum and Rubin (1983, 1984) Binary Probit model to compute predicted probabilities Match users and nonusers who have similar likelihood (propensity score) Pair matching/subclassification; one-to- one pair matching w/o replacement Controlling many covariates using one dimensional propensity score
HICSS-41, Propensity Score Matching 2 Rosenbaum and Rubin (1984); Dehejia and Wahba (1999) Matching (paired) T-test
HICSS-41, Treatment Effect Model Subjects decide whether or not to receive treatment: selection bias Selection equation estimates predicted probabilities of ICT use Impact is the dummy coefficient adjusted by correlation of ICT use and the dependent variable When ρ=0, the impact is δ
HICSS-41, Recursive Bivariate Probit Model Maddala (1983), Greene (1998) Two equations with an endogenous IV variable, ICT use Correlation between disturbances If ρ≠0, both direct/indirect effects are considered in RBPM If ρ=0, binary response model (BRM) examines direct impact only
HICSS-41, Specification (RBPM)
HICSS-41, Secondary Data The PEW Internet and American Life Project 2004 Post-Election Internet Tracking Survey (Crosssectional) N=2,146 The American National Election Studies Longitudinal data of 1996, 1998, 2000, 2004 N=6,014
HICSS-41, Illustration 1: E-government Use IV (d): whether citizens look for information from government websites DV: whether citizens sent about voting (deliberative civic engagement) DV: Attendance at a rally during the election campaign (action- oriented)
HICSS-41, Illustration 1: E-government Use Average effect: 9.8% vs. 2.2% Discrete change: 15.3% vs. 3.3% Method Rally T-test 17.1% (1,243) 6.6% (1,320) PSM (Pair) 9.8% (509) 2.2% (558) BRM (Probit) 14.1% (1,030) 3.3% (1,090) RBPM 15.3% (931) 3.3% (974)
HICSS-41, Illustration 1: E-government Use
HICSS-41, Illustration 2: Internet Use IV (d): whether citizens have used the Internet for political information DV: discussing politics (deliberative civic engagement) DV: whether citizens gave money to a candidate (action-oriented engagement)
HICSS-41, Illustration 2: Internet Use Average effect: 10.1% vs. 4.4% Discrete change: 8.3% vs. 5.2% MethodDiscussGive Money T-test 21.0% (5,419) 6.3% (5,425) PSM (Pair) 10.1% (1,091) 4.4% (1,090) BRM (Probit) 9.9% (4,956) 5.4% (4,959) RBPM 8.3% (4,956) 5.2% (4,959)
HICSS-41, Illustration 2: Internet Use
HICSS-41, Finding 1: T-test vs. PSM Robust estimation of PSM at the expense of loss of N T-test overestimates the impact on deliberative civic engagement due to missing data problem No big difference in action- oriented engagement
HICSS-41, Finding 2: BRM vs. RBPM BRM overestimates the impact on deliberative civic engagement: endogeneity matters Both direct and indirect effects No big difference in action- oriented engagement; the impact of ICT use is direct
HICSS-41, Finding 3: Deliberative Engagement Both direct and indirect effects considered Overall impact depends on signs and magnitude of effects They may have opposite signs that cancel out each other BRM may report misleading results
HICSS-41, Implication and Conclusion Types of civic engagement to be differentiated; variety of civic engagement (Verba et al. 1995) Characteristics of dependent variables carefully examined Causal structure, endogeneity, missing data problem, and sample size considered Specific use of ICT applications differentiated as well
HICSS-41, Questions? Question or suggestion?