Causal Structure, Endogeneity, and the Missing Data Problem in Modeling the Impact of Information and Communication Technology Use on Society Tuesday, December 01, 2015 Hun Myoung Park University Information Technology Services Indiana University

HICSS-41, January 7-10, Outline  ICT Use and Society  Competing Perspectives  Review of Traditional Approaches  Nature of Problems  Alternative Approaches  Data and Illustrations  Findings  Implications

HICSS-41, ICT Use and Society  Does ICT use influence society?  Positive, negative, or negligible effect?  Technological determinism Optimistic perspective Pessimistic perspective  Skeptical perspective

HICSS-41, Optimistic Perspective ICT UseSociety  Positive impact on society  Transformation Theory  Rheingold (1993); Grossman (1995); Morris (1999)  “Getting the general public engaged”

HICSS-41, Pessimistic Perspective ICT UseSociety  Negative impact on society  Reinforcement theory  David (1999, 2005); Norris (2001)  Digital inequality (digital divide)  “Engaging the engaged” rather than the disenfranchised

HICSS-41, Skeptical Perspective ICT UseSociety  I CT use shaped by society  R eflection of the real world  N ormalization theory  M argolis and Resnick (2000); Bimber (2001, 2003); Uslaner (2004)  ” Politics as usual ”

HICSS-41, Conflicting Evidence, How?  Conflicting empirical results depending on perspectives  What is wrong?  Failure to deal with the nature of problems properly  How do we assess the impact of ICT use (treatment effect) more correctly?

HICSS-41, Review: T-test (ANOVA)  Comparing means/proportions  Scott (2006)  Impact of ICT use: mean difference  Simplicity and easy interpretation  Two groups are assumed to have same characteristics except for the treatment

HICSS-41, Review: Linear Regression  Least squares dummy variable model (LSDV)  Jennings and Zeitner (2003); Uslaner (2004); Welch and Pandey (2007)  Impact: dummy coefficient δ  What if the dummy d are related to disturbance ε?

HICSS-41, Review: Binary Response Model  Binary logit and probit model for binary dependent variables  Bimber (2001, 2003) and Thomas and Streib (2003)  Impact: a discrete change of d, difference in predicted probabilities  Large N required

HICSS-41, Nature of Problems  Measurement issues: categorical and binary DVs  Limited DVs (self-selected)  Ambiguous causal structure  Endogeneity: d and ε are related  The “missing data problem” in nonexperimental research

HICSS-41, Causal Structure ICT UseSociety ICT UseSociety  Unidirectional versus bidirectional  Interactive and jointly determined?  Iterative and virtuous circle: Norris (2000)

HICSS-41, Endogeneity  ICT use may not be exogenous  Disturbance ε is related to the ICT use d  violation of key OLS assumption  Jointly determined in a system  Instrumental variable (IV) approach?

HICSS-41, Missing Data Problem  A subject is either ICT user (participant) or nonuser, not both.  NOT necessarily means many missing values in data  Users and nonusers may have different characteristics, which are not controlled in research (survey): self-selection bias

HICSS-41, Nonexperimental Design OBS pre  Treatment  OBS post_treatment OBS pre OBS post_control Treatment (?) OBS users OBS nonusers  Randomized control group pre-post test design  Non-randomized post test only design  Is ICT use a real treatment?

HICSS-41, Propensity Score Matching 1  Rosenbaum and Rubin (1983, 1984)  Binary Probit model to compute predicted probabilities  Match users and nonusers who have similar likelihood (propensity score)  Pair matching/subclassification; one-to- one pair matching w/o replacement  Controlling many covariates using one dimensional propensity score

HICSS-41, Propensity Score Matching 2  Rosenbaum and Rubin (1984); Dehejia and Wahba (1999)  Matching  (paired) T-test

HICSS-41, Treatment Effect Model  Subjects decide whether or not to receive treatment: selection bias  Selection equation estimates predicted probabilities of ICT use  Impact is the dummy coefficient adjusted by correlation of ICT use and the dependent variable  When ρ=0, the impact is δ

HICSS-41, Recursive Bivariate Probit Model  Maddala (1983), Greene (1998)  Two equations with an endogenous IV variable, ICT use  Correlation between disturbances  If ρ≠0, both direct/indirect effects are considered in RBPM  If ρ=0, binary response model (BRM) examines direct impact only

HICSS-41, Specification (RBPM)

HICSS-41, Secondary Data  The PEW Internet and American Life Project 2004 Post-Election Internet Tracking Survey (Crosssectional) N=2,146  The American National Election Studies Longitudinal data of 1996, 1998, 2000, 2004 N=6,014

HICSS-41, Illustration 1: E-government Use  IV (d): whether citizens look for information from government websites  DV: whether citizens sent about voting (deliberative civic engagement)  DV: Attendance at a rally during the election campaign (action- oriented)

HICSS-41, Illustration 1: E-government Use  Average effect: 9.8% vs. 2.2%  Discrete change: 15.3% vs. 3.3% Method Rally T-test 17.1% (1,243) 6.6% (1,320) PSM (Pair) 9.8% (509) 2.2% (558) BRM (Probit) 14.1% (1,030) 3.3% (1,090) RBPM 15.3% (931) 3.3% (974)

HICSS-41, Illustration 1: E-government Use

HICSS-41, Illustration 2: Internet Use  IV (d): whether citizens have used the Internet for political information  DV: discussing politics (deliberative civic engagement)  DV: whether citizens gave money to a candidate (action-oriented engagement)

HICSS-41, Illustration 2: Internet Use  Average effect: 10.1% vs. 4.4%  Discrete change: 8.3% vs. 5.2% MethodDiscussGive Money T-test 21.0% (5,419) 6.3% (5,425) PSM (Pair) 10.1% (1,091) 4.4% (1,090) BRM (Probit) 9.9% (4,956) 5.4% (4,959) RBPM 8.3% (4,956) 5.2% (4,959)

HICSS-41, Illustration 2: Internet Use

HICSS-41, Finding 1: T-test vs. PSM  Robust estimation of PSM at the expense of loss of N  T-test overestimates the impact on deliberative civic engagement due to missing data problem  No big difference in action- oriented engagement

HICSS-41, Finding 2: BRM vs. RBPM  BRM overestimates the impact on deliberative civic engagement: endogeneity matters  Both direct and indirect effects  No big difference in action- oriented engagement; the impact of ICT use is direct

HICSS-41, Finding 3: Deliberative Engagement  Both direct and indirect effects considered  Overall impact depends on signs and magnitude of effects  They may have opposite signs that cancel out each other  BRM may report misleading results

HICSS-41, Implication and Conclusion  Types of civic engagement to be differentiated; variety of civic engagement (Verba et al. 1995)  Characteristics of dependent variables carefully examined  Causal structure, endogeneity, missing data problem, and sample size considered  Specific use of ICT applications differentiated as well

HICSS-41, Questions?  Question or suggestion?