1. Causal Structure, Endogeneity, and the Missing Data Problem in Modeling the Impact of Information and Communication Technology Use on Society Tuesday, December 01, 2015 Hun Myoung Park University Information Technology Services Indiana University kucc625@indiana.edu

2. HICSS-41, January 7-10, 20082 Outline  ICT Use and Society  Competing Perspectives  Review of Traditional Approaches  Nature of Problems  Alternative Approaches  Data and Illustrations  Findings  Implications

3. HICSS-41, 20083 ICT Use and Society  Does ICT use influence society?  Positive, negative, or negligible effect?  Technological determinism Optimistic perspective Pessimistic perspective  Skeptical perspective

4. HICSS-41, 20084 Optimistic Perspective ICT UseSociety  Positive impact on society  Transformation Theory  Rheingold (1993); Grossman (1995); Morris (1999)  “Getting the general public engaged”

5. HICSS-41, 20085 Pessimistic Perspective ICT UseSociety  Negative impact on society  Reinforcement theory  David (1999, 2005); Norris (2001)  Digital inequality (digital divide)  “Engaging the engaged” rather than the disenfranchised

6. HICSS-41, 20086 Skeptical Perspective ICT UseSociety  I CT use shaped by society  R eflection of the real world  N ormalization theory  M argolis and Resnick (2000); Bimber (2001, 2003); Uslaner (2004)  ” Politics as usual ”

7. HICSS-41, 20087 Conflicting Evidence, How?  Conflicting empirical results depending on perspectives  What is wrong?  Failure to deal with the nature of problems properly  How do we assess the impact of ICT use (treatment effect) more correctly?

8. HICSS-41, 20088 Review: T-test (ANOVA)  Comparing means/proportions  Scott (2006)  Impact of ICT use: mean difference  Simplicity and easy interpretation  Two groups are assumed to have same characteristics except for the treatment

9. HICSS-41, 20089 Review: Linear Regression  Least squares dummy variable model (LSDV)  Jennings and Zeitner (2003); Uslaner (2004); Welch and Pandey (2007)  Impact: dummy coefficient δ  What if the dummy d are related to disturbance ε?

10. HICSS-41, 200810 Review: Binary Response Model  Binary logit and probit model for binary dependent variables  Bimber (2001, 2003) and Thomas and Streib (2003)  Impact: a discrete change of d, difference in predicted probabilities  Large N required

11. HICSS-41, 200811 Nature of Problems  Measurement issues: categorical and binary DVs  Limited DVs (self-selected)  Ambiguous causal structure  Endogeneity: d and ε are related  The “missing data problem” in nonexperimental research

12. HICSS-41, 200812 Causal Structure ICT UseSociety ICT UseSociety  Unidirectional versus bidirectional  Interactive and jointly determined?  Iterative and virtuous circle: Norris (2000)

13. HICSS-41, 200813 Endogeneity  ICT use may not be exogenous  Disturbance ε is related to the ICT use d  violation of key OLS assumption  Jointly determined in a system  Instrumental variable (IV) approach?

14. HICSS-41, 200814 Missing Data Problem  A subject is either ICT user (participant) or nonuser, not both.  NOT necessarily means many missing values in data  Users and nonusers may have different characteristics, which are not controlled in research (survey): self-selection bias

15. HICSS-41, 200815 Nonexperimental Design OBS pre  Treatment  OBS post_treatment OBS pre OBS post_control Treatment (?) OBS users OBS nonusers  Randomized control group pre-post test design  Non-randomized post test only design  Is ICT use a real treatment?

16. HICSS-41, 200816 Propensity Score Matching 1  Rosenbaum and Rubin (1983, 1984)  Binary Probit model to compute predicted probabilities  Match users and nonusers who have similar likelihood (propensity score)  Pair matching/subclassification; one-to- one pair matching w/o replacement  Controlling many covariates using one dimensional propensity score

17. HICSS-41, 200817 Propensity Score Matching 2  Rosenbaum and Rubin (1984); Dehejia and Wahba (1999)  Matching  (paired) T-test

18. HICSS-41, 200818 Treatment Effect Model  Subjects decide whether or not to receive treatment: selection bias  Selection equation estimates predicted probabilities of ICT use  Impact is the dummy coefficient adjusted by correlation of ICT use and the dependent variable  When ρ=0, the impact is δ

19. HICSS-41, 200819 Recursive Bivariate Probit Model  Maddala (1983), Greene (1998)  Two equations with an endogenous IV variable, ICT use  Correlation between disturbances  If ρ≠0, both direct/indirect effects are considered in RBPM  If ρ=0, binary response model (BRM) examines direct impact only

20. HICSS-41, 200820 Specification (RBPM)

21. HICSS-41, 200821 Secondary Data  The PEW Internet and American Life Project 2004 Post-Election Internet Tracking Survey (Crosssectional) N=2,146  The American National Election Studies Longitudinal data of 1996, 1998, 2000, 2004 N=6,014

22. HICSS-41, 200822 Illustration 1: E-government Use  IV (d): whether citizens look for information from government websites  DV: whether citizens sent email about voting (deliberative civic engagement)  DV: Attendance at a rally during the election campaign (action- oriented)

23. HICSS-41, 200823 Illustration 1: E-government Use  Average effect: 9.8% vs. 2.2%  Discrete change: 15.3% vs. 3.3% MethodEmailRally T-test 17.1% (1,243) 6.6% (1,320) PSM (Pair) 9.8% (509) 2.2% (558) BRM (Probit) 14.1% (1,030) 3.3% (1,090) RBPM 15.3% (931) 3.3% (974)

24. HICSS-41, 200824 Illustration 1: E-government Use

25. HICSS-41, 200825 Illustration 2: Internet Use  IV (d): whether citizens have used the Internet for political information  DV: discussing politics (deliberative civic engagement)  DV: whether citizens gave money to a candidate (action-oriented engagement)

26. HICSS-41, 200826 Illustration 2: Internet Use  Average effect: 10.1% vs. 4.4%  Discrete change: 8.3% vs. 5.2% MethodDiscussGive Money T-test 21.0% (5,419) 6.3% (5,425) PSM (Pair) 10.1% (1,091) 4.4% (1,090) BRM (Probit) 9.9% (4,956) 5.4% (4,959) RBPM 8.3% (4,956) 5.2% (4,959)

27. HICSS-41, 200827 Illustration 2: Internet Use

28. HICSS-41, 200828 Finding 1: T-test vs. PSM  Robust estimation of PSM at the expense of loss of N  T-test overestimates the impact on deliberative civic engagement due to missing data problem  No big difference in action- oriented engagement

29. HICSS-41, 200829 Finding 2: BRM vs. RBPM  BRM overestimates the impact on deliberative civic engagement: endogeneity matters  Both direct and indirect effects  No big difference in action- oriented engagement; the impact of ICT use is direct

30. HICSS-41, 200830 Finding 3: Deliberative Engagement  Both direct and indirect effects considered  Overall impact depends on signs and magnitude of effects  They may have opposite signs that cancel out each other  BRM may report misleading results

31. HICSS-41, 200831 Implication and Conclusion  Types of civic engagement to be differentiated; variety of civic engagement (Verba et al. 1995)  Characteristics of dependent variables carefully examined  Causal structure, endogeneity, missing data problem, and sample size considered  Specific use of ICT applications differentiated as well

32. HICSS-41, 200832 Questions?  Question or suggestion?