The Data Goldrush Andy WedelBodo Winter University of ArizonaUC Merced

The data revolution Griffiths, T. L. (2015). Manifesto for a new (computational) cognitive revolution. Cognition, 135,

ml?_r=1& “Data is merely the raw material of knowledge”

The “data science” rebranding

Jeff Leek The “data science” rebranding

Structure of this course Day 1 (1) general statistical issues (2) cross-cultural correlational studies Day 2 Andy’s day! (phonetics & phonology) Day 3 Bodo’s day! (semantics & gesture) Day 4 data-driven language contact research

The course website bodowinter.com/goldrush

My one slide on inferential stats Sample Population inference assume null hypothesis reject null hypothesis p<0.05

Type I error = erroneously rejecting the null hypothesis demo My one slide on inferential stats cont’d

My one slide on regression RT ~ Noise

My one slide on regression

Correlations when-it-does/ ~

Examples of linguistic correlations Object-VerbVerb-Object Postpositions47242 Prepositions14456 Dryer, M.S. (2013). Relationship between the order of object and verb and the order of adposition and noun. WALS,

Examples of linguistic correlations ~

~ sound change Eckert, P. (1989). The whole woman: Sex and gender differences in variation. Language Variation and Change, 1, Labov, W. (1990). The interaction of sex and social class in the course of linguistic change. Language Variation and Change, 2,

Examples of linguistic correlations ~ Chen, M. K. (2012). The effect of language on economic behavior: Evidence from savings rates, health behaviors, and retirement assets. American Economic Review, 103, future tense marking

Examples of linguistic correlations ~ Everett, C. (2013). Evidence for direct geographic influences on linguistic sounds: the case of ejectives. PloS one, 8(6), e kʼ tʼ qʼ

~ Everett, C., Blasi, D. E., & Roberts, S. G. (2015). Climate, vocal folds, and tonal languages: Connecting the physiological and geographic dots. Proceedings of the National Academy of Sciences, 112, Examples of linguistic correlations

Two types of correlations … between linguistic features … between linguistic & non-linguistic features Ladd, D. R., Roberts, S. G., & Dediu, D. (2015). Correlational studies in typological and historical linguistics. Annual Review of Linguistics, 1,

Examples of correlations Ladd, D. R., Roberts, S. G., & Dediu, D. (2015). Correlational studies in typological and historical linguistics. Annual Review of Linguistics, 1,

Three statistical problems Multiple comparisons Correlation is not causation Lack of independence

Three statistical problems Multiple comparisons Correlation is not causation Lack of independence

Three statistical problems Correlation is not causation Roberts, S., & Winters, J. (2013). Linguistic diversity and traffic accidents: Lessons from statistical studies of cultural traits. PLoS one, 8(8), e70902.

Three statistical problems Correlation is not causation Christian Bentz Bentz, C., & Winter, B. (2013). Languages with more second language learners tend to lose nominal case. Language Dynamics & Change, 3:1, 1-27.

Three statistical problems Correlation is not causation L2 learningmorphology L2 speakersmorphology L2 speakersmorphology confound?

Three statistical problems Correlation is not causation L2 speakersmorphology 4. shared history 5. chance

when-it-does/ Three statistical problems Correlation is not causation

Three statistical problems Correlation is not causation The data must be strong. The data must be consistent. The data must be coherent. The data must be specific. The causal effect must be plausible. Steps to support causality when-it-does/

Three statistical problems Correlation is not causation Data Ladd, D. R., Roberts, S. G., & Dediu, D. (2015). Correlational studies in typological and historical linguistics. Annual Review of Linguistics, 1, correlations increase the probability of compatible models, they decrease the probability of incompatible ones Hypothesis

Three statistical problems Correlation is not causation Experiments do not provide fool-proof access to causality

Three statistical problems Correlation is not causation

Three statistical problems Multiple comparisons Correlation is not causation Lack of independence

Three statistical problems Multiple comparisons Correlation is not causation Lack of independence

Three statistical problems Multiple comparisons Bennett, C. M., Baird, A. A., Miller, M. B., & Wolford, G. L. (2011). Neural correlates of interspecies perspective taking in the post-mortem atlantic salmon: an argument for proper multiple comparisons correction. Journal of Serendipitous and Unexpected Results, 1, 1-5.

Three statistical problems Multiple comparisons Example of doing 100 tests: If α = 0.05 is taken as the significance level For 100 tests, the expected number of incorrect rejections of the null hypothesis is 5 The probability of at least one statistical result being significant is:

Three statistical problems Multiple comparisons What to do? (1) Correcting for multiple comparisons (e.g., Bonferroni correction) (2) Avoid multiplicity at the design stage (Bender & Lange, 2001) Bender, R., & Lange, S. (2001). Adjusting for multiple testing—when and how?. Journal of clinical epidemiology, 54(4),

Three statistical problems Multiple comparisons Correlation is not causation Lack of independence

Three statistical problems Multiple comparisons Correlation is not causation Lack of independence

Three statistical problems Lack of independence

Three statistical problems Lack of independence

Three statistical problems Lack of independence

Three statistical problems Lack of independence

Three statistical problems Lack of independence Sample 1 Sample 2 A simple Type I error simulation:

Three statistical problems Lack of independence (Source code on class github repo) unique_items <- rnorm(nitems) unique_subs <- rnorm(nsub) resp <- unique_items[items] + unique_subs[subjects] + rnorm(nsub*nitems) 1,000 simulations

Three statistical problems Lack of independence

Three statistical problems Lack of independence

Three statistical problems Lack of independence

Two sources of non-independence Lack of independence Language genealogy (Dryer, 1989, 1991, 1992; Cysouw, 2010; Jaeger et al., 2011) Language areas (Dryer, 1989, 2000; Maslova, 2000, Bickel, 2008; Cysouw, 2010; Jaeger et al., 2011) Bickel, B. (2008). A refined sampling procedure for genealogical control. STUF-Language Typology and Universals, 61(3), 221–233. Cysouw M. (2010). Dealing with diversity: Towards an explanation of NP-internal word order frequencies. Linguistic Typology, 14: 253–286. Dryer, M. (1989). Large linguistic areas and language sampling. Studies in Language, 13(2), 257–292. Dryer, M. (1991). SVO languages and the OV: VO typology. Journal of Linguistics, 27(2), 443–482. Dryer, M. (1992). The Greenbergian word order correlations. Language, 68(1), 81–138. Dryer, M. (2000). Counting genera vs. counting languages. Linguistic Typology, 4, 334–350. Jaeger, T. F., Graff, P., Croft, W., & Pontillo, D. (2011). Mixed effect models for genetic and areal dependencies in linguistic typology. Linguistic Typology, 15(2), Maslova, E. (2000). A dynamic approach to the verification of distributional universals. Linguistic Typology, 4(3), 307–333.

Roberts, S., & Winter, J., & Chen, K. (upcoming). Future tense and economic decisions: controlling for cultural evolution. PLOS ONE. Genealogical relationships

Roberts, S., & Winter, J., & Chen, K. (upcoming). Future tense and economic decisions: controlling for cultural evolution. PLOS ONE. Contact relationships

Roberts, S., & Winter, J., & Chen, K. (upcoming). Future tense and economic decisions: controlling for cultural evolution. PLOS ONE. Contact relationships

Mark Liberman Contact relationships

Two sources of non-independence Lack of independence Sean Roberts James Winters Roberts, S., & Winter, J., & Chen, K. (upcoming). Future tense and economic decisions: controlling for cultural evolution. PLOS ONE.

Two sources of non-independence Controlling for area and family within a mixed effects modeling framework: lmer(case ~ L2prop + (1+L2prop|family) + (1+L2prop|area)) Lack of independence

Three statistical problems Multiple comparisons Correlation is not causation Lack of independence