Everything I wish I had known about research design and data analysis… Statlab Workshop Fall 2006 Kyle Hood and Frank Farach

Outline of a paper Introduction Theory Data Description Analysis Conclusion

Identifying a Question Tradeoff between work in and results Tradeoff between work in and results Easy to do, trivial results Easy to do, trivial results Result is interesting, but difficulty is high Result is interesting, but difficulty is high New tools open up new questions New tools open up new questions New statistical or computational tools make formerly difficult questions approachable New statistical or computational tools make formerly difficult questions approachable New theory opens up new questions New theory opens up new questions

Introduction Topic Most general level Most general levelQuestion What is the question you want to answer? What is the question you want to answer? Be specific Be specific Ask only what you can answer Ask only what you can answer Review the Literature “Stay the course” “Stay the course”

Theory Categorize your theory Descriptive vs. causal Descriptive vs. causal Write down your theory In paragraph form In paragraph form Using a statistical model Using a statistical model Identify testable hypotheses from theory Do you need statistics after all? Quantitative v. Qualitative research Quantitative v. Qualitative research

Variables Dependent Variable ( response, outcome, criterion) Independent Variables ( explanatory or predictor variables) Treatment Variable Treatment Variable Covariates / Confounding Variables Covariates / Confounding Variables Categorical and Continuous Variables Remember: Types of variables we choose, determine the statistics we use

You need Data Think about analyses early! Collecting your own data Retrospective, prospective, experimental & observational methods Retrospective, prospective, experimental & observational methods Can find most data you’ll need on-line! Statlab Webpage ( Advisors Advisors Yale StatCat ( Yale StatCat ( ICPSR ( ICPSR ( Reference Librarian (Julie Linden) Reference Librarian (Julie Linden)

Endogeneity Problem: “Independent” variables are not really independent The “dependent” and “independent” variables are determined in equilibrium (example: effect of education on wages) The “dependent” and “independent” variables are determined in equilibrium (example: effect of education on wages) Treatment effects will be biased Treatment effects will be biased Modeling approaches to deal with this Assumption-based methods Assumption-based methods Instruments Instruments

So, you want to make a survey Extensive on-line resources and software Extensive on-line resources and software Question types determine analyses Question types determine analyses Open vs. close ended questions, Likert scales, rank order data Open vs. close ended questions, Likert scales, rank order data Assumptions of normality Assumptions of normality Validity Validity Internal & External validity Internal & External validity Pilot testing Pilot testing You need variance to analyze! You need variance to analyze! Sample size Sample size It depends; power, effect size, cost (UCLA power calculator) It depends; power, effect size, cost (UCLA power calculator)

Once You’ve Found or Collected your data Download the data and documentation StatTransfer (Statlab) StatTransfer (Statlab) Determine data file type Probably a text file (.txt,.dat,.raw) Probably a text file (.txt,.dat,.raw) Converting text & delimited files Choose a statistical software program SPSS, Stata, SAS, Matlab, Excel, R, C++ SPSS, Stata, SAS, Matlab, Excel, R, C++

Managing your data Back up all Master Data Files CDR/CDRW, USB Key Codebook All codes All codes Adding variables, cases, computing new variables Adding variables, cases, computing new variables Keep a roadmap Keep a log of all analyses with what you have done Keep a log of all analyses with what you have done Save syntax files Save syntax files

Syntax Files What are they? Text-files used to enter commands in bulk Is it worth learning? You will make mistakes, need to make changes SPSS and many other programs let you use pull down menus How do I know what to write? Program’s manual provides the underlying command

So, how do I analyze my data? Correlation Correlation allows you to quantify relationships between variables (r, r-squared) Correlation allows you to quantify relationships between variables (r, r-squared) Regression allows prediction of dependent variable based on one or more independent variables Regression allows prediction of dependent variable based on one or more independent variables Group differences t-test & ANOVA t-test & ANOVA Chi-square for categorical and frequency data Chi-square for categorical and frequency data Significance v. effect size More Complex Models

Descriptive Statistics Variables Dependent Variable(s) Independent Variable(s) Important Covariates Graphs Summary Statistics on Key Variables Number, Mean, Minimum, Maximum, Standard Deviation Cross-Tabs

Putting Output into a Paper Cut and Paste Graphs Cut and Paste into Word Processing document Save as.jpeg or.tif file Tables Cut and Paste Format in Word Processing document Import into Excel, format, and then place in Word

More Advanced Analysis Multivariate techniques help to account for confounding factors, allow for testing change over time and more complex hypotheses … (See: Tabachnick & Fidell, Using Multivariate Statistics) 1) Be honest about your abilities. 2) Ask for help 3) Best off including techniques that you fully understand.

Take Away Messages 1) Determine your question, methods and statistics before you start 2) Keep a codebook of everything 3) Keep a log of all commands issued 4) Save data at every step 5) Ask for help 6) Don’t get in over your head