Chapter Eight: Quantitative Methods

Chapter Outline Defining Surveys and Experiments Components of a Survey Method Plan The Survey Design The Population and Sample Instrumentation Variables in the Study Data Analysis and Interpretation Components of an Experimental Method Plan Participants Variables Instrumentation and Materials Experimental Procedures Threats to Validity The Procedure Data Analysis Interpreting Results

Defining Surveys and Experiments Survey Design To provide a quantitative or numeric description of trends, attitudes, or opinions of a population by studying a sample of that population Experimental Design This approach tests the impact of a treatment or intervention on an outcome, controlling for all other factors that may influence that outcome. The sample is identified and generalizations are made to the population.

Components of a Survey Method Plan The Survey Design The Population and Sample Instrumentation Variables in the Study Data Analysis and Interpretation

Checklist of Questions for Designing a Survey Method Is the purpose of the survey stated? Are the reasons for choosing the design mentioned? Is the nature of the survey (cross-sectional vs. longitudinal) identified? Is the population and its size mentioned? Will the population be stratified? If so how? How many people will be in the sample? On what basis was this size chosen? What will be the procedure for sampling these individuals (e.g. random, nonrandom)? What instrument will be used in the survey? Who developed the instrument? What are the content areas addressed in the survey? The scales? What procedure will be used to pilot or field-test the survey? What is the timeline for administering the survey? What are the variables in the study? How do these variables cross-reference with the research questions and items on the survey?

Checklist of Questions for Designing a Survey Method Cont. What specific steps will be taken in data analysis to do the following: Analyze returns? Check for response bias? Conduct a descriptive analysis? Collapse items into scales? Check for reliability of scales? Run inferential statistics to answer research questions or assess practical implications of the results? How will the results be interpreted?

A Survey Method Plan The Survey Design Provide a purpose and rationale for using a survey for the proposed study Indicate why a survey is the preferred type of data collection procedure for the study, examine the advantages of this design for the study Indicate the type of survey design: Cross-sectional (data collected at one point in time) Longitudinal (data collected over time) Specify the form of data collection (telephone, mail, personal/group interviews, internet surveys), examine the strengths and weaknesses for the data collection method/s chosen

A Survey Method Plan The Population and Sample Identify the population in the study, state the size of the population, means of identifying individuals in the population and availability of sample frames Specify if the sampling design will be: Single-stage (there is access to the names of the people in the population and they may be sampled directly) Multi-stage (will identify groups or organizations who will assist in the gathering the names of individuals who will be sampled) Identify the selection process for the sample Probability: Randomized sampling e.g. random sampling Nonprobability: Nonrandomized sampling e.g. convenience sampling Indicate if the population and subsequent sample will be stratified based on specific population characteristics e.g. age, gender, income levels or education Discuss how sample will be selected from available lists Indicate the number of people in the sample and the procedures used to compute this number. Here attention must be given to the percentage/fraction of the population that will be included in the sample, margin of error, confidence interval, confidence level, and response rate

A Survey Method Plan Instrumentation Name the survey instrument used to collect data (designed or modified for this research) When using an existing instrument, describe the established validity scores from previous use. Validity scores in a survey identifies whether the instrument is a good one to be used: Content validity Predictive or concurrent validity Construct validity Mention whether scores resulting from past use of the instrument demonstrate reliability Test-retest correlations Consistency in test administration and scoring

A Survey Method Plan Instrumentation Cont. When one modifies or combines an instrument the original validity and reliability may not hold for the new instrument, reestablish validity and reliability in data analysis Include sample items from the instrument so that the readers can see the actual items used Label the major content sections in the instrument: Cover letter Items: demographics, attitude items, behavior items, factual items Closing instructions Type of scales used to measure the items: strongly agree to strongly disagree Discuss plant and rationalize the pilot testing and field-test the survey For mailed survey, identify steps for administering the survey and for following up

A Survey Method Plan Variables in the Study Useful in the methods section to relate the variables to research questions and items on the instrument Allows the reader to easily determine how the data collection connects to the variables and question or hypotheses Allows for cross-referencing the variables, the questions or hypotheses, and specific survey items

A Survey Method Plan Data Analysis and Interpretation Present the steps for analyzing the data Step 1. Report response rate Step 2. Determine response bias: the effect of nonresponses on survey estimates Step 3. Discuss plan to provide descriptive analyses Step 4. Check instrument's scales Step 5. Statistics and statistical computer program for inferential statistical analyses (see Table 8.3) Step 6. Present and interpret results Report how the results answered the research question or hypotheses Discuss the implications for the results for practice or future research on the topic, draw inferences and conclusions from results

Criteria for Choosing Select Statistical Tests

An Experimental Method Plan Experimental Design To test the impact of a treatment on an outcome, controlling for other factors that might influence that outcome Components of an Experimental Method Plan Participants Materials Procedures Measures

An Experimental Method Plan Checklist for designing an experimental procedure Who are the participants in the study? What is the population to which the results of the participants will be generalized? How are the participants selected? Was a random selection method used? How will the participants be randomly assigned? Will they be matched? How? How many participants will be in the experimental and control group(s)? What is the dependent variable or variables (i.e., outcome variable) in the study? How will it be measured? Will it be measured before or after the experiment? What is the treatment condition(s)? How was it operationalized? Will variables be covaried in the experiment? How will they be measured? What experimental research design will be used? What would a visual model of this design look like? What instrument(s) will be used to measure the outcomes of the study? Why was it chosen? Who developed it? Does it have established validity and reliability? Has permission been sought to use it? What are the steps in the procedure (e.g. random assignment of the participants to groups, collection of demographic information, administration of pretest, administration of treatment(s), administration of posttest)? What are potential treats to the internal and external validity for the experimental design and procedure? How will they be addressed? Will a pilot test of the experiment be conducted? What statistics will be used to analyze the data (e.g. descriptive and inferential)? How will the results be interpreted?

An Experimental Method Plan Participants Describe the selection of participants as either: Random Nonrandom (convenience) Indicate if it is a true experiment or not Identify other features in the experimental design that will influence the outcome Describe the assignment of participants to groups and the procedure for determining group size Level of statistical significance The amount of power desired The effect size The experiment is planned so that the size of each treatment group provides the greatest sensitivity that the effect on the outcome actually is due to the experimental manipulation in the study

An Experimental Method Plan Variables Specifying the variables in an experiment identifies the group receiving the experimental treatment and the outcomes being measured Clarify the groups Identify the independent variable(s), including the treatment variable Identify the dependent variable(s), the outcomes

An Experimental Method Plan Instrumentation and Materials Describe the instrument(s) participants complete in the experiment development, items, and scales reliability and validity reports of past uses Thoroughly discuss materials used for the treatment Experimental Procedures Identify the type of experiment Pre-experimental, true experiment, quasi-experiment, and single-subject designs Identify the type of comparisons: within-group or between-subject Provide a visual model to illustrate the research design used X = treatment O = observation See Examples: 8.2, 8.3, 8.4 and 8.5

An Experimental Method Plan Consider Threats to Validity Threats to internal validity: procedures, treatments, or experiences of the participants that threaten the researcher's ability to draw conclusions about cause and effect Threats to external validity: characteristics of the sample, setting, or timing that threaten the researcher's ability to generalize the conclusions to a population Threats to statistical conclusion validity: inadequate statistical power or violation of statistical assumptions that threaten the researcher's ability to draw statistical inferences Threats to construct validity: inadequate definitions and measures of variables that threaten the researcher's ability to measure relevant constructs

Threats to Validity (Tables 8.5 & 8.6) Threats to Internal Validity History Maturation Regression Selection Mortality Diffusion of treatment Compensatory/resentful demoralization Compensatory rivalry Testing Instrumentation Threats to External Validity Interaction of selection and treatment Interaction of setting and treatment Interaction of history and treatment

An Experimental Method Plan The Procedure Describe the step-by-step procedure for the experiment Administer measures of the dependent variable or a variable closely correlated with the dependent variable to the research participants Assign participants to match pairs on the basis of their scores on the measures described in in step 1 Randomly assign one member of each pair to the control and experimental group Expose experimental group to the treatment Measure dependent variables to experimental and control groups Compare performance of the experimental and control groups on the posttest(s) using statistical significance

An Experimental Method Plan Data Analysis Report descriptive statistics (e.g., means, standard deviations, ranges) Conduct inferential statistical tests (e.g., t test, ANOVA, ANCOVA, or MANOVA) Use line graphs for single subject designs Report confidence intervals and effect sizes in addition to statistical tests Interpreting Results Discuss results, limitations, and implications