1. Chapter Eight: Quantitative Methods

2. Chapter Outline Defining Surveys and Experiments
Components of a Survey Method Plan
The Survey Design
The Population and Sample
Instrumentation
Variables in the Study
Data Analysis and Interpretation
Components of an Experimental Method Plan
Participants
Variables
Instrumentation and Materials
Experimental Procedures
Threats to Validity
The Procedure
Data Analysis
Interpreting Results

3. Defining Surveys and Experiments
Survey Design
To provide a quantitative or numeric description of trends, attitudes, or opinions of a population by studying a sample of that population
Experimental Design
This approach tests the impact of a treatment or intervention on an outcome, controlling for all other factors that may influence that outcome. The sample is identified and generalizations are made to the population.

4. Components of a Survey Method Plan
The Survey Design
The Population and Sample
Instrumentation
Variables in the Study
Data Analysis and Interpretation

5. Checklist of Questions for Designing a Survey Method
Is the purpose of the survey stated?
Are the reasons for choosing the design mentioned?
Is the nature of the survey (cross-sectional vs. longitudinal) identified?
Is the population and its size mentioned?
Will the population be stratified? If so how?
How many people will be in the sample? On what basis was this size chosen?
What will be the procedure for sampling these individuals (e.g. random, nonrandom)?
What instrument will be used in the survey? Who developed the instrument?
What are the content areas addressed in the survey? The scales?
What procedure will be used to pilot or field-test the survey?
What is the timeline for administering the survey?
What are the variables in the study?
How do these variables cross-reference with the research questions and items on the survey?

6. Checklist of Questions for Designing a Survey Method Cont.
What specific steps will be taken in data analysis to do the following:
Analyze returns?
Check for response bias?
Conduct a descriptive analysis?
Collapse items into scales?
Check for reliability of scales?
Run inferential statistics to answer research questions or assess practical implications of the results?
How will the results be interpreted?

7. A Survey Method Plan The Survey Design
Provide a purpose and rationale for using a survey for the proposed study
Indicate why a survey is the preferred type of data collection procedure for the study, examine the advantages of this design for the study
Indicate the type of survey design:
Cross-sectional (data collected at one point in time)
Longitudinal (data collected over time)
Specify the form of data collection (telephone, mail, personal/group interviews, internet surveys), examine the strengths and weaknesses for the data collection method/s chosen

8. A Survey Method Plan The Population and Sample
Identify the population in the study, state the size of the population, means of identifying individuals in the population and availability of sample frames
Specify if the sampling design will be:
Single-stage (there is access to the names of the people in the population and they may be sampled directly)
Multi-stage (will identify groups or organizations who will assist in the gathering the names of individuals who will be sampled)
Identify the selection process for the sample
Probability: Randomized sampling e.g. random sampling
Nonprobability: Nonrandomized sampling e.g. convenience sampling
Indicate if the population and subsequent sample will be stratified based on specific population characteristics e.g. age, gender, income levels or education
Discuss how sample will be selected from available lists
Indicate the number of people in the sample and the procedures used to compute this number. Here attention must be given to the percentage/fraction of the population that will be included in the sample, margin of error, confidence interval, confidence level, and response rate

9. A Survey Method Plan Instrumentation
Name the survey instrument used to collect data (designed or modified for this research)
When using an existing instrument, describe the established validity scores from previous use. Validity scores in a survey identifies whether the instrument is a good one to be used:
Content validity
Predictive or concurrent validity
Construct validity
Mention whether scores resulting from past use of the instrument demonstrate reliability
Test-retest correlations
Consistency in test administration and scoring

10. A Survey Method Plan Instrumentation Cont.
When one modifies or combines an instrument the original validity and reliability may not hold for the new instrument, reestablish validity and reliability in data analysis
Include sample items from the instrument so that the readers can see the actual items used
Label the major content sections in the instrument:
Cover letter
Items: demographics, attitude items, behavior items, factual items
Closing instructions
Type of scales used to measure the items: strongly agree to strongly disagree
Discuss plant and rationalize the pilot testing and field-test the survey
For mailed survey, identify steps for administering the survey and for following up

11. A Survey Method Plan Variables in the Study
Useful in the methods section to relate the variables to research questions and items on the instrument
Allows the reader to easily determine how the data collection connects to the variables and question or hypotheses
Allows for cross-referencing the variables, the questions or hypotheses, and specific survey items

12. A Survey Method Plan Data Analysis and Interpretation
Present the steps for analyzing the data
Step 1. Report response rate
Step 2. Determine response bias: the effect of nonresponses on survey estimates
Step 3. Discuss plan to provide descriptive analyses
Step 4. Check instrument's scales
Step 5. Statistics and statistical computer program for inferential statistical analyses (see Table 8.3)
Step 6. Present and interpret results
Report how the results answered the research question or hypotheses
Discuss the implications for the results for practice or future research on the topic, draw inferences and conclusions from results

13. Criteria for Choosing Select Statistical Tests

14. An Experimental Method Plan
Experimental Design
To test the impact of a treatment on an outcome, controlling for other factors that might influence that outcome
Components of an Experimental Method Plan
Participants
Materials
Procedures
Measures

15. An Experimental Method Plan
Checklist for designing an experimental procedure
Who are the participants in the study?
What is the population to which the results of the participants will be generalized?
How are the participants selected? Was a random selection method used?
How will the participants be randomly assigned? Will they be matched? How? How many participants will be in the experimental and control group(s)?
What is the dependent variable or variables (i.e., outcome variable) in the study? How will it be measured? Will it be measured before or after the experiment?
What is the treatment condition(s)? How was it operationalized?
Will variables be covaried in the experiment? How will they be measured?
What experimental research design will be used? What would a visual model of this design look like?
What instrument(s) will be used to measure the outcomes of the study? Why was it chosen? Who developed it? Does it have established validity and reliability? Has permission been sought to use it?
What are the steps in the procedure (e.g. random assignment of the participants to groups, collection of demographic information, administration of pretest, administration of treatment(s), administration of posttest)?
What are potential treats to the internal and external validity for the experimental design and procedure? How will they be addressed?
Will a pilot test of the experiment be conducted?
What statistics will be used to analyze the data (e.g. descriptive and inferential)?
How will the results be interpreted?

16. An Experimental Method Plan
Participants
Describe the selection of participants as either:
Random
Nonrandom (convenience)
Indicate if it is a true experiment or not
Identify other features in the experimental design that will influence the outcome
Describe the assignment of participants to groups and the procedure for determining group size
Level of statistical significance
The amount of power desired
The effect size
The experiment is planned so that the size of each treatment group provides the greatest sensitivity that the effect on the outcome actually is due to the experimental manipulation in the study

17. An Experimental Method Plan
Variables
Specifying the variables in an experiment identifies the group receiving the experimental treatment and the outcomes being measured
Clarify the groups
Identify the independent variable(s), including the treatment variable
Identify the dependent variable(s), the outcomes

18. An Experimental Method Plan
Instrumentation and Materials
Describe the instrument(s) participants complete in the experiment
development, items, and scales
reliability and validity reports of past uses
Thoroughly discuss materials used for the treatment
Experimental Procedures
Identify the type of experiment
Pre-experimental, true experiment, quasi-experiment, and single-subject designs
Identify the type of comparisons: within-group or between-subject
Provide a visual model to illustrate the research design used
X = treatment
O = observation
See Examples: 8.2, 8.3, 8.4 and 8.5

19. An Experimental Method Plan
Consider Threats to Validity
Threats to internal validity: procedures, treatments, or experiences of the participants that threaten the researcher's ability to draw conclusions about cause and effect
Threats to external validity: characteristics of the sample, setting, or timing that threaten the researcher's ability to generalize the conclusions to a population
Threats to statistical conclusion validity: inadequate statistical power or violation of statistical assumptions that threaten the researcher's ability to draw statistical inferences
Threats to construct validity: inadequate definitions and measures of variables that threaten the researcher's ability to measure relevant constructs

20. Threats to Validity (Tables 8.5 & 8.6)
Threats to Internal Validity
History
Maturation
Regression
Selection
Mortality
Diffusion of treatment
Compensatory/resentful demoralization
Compensatory rivalry
Testing
Instrumentation
Threats to External Validity
Interaction of selection and treatment
Interaction of setting and treatment
Interaction of history and treatment

21. An Experimental Method Plan
The Procedure
Describe the step-by-step procedure for the experiment
Administer measures of the dependent variable or a variable closely correlated with the dependent variable to the research participants
Assign participants to match pairs on the basis of their scores on the measures described in in step 1
Randomly assign one member of each pair to the control and experimental group
Expose experimental group to the treatment
Measure dependent variables to experimental and control groups
Compare performance of the experimental and control groups on the posttest(s) using statistical significance

22. An Experimental Method Plan
Data Analysis
Report descriptive statistics (e.g., means, standard deviations, ranges)
Conduct inferential statistical tests (e.g., t test, ANOVA, ANCOVA, or MANOVA)
Use line graphs for single subject designs
Report confidence intervals and effect sizes in addition to statistical tests
Interpreting Results
Discuss results, limitations, and implications