1. CGT 411 Research Presentation
Conducting Research: What Do You Need to Think About?
Part 3 – Instruments and Participants

2. The Nature of Sampling
The idea is to examine elements of a population in order to draw conclusions about the population as a whole
The (experimental) population is the group of elements about which we want to make inferences
A count of all elements of a population would be known as a census
Sample elements = data points

3. Why do we sample?
Lower cost  it is normally too expensive to look at everyone
Greater accuracy  fewer data points means less to keep track of
Faster  this greatly reduces cost
Availability of population elements  every population element is often not accessible
A census is feasible when populations are small and necessary when elements are not homogeneous

4. Overview of Research
Good research should follow the standards of the scientific method
Purpose clearly defined
Research process detailed
Research design thoroughly planned
Limitations frankly revealed
High ethical standards applied
Adequate analysis for decision-maker’s needs
Findings presented unambiguously
Conclusions justified
Researcher’s experience reflected

5. Exploratory Research Interviewing Observation Extant data
Document analysis
Case studies
Ethnography
Expert interviewing
Secondary data analysis
Experience surveys
Focus groups
Two-stage designs
Instances of data should be analyzed that provide meaningful information within the context of the organization

6. Communicating with Participants
Observations
Interviews
Surveys

7. Getting to the Measurement Questions
What type of data is needed to answer the investigator’s dilemma?
What communication approach will be used?
Should the questions be structured, unstructured, or some combination?
Administrative: identify setting characteristics
Classification: sociological/demographic
Target: address the issues at hand
Should the questions be disguised or undisguised?  have to be careful here  potential IRB issues

8. Constructing and Refining Research Questions
Measurement questions should:
Consider subject content
Consider the wording of each question
Consider the response strategy
Question development and question sequencing are typically completed at the same time

9. Drafting and Refining the Instrument
The introduction has to hook ‘em and give them the particulars  also has to filter them
Some questions depend on other having or not having been asked already
Awaken interest and motivation
Put sensitive questions at the end
Balance complexity of questions and their location
General to specific
Group questions to minimize shifting frame of reference and subject matter

10. Drafting and Refining the Instrument
Pre-testing/pilot-testing is valuable for having a reliable instrument
Discover participants’ reactions to the instrument
Ensure meaning is conveyed
Why do participants “transform” questions they do not understand?
Improve continuity and flow of the instrument
Make sure question sequence is appropriate
Helps understand variability in population
Helps to gauge length and time

11. Interview Considerations
Should the question be asked?
Is the question of the proper scope and coverage?
Can the participant adequately answer the question as it is asked?
Will the participant willingly answer the question as it is asked?

12. Observation Considerations
Much of what we know comes from observation  but the collection process often leaves much to be desired
Scientifically based
Conducted to answer a specific question
Systematically planned and executed
Uses appropriate control mechanisms
Reliable and valid account of what happened
Observation is generally behavioral
Nonverbal analysis: body movement
Linguistic analysis: patterns/content of speech
Spatial analysis: physical relationship to surroundings
Extra-linguistic analysis: characteristics of speech

13. What is Experimentation?
The emphasis is towards discovering a causal relationship
Accepting the world as it is found (ex post fact, communication, observation) vs. systematically altering a variable and observing the change (experimentation)
Is the experimental treatment at the heart of the observed change, or are other factors influencing the outcome?

14. What is Experimentation?
Requires the intervention of a researcher beyond that required for measurement
Manipulation of independent variables to affect a hypothesized dependent variable  at least one IV and one DV in an experiment
Must be an agreement between variables  presence/absence of one relates to the presence/absence of another
Time order of variables must make sense
Researchers must be confident that extraneous factors did not cause such a change
Often involves the presence of a contrived research scenario  standardized to aid generalizability

15. Elements of Scientific Method
Direct observation of phenomena
Clearly defined variables, methods, and procedures
Empirically testable hypotheses
Ability to rule out rival hypotheses
Statistical justification of conclusions
Self-correcting process

16. Descriptive Study
Descriptions of phenomena associated with a particular population
Estimates of the proportion of the population that would have such characteristics
Discovery of associations among different variables
The simplest descriptive study concerns a statement about the size, form, distribution, or existence of a variable  no causal relationship implied

17. Causal Studies
Correlation is not causation essentially “A produces B”  an artifact of language
We cannot prove that A causes B, but we can find evidence that leads us to believe that A leads to B
Covariation of A and B
Time order of events in the direction specified
No other possible causes
Inference-making also requires:
Control  no other confounding causes
Random assignment along the IV

18. Advantages Disadvantages
Researcher’s ability to manipulate the IV
Influence of extraneous variables can be controlled
Convenient and cost-effective
Ability to replicate across people, places, and time
Can use naturally occurring events
Contrived setting
Generalization from non-probability samples can be problematic
Budgets can be deceiving
Most effective for problems in the present or the immediate future
Ethical issues in the manipulation of people

19. Conducting an Experiment
Select relevant variables
Specify levels of treatment
Control the experimental environment
Choose the experimental design
Select and assign the participants
Pilot test  revise  test
Analyze the data

20. The Logic of Hypothesis Testing
The null hypothesis  a statement that no significant difference exists between a parameter and a statistic being compared to it.
Why not state this in a positive form?
Evidence would preclude a positive hypothesis from being taken as conclusive evidence for accepting the hypothesis
Could be consistent with other hypotheses as well

21. The Logic of Hypothesis Testing
The alternative hypothesis  a statement that there is a significant difference between the parameter and the statistic
This is the logical opposite to the null hypothesis
May take several forms depending on the objective of the researcher
Not equal to (not the same as)
Greater than
Less than