1. Copyright © 2011 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill/Irwin Making Systematic Observations

2. 2 Research Tradition Your variables may be similar to those included in previous studies You may use previously used dependent variables while manipulating new independent variables Theory A theory on which you are relying may suggest certain variables to be included

3. 3 Availability of New Techniques Sometimes a new technique is developed, allowing you to observe a variable that previously could not be observed For example: fMRI, PET scans, implicit measures Availability of Equipment The variables you manipulate or observe may be limited by the equipment available to you

4. 4 The RELIABILITY of a Measure A reliable measure produces similar results when repeated measurements are made under identical conditions How you assess reliability depends on the type of measure Physical measure: Repeatedly measure a fixed attribute (e.g., weight) to establish the precision of the measure Population estimates: Expressed as a margin of error Ratings of multiple observers: Demonstrate interrater reliability with the appropriate statistic Psychological tests: Various methods used

5. 5 Reliability of a psychological test can be established in several ways Test-retest reliability: Administer the same test twice Problems Respondents remember your test, inflating reliability Respondents may change between administrations, reducing reliability Best used for Stable characteristics (e.g., intelligence) Variables that will not change much between administrations Test administrations that can be spaced widely over time

6. 6 Methods to avoid problems with test- retest reliability Parallel-forms reliability: Alternate forms of the same test used Items on parallel forms must be equivalent Split-half reliability: Parallel forms are included on one test and later separated for comparison

7. 7 ACCURACY of a Measure An accurate measure produces results that agree with a known standard A measurement instrument can be inaccurate but reliable The reverse cannot be true In psychology, standards are rare so accuracy may not be possible to establish

8. 8 The VALIDITY of a Measure A valid measure measures what you intend it to measure Most important when using psychological tests (e.g., IQ test) Validity can be established in a variety of ways Face validity: Assessment of adequacy of content Least powerful method Content validity: How adequately does a test sample behavior it is intended to measure? A measure has content validity if its items are relevant to construct being measured For example, a final exam for a course has content validity if items on the exam sample the course material

9. 9 Criterion-related validity: How adequately does a test score match some criterion score? Takes two forms Concurrent validity: Does test score correlate highly with score from a measure with known validity? For example, correlate scores on a new IQ test with those from the Stanford-Binet Predictive validity: Does test predict behavior known to be associated with the behavior being measured? For example, does the GRE predict performance in graduate school? Construct validity: Do the results of a test correlate with what is theoretically known about the construct being evaluated? For example, do scores on an IQ test correlate with how intelligent people behave?

10. 10 Using Established Versus New Measures You need not spend time assessing validity and reliability of established measures However, they may not meet your research needs New measures must be evaluated for validity and reliability, which takes time and effort However, they may better meet your research needs

11. 11 Nominal Scale Lowest scale of measurement involving variables whose values differ by category (e.g., male/female) Values of variables have different names, but no ordering of values is implied You can count number of observations falling into categories, but cannot apply mathematical operations Ordinal Scale Higher scale of measurement than nominal scale Different values of a variable can be ranked according to quantity (e.g., high, moderate, or low self-esteem) Mathematical operations likely to produce misleading results

12. 12 Interval Scale Scale of measurement on which the spacing between values is known For example, rating a book on a scale ranging from 0 to 10 No true zero point Can apply mathematical operations Cannot make ratio judgments (e.g., on book liked twice as much as another) Ratio Scale Similar to interval scale, but with a true zero point For example, number of lever presses Can apply mathematical operations and make ration comparisons (e.g., 4 lever presses are twice as many as 2 lever presses)

13. 13 Information Yielded A nominal scale yields the least information An ordinal scale adds some crude information Interval and ratio scales yield the most information Statistical Tests Available The statistical tests available for nominal and ordinal data (nonparametric) are less powerful than those available for interval and ratio data (parametric) Use the scale that allows you to use the most powerful statistical test

14. 14 Ecological Validity Sometimes your research requires you to use a measure that reflects what people do in the real world (e.g., a nominal guilty/not guilty verdict) Such scales have ECOLOGICAL VALIDITY When necessary, choose an ecologically valid measure, even if it means loss of information There are things you can do to gain needed information Include an additional scale (e.g., an interval scale) Create a composite scale by combining a nominal and interval scale Incorporate features of ecologically valid scale into a more informative scale

15. 15 Adequacy of a dependent measure is related to the sensitivity of the measure and range effects Sensitivity Is a dependent measure sensitive enough to detect behavior change? An insensitive measure will not detect subtle behaviors

16. 16 Range Effects Occur when a dependent measure has an upper or lower limit Ceiling effect: When a dependent measure has an upper limit Floor effect: When a dependent measure has a lower limit. Affect data in two ways Limiting values of your highest or lowest data point Variability of scores within affected treatments is reduced May cause misleading results from statistical analysis of data

17. 17 Measures may need to be tailored to special needs of population of participants Measures used for young children must be tailored to their level of understanding Represent rating scales graphically rather than numerically Special measures are used for preverbal infants Habituation technique: Capitalizes on fact that infants get bored with repeatedly presented stimuli Preference technique: Present two stimuli and measure fixation time to each Discrimination learning: Train different behaviors to different stimuli

18. 18 Measures may also need to be tailored for special adult populations May need to represent complex rating scales visually Pretesting measures can help identify situations where measures must be tailored

19. 19 Behavioral Measure Record actual behavior of subjects Several types Frequency: Count of the number of behaviors that occur Latency: The amount of time it takes for a behavior to occur Number of errors: The number of incorrect responses made May not allow you to determine the underlying cause for behavior

20. 20 Physiological Measure Physical measure of body function (e.g., EEG) Typically requires special equipment Most physiological measures are noninvasive Some require an invasive procedure (e.g., implanting an electrode in the brain of a rat) Allow you to make precise measurements of arousal of a subject’s body Must infer psychological states

21. 21 Self-Report Measure Participants report on their own behavior or state of mind A rating scale is a commonly used self-report measure e.g., rate the attractiveness of a person on a 0 to 10 scale Q-sort methodology is another popular self-report measure Self-report measures are popular and easy to use, but may have questionable reliability and validity You cannot be sure that a participant is telling you the truth when using a self-report measure

22. 22 Implicit Measures Measure of responses that are not under direct conscious control Implicit Associations Test (IAT) is a popular example of an implicit measure A person should react more quickly to positive characteristics associated with a preferred group than to a nonpreferred group These measures often show negative attitudes that are not picked up with more traditional measures

23. 23 A psychological study is a social situation A participant’s social history can affect how he or she responds to a study You should not assume that your participant is a passive recipient of the parameters of your study Simply observing someone changes his or her behavior A problem in human and animal research

24. 24 Cues provided by the researcher and the research context that give participants information about a study The cues focused on by the participant may not be relevant to your study and can adversely affect results

25. 25 Role attitude cues (attitude adopted by a participant) can affect outcome of a study o Cooperative attitude: Participant wants to help researcher o Defensive or apprehensive attitude: Participant is suspicious of experimenter and situation o Negative attitude: Participant motivated to ruin a study Events outside the laboratory can affect results of an experiment

26. 26 An experimenter can unintentionally affect how a participant behaves in a study Experimenter bias occurs when the experimenter’s behavior influences a participant’s behavior Two sources of experimenter bias Expectancy effects: When an experimenter expects certain types of behavior from participants Treating different groups differently: Treating participants differently, depending on the condition to which they were assigned

27. 27 Experimenter bias affects internal and external validity Steps must be taken to reduce experimenter bias Use a blind technique where the experimenter does not know the condition to which a participant has been assigned Use a double-blind technique where neither the experimenter nor participant knows the condition to which a participant has been assigned Automate the experiment

28. 28 Animal subjects may be affected by demand characteristics and experimenter bias Expectations about learning capabilities of rats Blind techniques should be used Demand characteristics affect animal behavior Failure to clean apparatus between subjects leave odor cues for subsequent subjects Reactivity issues affect the internal validity of an experiment

29. 29 Automating experiments Can help eliminate experimenter bias Can save time because you do not need to be present to run subjects Increase the accuracy and reduce variability of measurement If you automate, you may miss important details of behavior Make some visual observations Special equipment is needed for automation (e.g., computers, videotape)

30. 30 You can take two steps to avoid problems with your research design Conduct a pilot study Run a small-scale version of your study in which you can test your materials and procedures Problems can be corrected before you run your actual study Include manipulation checks in your experiment Measures to assess the success of your independent variables May provide information that can help you interpret your results later on