 Research Design Part 2 Variability, Validity, Reliability

Selecting an Instrument  Reliability & validity of the instrument  Are the subject characteristics the same  Is the instrument the best one? Why?  Purchase cost?  Availability? Copyright?  Simplicity of administering & scoring

 Should you use an existing instrument, modify an instrument, develop a new instrument?

Existing, Modifying, New Instruments  Pros  Quick to use  Have established R & V  Can build on knowledge base est. with instrument  Cons  May not “fit” research question exactly  May require training for administration, scoring, or analysis  May incur cost to purchase, score, or analyze…MBTI  May be too long for the purposes at hand, take too much time to complete

Existing, Modifying, New Instruments  Pros  Can be modified to better suit research question  Most of the work of creating the tool has been completed  May be able to compare some results with previous results  Cons  Changing a known quantity into something unknown  Previous reliability and validity indicators may no longer apply

Existing, Modifying, New Instruments  Pros  Can develop instrument to fit specific need  Instrument itself may make a significant contribution to the field of research  Cons  Requires time, effort, resources, expertise  Requires knowledge of scale development procedures  Runs risk that instrument will not be reliable or valid for purpose at hand

Modifying or Developing a New Inst.  Determine what behaviors/traits to measure  Intrinsic vs. extrinsic rewards  Review the literature to determine how traits are measured  Herzberg; rec literature  Consult experts in the field to review the instrument  Pilot test  Clarity, ambiguity, time of completion, directions  Revise  Check for a distribution of scores  Is there a problem with question or lack of variance on the item

 Design Validity

 4 types  Statistical conclusion validity  Construct validity  Internal validity  External validity

Statistical Conclusion Validity  Accurate determination of whether a relationship exists  Inflated error rates from multiple tests  Extraneous variance  Low statistical power

Construct Validity  Degree to which a test/ measurement measures a hypothetical construct  Threats  Using one method to measure the construct  Inadequate explanation of a construct  Measuring just one construct & making inferences  Using 1 item to measure personality

External Validity  Generalizability of the results  Population external validity  Characteristics & results can only be generalized to those with similar characteristics  Demographics  Psych experiments with college students  Ecological external validity  Conditions of the research are generalizable to similar characteristics  Physical surroundings  Time of day

Internal Validity  Internal validity is strongest when the study’s design (subjects, instruments/measurements, and procedures) effectively controls possible sources of error so that those sources are not reasonably related to the study’s results.

Internal Validity  History  Extraneous incidents/events that occur during the research to effect results  Only impacts studies across time  Attendance at football games/coaching change  Selection  If there are systematic differences in groups of subjects  Gender  Compare GRE scores & grad school performance between sequences  Occurs when random sampling isn’t used

Internal Validity  Statistical regression  If doing pre-test/post-test those scoring extremely high or low on first test will often “regress to the mean” on the second test  Scoring based more on luck than actual performance  The regression effect causes the change & not the treatment  Don’t group the high/low scores for the post-test

Internal Validity  Pre-testing  Pre-test can increase/decrease motivation  Gives subjects opportunities to practice  Practice can be a positive so they get a true score

Internal Validity  Instrumentation  Changes in calibration of the exam  GRE – new & old  Changes in observer scoring  Fatigue/ boredom  American Idol  Attrition/Mortality  Subjects drop out/lost  Low scorers on GRE drop out of grad school school

Internal Validity  Experimenter effect  Presence, demeanor of researcher impacts +/-  Course instructor is PI  Course evals  Subject effect  Subjects’ behaviors change because they are subjects  Subjects want to present themselves in the best light  Hawthorn effect

 Practice problems