CHAPTER OVERVIEW Deciding on a Method Tests and Their Development Types of Tests Observational Techniques Questionnaires

SOME IMPORTANT FIRST CONSIDERATIONS Is the tool you propose to use reliable and valid? Base your choice of research tools on how you have asked the research question

TESTS AND THEIR DEVELOPMENT A test should measure the nature and extent of individual differences A good test differentiates people based on true scores

WHY USE TESTS? Help determine outcomes of experiments Can be used to diagnose strengths and weaknesses Assist in placing individuals in appropriate programs Assist in selecting applicants Used to evaluate a program’s effectiveness

TYPES OF TESTS Overview What It Does Achievement TestAssesses an individual’s knowledge of a specific area Attitude TestAssesses an individual’s feelings about an issue Personality TestAssesses stable individual behavior patterns

ACHIEVEMENT TEST SOURCES Standardized –Commercially prepared for wide use –Scoring instructions included Researcher-made –Designed by user for specific purpose –Scoring instructions specific to test

ACHIEVEMENT TESTS REFERENCING What Comparison Do Tests Make? Norm-referenced—Individual’s scores are compared to results from a larger group Criterion-referenced—Individual’s scores are compared to defined performance standards

MULTIPLE-CHOICE ACHIEVEMENT ITEMS Anatomy of a Multiple-Choice Item 12. Intelligence tests that are given to preschool children a. favor middle-class children b. have questionable construct validity c. are based on motor skills d. are no fun at all STEM Clearly written CORRECT ANSWER DISTRACTERS Should be plausible (b & c), not easily ruled out (d)

TO USE OR NOT TO USE? Advantages –Versatile –Easy to score –Simple to take –Poor writers not penalized –Good items used again –Good distracters are diagnostic –Hard to fake correct answer Disadvantages –Limit student’s options –No opportunity to practice writing –Some people don’t do well on them –Limits content to be assessed –Items must be well written

ITEM ANALYSIS: HOW TO TELL IF YOUR ITEMS WORK Questions should discriminate those who know the material from those who don’t Item analysis provides two measures of a question’s ability to discriminate –Difficulty index –Discrimination index

COMPUTING INDICES First Steps 1.Rank scores from highest to lowest 2.Choose top 27% of scores for “high” group 3.Choose bottom 27% of scores for “low” group

COMPUTING INDICES 12. Intelligence tests that are given to preschool children a. favor middle-class children b. have questionable construct validity c. are based on motor skills d. are no fun at all Item AlternativeABCDTotal High Group (n = 41) Low Group (n = 41) Total

COMPUTING INDICES Difficulty index Proportion who answered item correctly D = NC h + NC l T Discrimination index Proportion in high group who answered correctly – proportion in low group who answered correctly D = NC h - NC l (.5)T NC h = number of people in high group answering correctly NC l = number of people in low group answering correctly T = total number of people in high and low groups

RELATIONSHIP BETWEEN ITEM DISCRIMINATION AND DIFFICULTY 0% % 100% 0 Difficulty Level Discrimination Level Perfect Discrimination When: ½ gets item right, ½ gets item wrong & ½ right is in upper half, ½ wrong is in lower half

ATTITUDE TESTS Assess an Individual’s Feelings About a Topic ItemAgreeNo Strong Feeling Disagree The day before Thanksgiving should be a holiday. ____ ____ ____ Final exams should be elective.____ The dining room should serve gourmet food. ____ ____ ____ My parents don’t appreciate how smart I am. ____ ____ ____ My professors don’t appreciate how smart I am, either. ____ ____ ____

THURSTONE SCALES Method of Equal Appearing Intervals Statements indicating an attitude toward a topic are written Judges rank the statements from least favorable to most favorable Statements receiving consistent ratings are given the average score A set of statements covering the entire range of attitudes is selected

THURSTONE SCALES Administration Respondents check items with which they agree –Well-formed attitudes indicated by consistently checking either high or low items –Poorly-formed or inconsistent attitudes indicated by inconsistent pattern or by checking off many neutral items

LIKERT SCALES Statements indicating an attitude toward a topic are written Items with clearly positive or negative attitudes are selected Statements are listed with a space for respondent to indicate degree of agreement

ItemRating Government has no business funding child care programs. SDDUASA Child care should be supported by federal, state, and local tax dollars. SDDUASA A LIKERT SCALE Directions: Indicate to what extent you agree or disagree with the statements listed below by circling one of the following: SA means that you strongly agree with the statement A means that you agree with the statement U means that you are undecided about the statement D means that you disagree with the statement SD means that you strongly disagree with the statement

SCORING LIKERT RESPONSES Method of Summated Ratings Items are weighted Weights of unfavorable items are reversed Average score is computed ItemRating Government has no business funding child care programs. SDDUASA Child care should be supported by federal, state, and local tax dollars. SDDUASA

PERSONALITY TESTS Projective tests –Present respondent with ambiguous stimulus Structured tests –Questions are objective

OBSERVATIONAL TECHNIQUES Researcher observes and records Does not interfere with behavior

TECHNIQUES FOR RECORDING BEHAVIOR Duration—how long the behavior lasts Frequency—how frequently the behavior occurs Interval (time sampling)—target behaviors of each subject during set period of time Continuous—all behaviors of the subject

TECHNIQUES FOR RECORDING BEHAVIOR TechniqueHow it WorksExample Duration recordingThe researcher records the length of time that a behavior occurs. How much time is spent in verbal interaction between two children? Frequency recordingThe researcher records the number of times a behavior occurs. How often are questions asked? Interval recordingThe researcher observes a subject for a fixed amount of time. Within a 60-second period, how many times do members of the group talk to another person? Continuous recordingThe researcher records everything that happens. During a 1-hour period, all the behavior of a 6-year-old boy is recorded.

OBSERVATIONAL TECHNIQUES? BE CAREFUL! Pitfalls to Avoid Observer effects Observer bias Fatigue Changing definitions

QUESTIONNAIRES What they are –Paper and pencil tests with structured questions –Self-administered

QUESTIONNAIRES Advantages –Can be mailed out –Survey broad geographic area –Cheaper than one-on-one interview –Respondents may be more honest –Data easy to share with other researchers Disadvantage –Low return rate

BASIC ASSUMPTIONS OF A QUESTIONNAIRE Does not make unreasonable demands upon the respondent Does not have a hidden purpose Requests information that respondents presumably have Contains interesting questions Does not request information that could be obtained by other means

THE QUESTIONS The questionnaire contains questions that can be answered The questionnaire contains questions that are straightforward

THE FORMAT The presentation is attractive, professional, and easy-to-understand Questions and pages are clearly numbered Directions are clear and explicit Questions are objective Questions are ordered sensibly Transitions are used from one topic to the next

THE IMPORTANCE OF USING A COVER LETTER Informs the recipient about the research Establishes the importance of the research Makes the recipient a part of the research