EDU 385 Classroom Assessment Session 6 Preparing and Using Achievement Tests

Bell Work

Content Objectives Understand how to prepare and use an achievement test Understand when to use an achievement test

Language Objectives List steps to prepare an Achievement Test andPrepare a set of Specifications for a Test Describe merits of Selection-Type and Supply-Type Tests Match test items to specific Learning Outcomes they measure Describe factors to consider in preparing items for test Write Clear Directions for a test Review and Evaluate an assembled test Administer a test properly Evaluate the effectiveness of a test or test items

Preparing Achievement Tests Specify the Instructional Objectives Prepare Test Specification Construct relevant Test Items Arrange the items in the Test Prepare Clear Directions Review and Evaluate the Assembled Test Administer the Test and Make an Item Analysis

Specify Instructional Objectives At End of Unit the Student will: Remember the meaning of Common Terms Remember Specific Facts Remember Basic Procedures Understand Relevant Principles Apply the Principles

Define General Outcomes in Specific Terms pe Next Step is to list Specific Types of Student Performance Accepted as Evidence that Outcomes have been Achieved Specific Observable Performance Terms (Action Verbs) Identifies, Distinguishes between Selects, Describes, Matches Explains, Predicts, Formulates

Vague and Indefinite Terms Avoid Using Terms As: Learns Sees Realizes

Taxonomy Categories Sample Verbs for Stating Specific Learning Outcomes Remember Identifies, names, defines, describes, lists, matches, selects, outlines Understand Classifies, explains, summarizes, converts, predicts, interprets, illustrates Apply Demonstrates, computes, solves, modifies, arranges, operates, relates, uses Analyze Differentiates, diagrams, estimates, separates, orders, subdivides, distinguishes between Evaluate judges, criticizes, compares, justifies, concludes, discriminates, detects Create Combines, creates, formulates, designs, composes, constructs, rearranges, revises Illustrative Action Verbs for Defining Objectives in the Cognitive Domain of Blooms Revised Taxonomy

Preparing Test Specifications The writing of Test Items Should be Guided by set of Test Specifications Describe the Achievement Domain to be Measured Link the Achievement Domain to Course Subject Matter Provide Guidelines for Obtaining Representative Sample of Test Tasks

Table of Specifications Select the Learning Outcome to be Tested Outline the Subject Matter Make a Two-Way Chart (describes the sample of items to be included in the test)

Select Learning Outcomes From Following Items Recall of Knowledge Intellectual Abilities and Skills General Skills (laboratory, performance, communication, work-study) Attitudes, Interests, and Appreciations

Outlining Subject Matter Content of a Course may be outlined in detail for teaching purposes For Test Planning only the major categories need be listed

Making 2-Way Chart or Table of Specifications Purpose helps provide assurance that the test will measure a representative sample of learning outcomes It relates outcomes to content Indicates relative weight given to various areas

ContentTermsFactsProcedures Understands Principles Applies Principles Total # of Items Role of Tests in Instruction Principles of Testing Norm Referenced vs. Criterion Referenced Planning the Test Total Number of Items Remembers Table of Specifications for a Summative Test on a Unit for Achievement Testing (example)

Factors to Consider on Weighting Test Items Importance of area in learning outcome Importance in total learning experience Which outcomes have greater retention and transfer value What importance does CORE curriculum place on each area Should reflect the importance placed during instruction

Constructing Test Items Selection-Type Items Multiple Choice True-False Matching Interpretive Exercise Supply-Type Items Short Answer Essay (restricted Response) Essay (Extended Response)

Which to Use Guiding Principle Use the item types that provide the most direct measures of student performance specified by the intended learning outcomes

Measures Factual InformationYes Yes(*) Measures UnderstandingYesNo(**)Yes Measures SynthesisNo(**) Yes Easy to ConstructNoYes Samples BroadlyYes No Eliminates BluffingYesNo Eliminates Writing SkillYesNo Eliminates Blind GuessingNoYes Easy to ScoreYesNo Scoring is ObjectiveYesNo Pinpoints learning ErrorsYes No Encourages OriginalityNo Yes Characteristic Selection-Type ItemsShort AnswerEssay Supply-Type Items (*) The essay Test can measure knowledge of facts, but because of scoring and sampling problems it probably should not be used for this purpose (**) These items can be designed to measure limited aspects of these characteristics

Matching Items (examples) Specific Learning Outcome: Defines terms in a student’s own words Directions: Define each of the following terms in a sentence or two Taxonomy Cognitive Measurement Evaluation

Matching Items (examples) S pecific Learning Outcome: Identifies procedural steps in planning for a test 1. Which one of the following steps should be completed first in planning for an achievement test? A. Select the types of test items to use B. Decide on the length of the test *C. Define the intended learning outcomes D. Prepare the test specifications

Matching Items (examples) Specific Learning Outcome: Distinguish between sound and unsound principles in achievement testing Directions: Read each of the following statements. If the statement indicates a sound principle of achievement testing, circle the S; if it indicates and unsound principles circle the U. *S U 1. The specific learning outcome to be tested should be stated in terms of student performance S U 2. Achievement testing should be limited to outcomes that can be measured objectively S U 3. Each achievement test item should measure a clearly defined type of student performance

Cautions in Phrasing Test Items Vocabulary that is unnecessarily difficult Sentence structure that is unnecessarily complex Statements containing ambiguity Unclear pictorial materials Directions that are vague Material reflecting race, ethnic, or sex bias

Common Clues to Avoid Verbal associations that give away the answer Grammatical inconsistencies that eliminate wrong answers Specific determiners that make certain answers probable (e.g., sometimes) and others improbable (e.g. always) Stereotyped or textbook phrasing of the correct answer Length and location of correct answers Material in an item that aids in answering another item

Deciding Number of Test Items Age of Students Tested Time Available for Testing Type of Test Items Used Type of Interpretation to be Made

Checklist for Evaluating the Test Plan 1. Is the purpose of the test clear? 2. Have the intended learning outcomes been identified & defined? 3. Are the intended learning outcomes stated in performance (measurable terms? 4. Have test specifications been prepared that indicate the nature and distribution of items to be included in the test: 5. Does the specified set of items provide a representative smaple of the tasks contained in the achievement domain? 6. Are the types of items appropriate for the learning outcomes to be measured? 7. Is the difficulty of the items appropriate for the students to be tested and the nature of the measurement (e.g., mastery or survey) 8. Is the number of items appropriate for the students to be tested, the time available for testing and the interpretations to be made? 9. Does the test plan include built-in features that contribute to valid and reliable scores? 10. Have plans been made for arranging the items in the test, writing directions, scoring, and using the results?

Guidelines for Writing Items Select type of test item that measures intended outcome most directly Write test item so the performance it elicits matches the performance of the learning task Write test item so test take is clear and definite Write test item so it is free from nonfunctional material Write test item so irrelevant factors do not prevent an informed student from performing correctly

Guidelines for Writing Items Write test question so irrelevant clues do not enable the uniformed student to respond correctly Write test item so difficulty level matches the intent of the learning outcome, age group tested, & use made of results Write test item so there is no disagreement concerning the answer Write the test items in advance so they can be reviewed and modified Write more test items than called for in test plan

Arranging Items in Test Guidelines For instructional purpose, it is usually desirable to group together items that measure the same outcomes Where possible, all items of the same type should be grouped together Items should be arranged in terms of increasing difficulty

Preparing Directions Directions should be simple and concise Yet contain information concerning each of following: Purpose of Test Time allowed to complete the test How to record the answers Whether to guess when in doubt about an answer

Checklist for Evaluating the Assembled Test 1. Balance Do the items measure a representative sample of the learning tasks in the achievement domain? 2. Relevance Do the test items present relevant tasks? 3. ConcisenessAre the test tasks stated in simple, clear language? 4. Soundness Are the items of proper difficulty, free from defects, and do they have answers that are defensible 5. independence Are the items free from overlapping, so that one item does not aid in answering another? 6. Arrangement Are items measuring the same outcome grouped together? Are items of the same type grouped together? Are items in order of increased difficulty? 7. Numbering Are the items numbered in order throughout the test? 8. Directions Are there directions for the whole test and each part? Are the directions concise and at proper reading level? Do directions include time limts and how to record answers? Do directions tell what to do about guessing? 9. Spacing Does the spacing on the page contribute to ease of reading and responding? 10. Typing Is the final copy free from typographical errors?

Administering Tests Provide space with desks far enough apart to prevent cheating Keep interruptions at a minimum May want to give oral directions in addition to written Make sure all students know exactly what to do and provide them with most favorable conditions possible

Scoring Test Scoring is facilitated by having answers on left side of each test page If using a separate answer sheet may want to use a scoring stencil (punch out letters of correct answers and lay it over the answer sheet)

Item Analysis Provide students a copy of the test and review each question with them, allow discussion may find test items which can be improved plus students will learn from review After tests are scored tally student responses on a master copy (number right and number wrong for each question) Analyze top 1/3 and bottom 1/3 (Did students miss same items or were they random. Review will help next time you teach the course

Summary Preparing and using achievement test includes specifying the instructional objectives in performance terms, preparing table of specifications, constructing relevant test items, arranging items in test, preparing clear directions, reviewing and evaluating the assembled test, administering the test, and making an item analysis Test specifications typically consist of twofold table of specifications that indicates the sample of performance tasks to be measured Type of test items used in a test are determined by how directly they measure intended learning outcomes and how effective as measuring instruments

Summary Each item should provide a task that matches student performance described in specific learning outcomes Functioning content of test items can be improved by eliminating irrelevant barriers and unintended clues Difficulty of test item should match the difficulty of the learning task to be measured. Beware of irrelevant sources of difficulty (e.g., obscure material) Achievement test should be short enough to permit all students to attempt all items during time available

Summary Test should contain sufficient number of test items for each type of interpretation to be made. (Interpretation of fewer that 10 should be considered highly tentative) Following general set of guidelines during item writing will result in higher quality items that contribute to validity and reliability of test results Item arrangement within test varies with type of test used. When possible items should be grouped by major learning outcomes (e.g., knowledge, understanding, application) similar items should be grouped together and arranged in increasing difficulty Test directions should indicate purpose of test, time allowed, how to record answers, and answer all items

Summary A correction for guessing should be used for speed tests only. Review and evaluate the assembled test before using Administer the test under controlled conditions Item analysis provides a means of evaluating the items and is useful in reviewing the test with students

Now Go Forth and Do Good Things