New Teacher Institute Day 2 August 8, 2017

Writing Test Questions Effective Rubric Design Preliminary Slides evaluating students Classroom Assessment Writing Test Questions Effective Rubric Design Slide 2 This is the opening claim from the CASL book, page 3. © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

why do we assessment Motivate the unmotivated Preliminary Slides why do we assessment Motivate the unmotivated Restore students’ desire to learn Encourage students to keep learning Create—not simply measure—increased achievement Slide 2 This is the opening claim from the CASL book, page 3. © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

Classroom Assessment Guiding Principles Preliminary Slides Classroom Assessment Guiding Principles Gather accurate information about student achievement Use assessment process and results to promote maximum student learning Slide 3 This slide leads into the content of the presentation. © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

Think of a time you were assessed and it was a negative experience. What made it negative? Slide 5

Now think of a time you were assessed and it was a positive experience. What made it positive? Slide 6

ACCURACY EFFECTIVE USE PURPOSE 1. Keys to Quality “Keys to Quality” Slide 7 “Key 1 is Clear Purpose. Sound classroom assessment requires that both processes and results serve clear and appropriate purposes. A central question for this key is, Who will use the assessment results and how will they be used?” © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

clear assessment purpose 1. Keys to Quality clear assessment purpose Always begin by asking… Who’s going to use the information? How will they use it? What information will they need? Do our assessment practices meet students’ information needs? “Keys to Quality” Slide 8 “To establish a clear assessment purpose, we ask the following questions: What decisions will this assessment inform? Who is making these decisions? What information will be most useful in making those decisions? And, have we included student information needs in planning our assessment? The answers to these questions determine what information is needed and in what form—this is the first step in assessment planning, whether you are developing your own assessment or selecting one already developed.” © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

two assessment purposes 1. Keys to Quality two assessment purposes SUMMATIVE Assessments OF Learning How much have students learned as of a particular point in time? FORMATIVE Assessments FOR Learning How can we use assessment information to help students learn more? “Keys to Quality” Slide 9 “We can divide the purposes of assessment into two categories: assessment of learning and assessment for learning. Summative assessment, or assessment of learning, measures the level of achievement at a point in time. Standardized tests and benchmark assessments generally fall into this category. Any work that is evaluated that counts toward the report card grade we can consider an assessment of learning. So, if you think about all the assessments given over a trimester or quarter and how many of them are figured into the grade, you’ll discover that a lot of them, if not most of them, are assessments of learning. Assessment of learning is periodic and is for accountability (for both students and teachers). Formative assessment, or assessment for learning, on the other hand, is not an accountability tool, but a teaching tool. We can conduct assessments to make decisions about instruction before the learning process or during the learning process. For example, we conduct pretests to help us decide where to begin with certain groups of students. We can also set assignments and quizzes up so that students are able to determine what they’ve mastered and what they still need to learn.” Assessment for learning is continuous. It can be viewed as “teaching with the eyes open”. Teachers watch as students practice and intervene if they are practicing things the wrong way. © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

ACCURACY EFFECTIVE USE PURPOSE TARGET 1. Keys to Quality “Keys to Quality” Slide 11 “The second key to quality is Clear Targets. Good assessments reflect clear and valued student learning targets.” © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

clear learning targets 1. Keys to Quality clear learning targets Are our targets clear to us? Are they clear to students? Can we identify what kinds of targets we have? Do our assignments and assessments reflect the targets students have had the opportunity to learn? “Keys to Quality” Slide 12 “Key 2 asks us to be clear about the targets we intend to teach and assess before we plan our assessments. We need to know what kind of target the content standard represents, so we teach to the intended level of cognitive challenge, and so we can select the appropriate assessment method.” “We need to be competent, confident masters of the targets ourselves. We need to know what targets each assessment measures so we can ensure that our tests match our teaching and so that we can keep track of learning standard by standard. And we need to make our targets clear to students in advance of, or during, the learning.” Targets must be communicated in student friendly language, such as “I can multiply two digit numbers”. © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

ACCURACY EFFECTIVE USE PURPOSE DESIGN TARGET 1. Keys to Quality “Keys to Quality” Slide 13 “So, good assessment begins with Keys 1 and 2, Clear Purpose and Clear Targets. Once those prerequisite decisions are in place, we are ready to take action in Key 3, Sound Assessment Design.” © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

sound assessment design 1. Keys to Quality sound assessment design Which assessment methods are the best match for the learning targets? What is an appropriate sample size? Are the items, tasks, and scoring rubrics of high quality? Does the assessment control for bias? Will the assessment yield information that students can use, when appropriate? “Keys to Quality” Slide 14 “High-quality assessments are not built first and then retro-fitted into a context. After we have established the context—identified the purpose and targets to be assessed—then we are ready to design the assessment with these questions in mind.” (Read slide. You may also want to add a sentence or two of explanation for each design feature.) After Question 1: Different methods include selected response, extended written response, performance assessment, and personal communication. After question 3: Do the items/tasks address the learning targets from teacher instruction? Do the rubrics specifically address the intent of the question/task? After question 4: A source of bias is anything that may interfere with assessment results (such as a death in the family or other traumatizing event occurring in a student’s life). © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

ACCURACY EFFECTIVE USE PURPOSE EFFECTIVE COMMUNICATION DESIGN TARGET 1. Keys to Quality ACCURACY EFFECTIVE USE PURPOSE EFFECTIVE COMMUNICATION DESIGN TARGET “Keys to Quality” Slide 15 “Once we have ensured the accuracy of our assessments, we are ready to address their effective use. The fourth key to quality is Effective Communication: Assessment results are managed well and communicated effectively.” © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

effective communication 1. Keys to Quality effective communication Formative Purpose Do formative assessment results function as effective feedback to students and teachers? Are students engaged in tracking and communicating about their evolving learning? Summative Purpose Is achievement tracked by learning target and reported by standard? Do grades communicate achievement accurately? Are standardized test results used appropriately? “Keys to Quality” Slide 16 “The most accurate assessment is wasted if its results are miscommunicated, or if they are communicated to students in ways that shut learning down. In Key 4, we focus on what needs to be in place so that everyone who receives assessment information can use it effectively. This includes making sure that formative assessment results can function as effective feedback, and involving students in tracking and communicating about their progress. For summative assessment, it means that our record keeping and grading practices accurately reflect student learning and that we are able to interpret and use standardized test results correctly.” © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

ACCURACY EFFECTIVE USE PURPOSE EFFECTIVE COMMUNICATION DESIGN STUDENT 1. Keys to Quality ACCURACY EFFECTIVE USE PURPOSE EFFECTIVE COMMUNICATION DESIGN STUDENT INVOLVEMENT TARGET “Keys to Quality” Slide 17 “The fifth key to quality is Student Involvement and it is not a stand-alone key, but rather a part of all of the other keys.” © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

student involvement Are students’ information needs planned for? 1. Keys to Quality student involvement Are students’ information needs planned for? Do students have a clear vision of the intended learning? Are students able to self-assess and set goals on the basis of their assessment results? Do students receive and offer effective feedback during the learning? Do students track, reflect on, and share their learning progress? “Keys to Quality” Slide 16 “The most accurate assessment is wasted if its results are miscommunicated, or if they are communicated to students in ways that shut learning down. In Key 4, we focus on what needs to be in place so that everyone who receives assessment information can use it effectively. This includes making sure that formative assessment results can function as effective feedback, and involving students in tracking and communicating about their progress. For summative assessment, it means that our record keeping and grading practices accurately reflect student learning and that we are able to interpret and use standardized test results correctly.” © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

The ultimate user of assessment information is… the student. 2. Formative Assessment Research The ultimate user of assessment information is… the student. “Formative Assessment Research” Slide 27 You could ask “Which of our current assessments are set up so students can act on the information?” Or, you could use this slide as a probe with a partner discussion question such as: “How does this match your current thinking about formative assessment?” This slide concludes the introductory part that focuses on the connection between assessment and motivation and introduces assessment for learning in a general sense. © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

constructing exam questions

objectives Use Bloom’s Taxonomy to identify the knowledge level of assessment items Compose an effective and meaningful stem for Multiple Choice items Construct effective and plausible distractors for Multiple Choice items Construct effective assessment questions in the form of: Multiple Choice, True/False and Essay/Short Answer

planning the assessment

constructing the stem The stem should be meaningful and present a problem Focus on a learning outcome Lack of a definite problem in the stem often assesses student’s inference as opposed to a direct measurement of performance

constructing the stem

constructing the stem Try to avoid including irrelevant material in the stem as it can decrease the reliability and validity of the assessment item

constructing the stem Consider making the stem a question or partial sentence A question stem allows students to focus on answering the assessment item, increasing validity Avoid using inserted blanks as it increases cognitive load- can decrease validity of assessment item

constructing the distractors All distractors should be plausible Serve to effectively indicate what students know Incorrect distractors should be chosen by students who did not achieve the learning outcome Incorrect distractors should be ignored by the students who did

constructing the distractors Distractors should be clear and concise Excessive wordiness assesses reading/comprehension as opposed to the actual desired outcome

constructing the distractors Avoid using “all of the above” or “none of the above” Students who can identify more than one correct distractor can choose correct answer without achieving the learning outcome Conversely, students who can identify on incorrect distractor can choose correct answer without achieving the learning outcome Distractors can contain inadvertent clues in the language, length, format, and grammar Keep grammar consistent with the stem Keep length similar Keep format similar Keep language similar

constructing true/false questions Typically used at the knowledge level of assessment to measure recall of factual knowledge Potential to measure higher level knowledge Strengths of True/False Questions Easy to write and answered quickly by students Provide a large sampling of content vs time allotted Weaknesses of True/False Questions Problem with guessing- 50/50 chance of guessing correctly without achieving learning outcome Ambiguity due to writing statements that are indisputably true or false

constructing true/false questions Guidelines to Constructing True/False questions: Assess significant material Write questions that are indisputably true or false Avoid taking statements directly from textbooks Assess only one point in each item Avoid trick questions Avoid terms such as : always, all, never…instead use: usually, often, many times Avoid negatively worded statements Randomize the order to avoid giving clues Keep true and false statements/questions similarly formatted

constructing essay/short answer questions Essay/Short Answer Items Consider class-size Consider time available to prepare the assessment Strengths of Essay/Short Answer Items Effective way to measure higher level thinking Require less time to construct Discourages students from memorizing and regurgitating facts Can provide students with realistic tasks/scenarios Requires student s to organize, construct, and communicate their thoughts Weaknesses of True/False Items Provide a smaller sampling of content due to time needed to answer each question Require more time for grading: reading and scoring Scoring is often more subjective than objective which can decrease reliability

constructing essay/short answer questions Guidelines to Constructing Essay/Short Answer questions: Create clearly defined questions…lead the student towards the approach you desire To cover more content, consider more questions requiring shorter answers Avoid giving optional essay questions- students choosing different essay questions to answer are essentially taking different exams Indicate how much each essay/short answer question is worth Avoid questions that only require providing factual information

constructing essay/short answer questions Examples of essay/short answer items by level of Bloom’s Taxonomy

effective rubric design Experts agree: Rubrics are hard to design. Rubrics are time-consuming to design. “A rubric is only as useful as it is good. Using a bad rubric is a waste of time…” --Michael Simkins in “Designing Great Rubrics” Experts disagree: how to design a “good” rubric Bottom line: Is it working for you and for your students? Checklists do not reflect developmental—indicates only presence or lack of a trait

let’s make a cookie Task: Make a chocolate chip cookie that I would want to eat. Criteria: Texture, Taste, Number of Chocolate Chips, Richness Range of performance: Delicious(14-16 pts) Tasty(11-13 pts) Edible(8-10 pts) Not yet edible(0-7 pts)

the rubric Delicious 4 Tasty 3 Edible 2 Not yet edible 1 # chips Chips in every bite 75% chips 50% chips Less than 50% chips texture Consistentlychewy Chewy middle, crispy edges Crunchy Like a dog biscuit color Even golden brown Brown with pale center All brown Or all pale Burned richness Buttery, high fat Medium fat Low-fat flavor Nonfat flavor

assess the cookie Overall score By criteria Delicious Tasty Edible Not yet edible By criteria Number of chips Texture Taste Richness

Steps in Developing a Rubric Design backwards—rubric first; then product/performance. Decide on the criteria for the product or performance to be assessed. Write a definition or make a list of concrete descriptors—identifiable-- for each criterion. Develop a continuum for describing the range of performance for each criterion. Keep track of strengths and weaknesses of rubric as you use it to assess student work. Revise accordingly. Step back; ask yourself, “What didn’t I make clear instructionally?” The weakness may not be the rubric. See “cookie”

Modifying a “Canned” Rubric Find a rubric that most closely matches your performance task. Evaluate and adjust to reflect your instruction, language, expectations, content, students Criteria Descriptors Performance levels Eliminate criteria that do not reflect your assignment.

It’s hard work… Expect to revise…and revise… One problem is that the rubric must cover all potential performances; each should fit somewhere on the rubric. “There are no final versions, only drafts and deadlines.” When you’ve got a good one, SHARE IT!

Writing Test Questions Effective Rubric Design Preliminary Slides evaluating students Classroom Assessment Writing Test Questions Effective Rubric Design Slide 2 This is the opening claim from the CASL book, page 3. © 2010 Pearson Assessment Training Institute, Portland, OR www.assessmentinst.com

Go forth and assess for success

Contact info Art Rousseau Director of Career and Technical Education arthur.rousseau@accs.edu 334-782-2509 Barbara Anne Spears Executive Director of Academic & Student Affairs barbaraanne.spears@accs.edu 334-730.8156