District Determined Measures MSSAA District Determined Measures October 29, 2015 Dr. Deborah Brady dbrady3702@msn.com

Do Now http://ddmsbrady.wikispaces.com/MSSAA or Getting Online http://ddmsbrady.wikispaces.com/MSSAA or On-line access to materials now at: https://wikispaces.com/join/HQ4FDRM Code HQ4FDRM

Agenda II. Scoring DDMs I. Collecting DDMs, Assuring High Quality Coversheet and Checklist Examples The good the bad, the ugly “Mock” evaluation of sample DDMs II. Scoring DDMs Group Work: Calibration Protocols—Calibrating with Rubrics Group Work: Excel Calculation of pre-post, rubrics, and MCAS SGP Lunch III. Time to work your district’s plan for Communication Designation and Documentation of DDMs Assessment of DDMs Analysis of pre and post test Calculating Individual Teacher Growth Scores 1:00 Optional: Union Negotiations (30 minutes—table talk) 1:30 Optional: Indirect Measures—Administrator, Guidance Counselor, Nurse, School Psychologists, for example 2:00 Optional: Your choice

Why Flunking Exams is Actually a Good Thing “That is: The (bombed) pretest drives home the information in a way that studying as usual does not. We fail, but we fail forward.” The excitement around pre-finals is rooted in the fact that the tests appear to improve subsequent performance in topics that are not already familiar, whether geography, sociology or psychology. At least they do so in experiments in controlled laboratory conditions. A just-completed study — the first of its kind, carried out by the U.C.L.A. psychologist Elizabeth Ligon Bjork — found that in a live classroom of Bjork’s own students, pretesting raised performance on final-exam questions by an average of 10 percent compared with a control group. Full article is on wiki "Why flunking exams is actually a good thing" from NY Times. http://www.nytimes.com/2014/09/07/magazine/why-flunking-exams-is-actually-a-good-thing.html?module=Search&mabReward=relbias:r,{1:RI:7}&_r=1

I. Collecting, Organizing, Determining the Quality of Local DDMS Cover Sheet (2 DDMs per Educator) Checklist (Rigor, Validity, Substantive, Types of Questions, etc.) Examples (Good, Bad, and Mediocre) Mock Assessments as Teams

Coversheet for Documenting 2 DDMs per Educator Last Name First Name Grade/Dept. DDM1 DDM2 DDM3 (optional) Jones Brigit ELA 6 MCAS 6 Growth Score ELA ELA 6 DDM (writing to text)   Smith Marion 9-12 library Library Search Tools DDM Indirect: Increase teachers who do research in library. Watson Elspeth 5 ELA team MCAS SGP (Student Growth Percentile) Galileo Math Growth Score History Unit Exam DDM Holmes Sharon 2 Fountas and Pinnell DDM Galileo DDM Campbell John Guidance Indirect: Increase PSAT participation Direct: post-test after College Application workshops

Checklist for Collecting and Checking Quality Areas Check all Items that are completed Source of DDM Locally developed Standardized test Course DDM “Title” Possible educators who will use this DDM Check one Direct Measure Indirect Measure (Describe) Content Area Alignment to State and/or District Standards Rigor: Check the levels of Blooms that are assessed Check all that apply Type(s) of questions Duration of assessment When assessment(s) will take place Components of assessment that are completed so far. NOTE: Indirect measures do require a clear explanation of what you will measure, when, and what counts. They also require directions to others involved in this same DDM. Rubric included Not Yet Does not apply

“Sample Plan” Part I for Core Areas ELA Math Science Social Studies 12 CA/CA 11 10 9 8 MCAS SGP/CA 7 6 5 4 3 2 1 K CA

Sample Plan Part II with Specialists, Singletons Singleton, Art, Music, Technology, PE by Grade or grade-spans Special Education Specialists, Co-Teachers, substantially separate Indirect Measures (IM) Central Office, Psychologist Administrators Principals, Assitant Principals, Superintendent, 12 CA/CA Inclusion Co-Teachers Can “share” scores with General Ed Teachers Or If their students’ goals are substantially different, the assessments can be modified or can focus on the goal of inclusion Specialists Measure Goals Can develop K-12 rubric Speech PE Adaptation of the SMART goal process Measure goal that has an impact on student growth Attendance College Applications Technology Growth MCAS Growth Either ELA or Math PLUS IM 11 10 9 8 7 6 5 4 3 2 1 K

Consistency in Directions for Administrating Assessments Directions to teachers need to define rules for giving support, dictionary use, etc. What can be done? What cannot? “Are you sure you are finished?” How much time? Accommodations and modifications?

Examples: The Good, the Bad, the Ugly Scores:. Thumbs up—All is good Examples: The Good, the Bad, the Ugly Scores: Thumbs up—All is good Thumbs horizontal—Some questions Thumbs Down—Needs significant work

Quick Reminder: Assessment Quality Requirements and Definitions from DESE (See Checklist.) Alignment to Frameworks and District Curriculum content and/or district standards Rigor Comparability across all classes and in all disciplines “Substantial” assessment of the course; core content and skills Modifications are allowed as with MCAS

Learning Skills Criteria (Special Education) Individual Goals; measured weekly; permanent folder Notes Planner Work/Action Plan Flexible when Necessary Prepared for Class (materials, work) Revises work Follows instructions Uses time well Gets to work Asks for help when needed Advocates for self Moving toward independence Works collaboratively

Essay Prompt from Text Read a primary source about Mohammed based on Muhammad’s Wife’s memories of her husband. Essay: Identify and describe Mohammed’s most admirable quality based on this excerpt. Then, select someone from your life who has this quality. Identify who they are and describe how they demonstrate this trait. What’s wrong with this prompt about a primary source and is a text-based question?

Scoring Guides from Text A scoring guide from a textbook for building a Lou Vee Air Car. Is it good enough to ensure inter-rater reliability? Lou Vee Air Car built to specs (50 points) Propeller Spins Freely (60 points) Distance car travels 1m 70 2m 80 3m 90 4m 100 Best distance (10,8,5) Best car(10,8,5) Best all time distance all classes (+5) 235 points total

Grade 2 for overhand throw and catching. PE Rubric in Progress. Grade 2 for overhand throw and catching.

Music: Teacher and Student Instructions

Beware Rubrics! Holistic Rubric Show Progress across a Scale, Continuum, Descriptors 1 2 3 4 Details No improvement in the level of detail. One is true * No new details across versions * New details are added, but not included in future versions. * A few new details are added that are not relevant, accurate or meaningful Modest improvement in the level of detail * There are a few details included across all versions * There are many added details are included, but they are not included consistently, or none are improved or elaborated upon. * There are many added details, but several are not relevant, accurate or meaningful Considerable Improvement in the level of detail All are true * There are many examples of added details across all versions, * At least one example of a detail that is improved or elaborated in future versions *Details are consistently included in future versions *The added details reflect relevant and meaningful additions Outstanding Improvement in the level of detail * On average there are multiple details added across every version * There are multiple examples of details that build and elaborate on previous versions * The added details reflect the most relevant and meaningful additions Example taken from Austin, a first grader from Answer Charter School in Boise, Idaho.  Used with permission from Expeditionary Learning. Learn more about this and other examples at http://elschools.org/student-work/butterfly-drafts

Preconventional Emerging Developing Beginning Expanding Bridging Ages 3-5 Emerging Ages 4-6 Developing Ages 5-7 Beginning Ages 6-8 Expanding Ages 7-9 Relies primarily on pictures to convey meaning. Begins to label and add “words” to pictures. Writes first name.   Uses pictures and print to convey meaning. Writes words to describe or support pictures. Copies signs, labels, names, and words (environmental print). Writes 1-2 sentences about a topic. Writes names and familiar words. . Writes several sentences about a topic. Writes about observations and experiences. Writes short nonfiction pieces (simple facts about a topic) with guidance. Writes short fiction and poetry with guidance. Writes a variety of short nonfiction pieces (e.g., facts about a topic, letters, lists) with guidance. Writes with a central idea. Writes using complete sentences. Bridging Ages 8-10 Fluent Ages 9-11 Proficient Ages 10-13 Connecting Ages 11-14 Independent Writes about feelings and opinions. Writes fiction with clear beginning, middle, and end. Writes poetry using carefully chosen language with guidance. Writes organized nonfiction pieces (e.g., reports, letters, and lists) with guidance. Begins to use paragraphs to organize ideas. Uses strong verbs, interesting language, and dialogue with guidance. Begins to write organized fiction and nonfiction (e.g., reports, letters, biographies, and autobiographies). Develops stories with plots that include problems and solutions with guidance. Creates characters in stories with guidance. Writes poetry using carefully chosen language. Begins to experiment with sentence length and complex sentence structure. Varies leads and endings with guidance. Uses description, details, and similes with guidance. 1 Uses dialogue with guidance. Writes persuasively about ideas, feelings, and opinions. Creates plots with problems and solutions. Begins to develop the main characters and describe detailed settings. Begins to write organized and fluent nonfiction, including simple bibliographies. Writes cohesive paragraphs including reasons and examples with guidance. Uses transitional sentences to connect paragraphs. Varies sentence structure, leads, and endings. Begins to use descriptive language, details, and similes. Uses voice to evoke emotional response from readers. Begins to integrate information on a topic from a variety of sources. Writes in a variety of genres and forms for different audiences and purposes independently. Creates plots with a climax. Creates detailed, believable settings and characters in stories. Writes organized, fluent, and detailed nonfiction independently, including bibliographies with correct format. Writes cohesive paragraphs including supportive reasons and examples. Uses descriptive language, details, similes, and imagery to enhance ideas independently. Begins to use dialogue to enhance character development. Incorporates personal voice in writing with increasing frequency. Writes organized, fluent, accurate, and in-depth nonfiction, including references with correct bibliographic format. Writes cohesive, fluent, and effective poetry and fiction. Uses a clear sequence of paragraphs with effective transitions. Begins to incorporate literary devices (e.g., imagery, metaphors, personification, and foreshadowing). Weaves dialogue effectively into stories. Develops plots, characters, setting, and mood (literary elements) effectively. Begins to develop personal voice and style of writing. ?

x x x x Criterion Referenced Rubric and Raw Scores or % of 100 4(25)= 4(22)= 88 x 4(18)=72 x x 4(15)= 60 + 18 + 22 + 15 = 80%

AP Rubric of Rubrics Prose Analysis (9 levels give students room to improve Holistic) 9-8 Answers all parts of the question completely. Using specific evidence from the work and showing how that evidence is relevant to the point being made. Fashions a convincing thesis and guides reader through the intricacies of argument with sophisticated transitions. Demonstrates clear understanding of the work and recognizes complexities of attitude/tone. Demonstrates stylistic maturity by an effective command of sentence structure, diction, and organization. Need not be without flaws, but must reveal an ability to choose from and control a wide range of the elements of effective writing. 7-6 Also accurately answers all parts of the question, but does so less fully or effectively than essays in the top range. Fashions a sound thesis. Discussion will be less thorough and less specific, not so responsive to the rich suggestiveness of the passage or precise in discussing its impact. Well written in an appropriate style, but with less maturity than the top papers. Some lapses in diction or syntax may appear, but demonstrates sufficient control over the elements of composition to present the writer’s ideas clearly. Confirms the writer’s ability to read literary texts with comprehension and to write with organization and control. 5 Discusses the question, but may be simplistic or imprecise. Constructs a reasonable if reductive thesis. May attempt to discuss techniques or evidence in the passage, but may be overly general or vague. Adequately written, but may demonstrate inconsistent control over the elements of composition. Organization is attempted, but may not be fully realized or particularly effective. 4-3 Attempts to answer the question, but does so either inaccurately or without the support of specific evidence. May confuse the attitude / tone of the passage or may overlook tone shift(s) or otherwise misrepresent the passage. Discussion of illustrations / techniques / necessary parts of the prompt may be omitted or inaccurate. Writing may convey the writer’s ideas, but reveals weak control over diction, syntax, or organization. May contain many spelling or grammatical errors. Essays scored three are even less able and may not refer to illustrations / techniques at all. 2-1 Fails to respond adequately to the question. May misunderstand the question or the passage. May fail to discuss techniques / evidence used or otherwise fail to respond adequately to the question. Unacceptably brief or poorly written on several counts. Writing reveals consistent weakness in grammar or other basic elements of composition. Although may make some attempt to answer the question, response has little clarity and only slight, if any, evidence in its support. Although the writer may have made some attempt to answer the prompt, the views presented have little clarity or coherence; significant problems with reading comprehension seem evident. Essays that are especially inexact, vacuous, and /or mechanically unsound should be scored 1. 0 A blank paper or one that makes no attempt to deal with the question receives no credit. Rubric from Sharon Kingston

Rubric “Cut Scores” Create a “growth” rubric and describe a typical year’s growth Translate into 100% www.roobrix.com

Post Test Only Use District’s History to predict scores on AP Calculus Exam, for example Previous Grade’s Math Score Low Growth Moderate High A 3 4 5 B 2 C 1 D

Calibration Protocol

Considerations for Scoring Student Work Districts will need to determine fair, efficient and accurate methods for scoring students’ work. (Use consistent directions for teachers.) DDMs can be scored by the educators themselves, groups of teachers within the district, external raters, or commercial vendors. For districts concerned about the quality of scoring when educators score their own student’s work, processes such as randomly re-scoring a selection of student work to ensure proper calibration or using teams of educators to score together, can improve the quality of the results. When an educator plays a large role in scoring his/her own work, a supervisor may also choose to include the scoring process into making a determination of a Student Impact.

Mock Calibration 1. All of the readers come together and are provided student compositions for calibration. These compositions can be pre-selected by the facilitator from all of the writing submitted or can be provided by the teachers. To assure fairness in assessment, teachers will not evaluate their own students’ compositions, but will be provided with their students’ work once compositions are graded and the scores are entered for the entire class. The purpose of the calibration meeting is to make sure that all evaluators are assessing student work on the same scale. In addition, the papers used for calibration become the exemplars that all teachers will use during scoring. Rubrics alone are not sufficient for precise assessment. Encourage each scorer to make notations on these compositions. Differences are ezpected and will be discussed until it is clear that all compositions are assessed with the same standards. 2. Time is given for each scorer to read and to use the rubric provided (local, textbook, MCAS, PARCC, or 6-Trait rubrics are all appropriate) to score the first composition and to enter their score on a chart like the one illustrated below. After each scorer has entered his or her score, the facilitator discusses the reasons for the scores. Scorers may need to find examples for their scores within the compositions. 3. After there is consensus for the first paper, the scorers go on to a second and third until there is a composition that represents each level of the rubric. Generally, after the first one or two compositions are calibrated, the process goes quickly.

Mpck Calibration Ask teachers to select (or you can pre-select 2 low compositions, 2 average, and 2 high compositions (altogether) Try to select a clear range. Photocopy all of the compositions and ask teachers to evaluate the top one and using the 6-level rubric and the 4-level rubric to enter their scores under the first composition. Suggest that they can comment on their copies. Continue with the calibration until you have an exemplar for all levels. Then the discussion needs to begin to discuss the specific reasoning for each scoring Assume each person has a good reason, but the purpose is to work toward consensus. Composition # 1 2 3 4 5 6 Content 2,2,3,2,4,2 Conventions 1,1,1,1,2,1

MCAS Rubric 1 2 3 4 5 6 Content Little topic/idea development, organization, and/or details Little or no awareness of audience and/or task Limited or weak topic/idea development, organization, and/or details Limited awareness of audience and/or task Rudimentary topic/idea development and/or organization Basic supporting details Simplistic language Moderate topic/idea development and organization Adequate, relevant details Some variety in language Full topic/idea development Logical organization Strong details Appropriate use of language Rich topic/idea development Careful and/or subtle organization Effective/rich use of language Conventions Errors seriously interfere with communication AND Little control of sentence structure, grammar and usage, and mechanics Errors interfere somewhat with communication and/or Too many errors relative to the length of the essay or complexity of sentence structure, grammar and usage, and mechanics Errors do not interfere with communication and/or Few errors relative to length of essay or complexity of sentence structure, grammar and usage, and mechanics Control of sentence structure, grammar and usage, and mechanics (length and complexity of essay provide opportunity for student to show control of standard English conventions)

4th Grade Prompt http://www. doe. mass. edu/mcas/student/2014/question You are finally old enough to baby-sit, and your first job is this afternoon! You will be spending the entire afternoon with a one- year-old. When you open the door you realize that instead of watching a one-year-old child, you will be watching a one-year-old elephant! Write a story about spending your afternoon with a baby elephant. Give enough details to show readers what your afternoon is like baby-sitting the elephant

2014 MCAS Grade 4 English Language Arts Composition Topic/Idea Development - Score Point 3 This composition is rudimentary in topic development and organization. The straightforward introduction moves immediately to the surprise of discovering that the "baby" is a baby elephant. From here, though, only basic supporting details are demonstrated as this composition of five paragraphs unfolds. There is an interesting "snoring scenario" which briefly captures the babysitter's personality in his or her impatience with the elephant: "It was as if there were 100 bells surrounding the house and all ringing at the same time. I covered my ears with pillows, it didn't work. I put ear muffs on, it still didn't work. Finally I just woke him up. He was pretty upset." The job ends as the mom comes home and there is a brief exchange of mildly humorous dialogue. The conclusion is simplistic, reiterating that the experience was not enjoyable

Calculating Growth Scores MCAS and Local What you need to understand as you are creating assessments

Growth Score FAQs from DESE Do the same numbers of students have to be identified as having high, moderate, and low growth? There is no set percentage of students who need to be included in each category. Districts should set parameters for high, moderate, and low growth using a variety of approaches. How do I know what low growth looks like? Districts should be guided by the professional judgment of educators. The guiding definition of low growth is that it is less than a year’s worth of growth relative to academic peers, while high growth is more than a year’s worth of growth. If the course meets for less than a year, districts should make inferences about a year’s worth of growth based on the growth expected during the time of the course. Can I change scoring decisions when we use a DDM in the second year? It is expected that districts are building their knowledge and experience with DDMs. DDMs will undergo both small and large modifications from year to year. Changing or modifying scoring procedures is part of the continuous improvement of DDMs over time. Will parameters of growth be comparable from one district to another? Different assessments serve different purposes. While statewide SGPs will provide a consistent metric across the Commonwealth and allow for district-to-district comparisons, DDMs are selected

MCAS SGP Local Manipulation of Scores (4-8; ELA or Math; not grade 10)

288 to 244/ 25 SGP 4503699 230 to 230/ 35 SGP 214 to 225/ 92 SGP

Median student growth percentile Last name SGP Lennon 6 McCartney 12 Starr 21 Harrison 32 Jagger 34 Richards 47 Crosby 55 Stills 61 Nash 63 Young 74 Joplin 81 Hendrix 88 Jones 95 Imagine that the list of students to the left are all the students in your 6th grade class. Note that they are sorted from lowest to highest SGP. The point where 50% of students have a higher SGP and 50% have a lower SGP is the median. Median SGP for the 6th grade class Key points: A reminder on how to calculate medians: sort student SGPs from lowest to highest and find the middle student. More info: The class of students is all 1960s era rockers. “Jones” is Davy Jones.

Calculating Local Growth Scores Using Median

Sample Cut Score Determination (for local assessments) Pre-test Post test Difference Student Scores Sorted low to high Teacher score is based on the MEDIAN Score of her class for each DDM 20 35 15 5 Cut score LOW Growth Lowest ___% 25 30 50 60 median teacher score 40 70 Teacher score 65 75 80 Top 20% 85 HIGH GROWTH Highest ___?

Measuring Growth Example: Fountas and Pinnell P to Q; N to P; D to K

12.0 Median for whole Grade 3 DDM Teacher A Teacher B Teacher C Teacher D Teacher E Teacher F 5 7.5 6.5 3.5 9 5.5 7 10 8.5 1 3 12 6 12.5 11.5 None 13 16 13.5 13.6 17 16.5 15.5 19 Median 6.5 Median 9 Median 12 Median 10 Median 16 Below 6.5 Between 6.5 and 16 LOW Moderate 103 Third Graders All Classes 1 3 3.5 5.5 6 6.5 7 7.5 8.5 9 9.5 10 10.7 10.9 11.0 11.2 11.4 11.5 11.7 11.9 12.0 12.2 12.5 12.6 12.7 12.9 13.1 13.2 13.4 13.6 13.8 13.9 14.1 14.3 14.4 14.6 14.8 15.0 15.1 15.3 15.5 15.6 13 13.5 16 16.5 17 19 26 6.5 Cut Score lowest 15% 12.0 Median for whole Grade 3 DDM 16 cut score highest 15%

Fountas and Pinnell Growth for Each Student Is Based on 10 Months of Growth Second Grade Student Level Achievement Level End of Year Levels from beginning to the end of the year Pre-Post F&P Levels Growth HIGH, MODERATE, OR LOW GROWTH (10 MONTHS=YEAR) Q Above Benchmark PQ 7 MONTHS GROWTH LOW GROWTH P At Benchmark NOP 10 MONTHS OF GROWTH MODERATE GROWTH K Below Benchmark DEFGHIJK 17 MONTHS OF GROWTH HIGH GROWTH

Summary DDM Process for Determining L, M, H for Every Teacher Whole Grade Level or Course Score the entire grade level or course or take the MCAS Growth Scores for all students Rank the scores from highest to lowest (post minus pre or MCAS SGP) Identify the median score for the entire group Determine the “cut” scores for local assessments; MCAS 35 and 65 for classrooms Individual Teacher Select students for each teacher Rank the scores from highest to lowest Identify the Median score Is the median below or above the “cut” score? Is it in the middle? Don’t forget Roster Verification might change the specific scores and, therefore, change the Median Distribute scores to teachers for each DDM

Mock Scoring, Storing, Determining Cut Scores Protocol 1. Assess all students for the course or grade level. 2. Enter student names, teacher names pre-test scores, post-test scores, and the gain from pre-test to post-test. 3. Using the SORT function, rank all assessments from highest to lowest. Determine the local cut scores. DESE recommends 1.5 Standard Deviations from 50; local districts have selected 10%, 15%, and 20% as their “cut scores.” 4. Look at samples of student work just above and just below the cut scores. Professional judgment about the appropriateness of this number is then used to determine the local cut scores to determine Low, Moderate, and High Growth. This number may vary from assessment to assessment and from year to year based upon the district’s determination and upon the professional judgment of the district.

Using Excel Excel (really simple method) 1. Enter data 2. Simple pre-post formula 3. Cut and Paste Values 4.Sort Highest to Lowest Tabs include (quick tour): Fountas and Pinnell Sample for 6 teachers, 103 students with median for full assessment and for each teacher and with High, Moderate, and Low determinations Pre-Post-Test Calculations (for three teachers) Rubric Pre-Post (for three teaches) MCAS SGP (Student Growth Percentile) Calculations for three teachers A “Test” file that calculates the “gain,” but does not determine cut scores   Use the “test” tab to enter your local data Or use the data provided

Next Steps Gather data/tests themselves for analysis MCAS SGP on line Local Assessments Transfer to Excel Students Pre and Post or MCAS SGP or Post test only Cut scores Determine whole group’s median Determine teacher’s median Roster Verification Determine Low, Medium, or High Growth score for each teacher for each DDM