Auburn and Leicester DDM Scoring March 2015 Dr. Deborah Brady

Achievement Versus Growth Think-Pair-Share Growth

Using an Excel File  Laptop: Username: Nichols Password: Nichols  Upper Right Corner File: DDM  DDM Grade 8: ORQ  Grade 1 DRA  Grade 2 math  K number Grid Assessment  Spanish and French pre-post test data  Select one and save it with your name so that you can make changes  1) EXCEL file

Excel Directions On Line File  The process recommended by DESE is:  1) List the students and their teachers  2) Enter the pre test scores  3) Enter the post-test scores  4) Calculate the gain score (post minus pre) for each student  5) Rank the students from lowest to highest using the Excel SORT function  6) Determine the MEDIAN for the entire group  7) Determine the "cut scores" for the ranked students. DESE says the cut scores should be 1.5 SD from center, but for simplicity's sake, you can use 10% then look at the students' scores to determine if these students were low or high performing. This 10% can be adjusted based upon your professional judgment. Also note: The high gain scores should include at risk and typically high achieving students; the low gain scores should also show a mix of achievement levels to indicate that the test provided an opportunity for students to improve.  8) Determine the median for each teacher  9) If the teacher's median is above the low cut score and below the high cut score, the gain is MODERATAE.  10) If the teacher's median is below the low cut score, the teacher's impact rating is LOW  11) If the teacher's median is above the high cut score, the teacher's impact rating is HIGH

Excel File Tour

Scoring and DDM Design Considerations  Districts will need to determine fair, efficient and accurate methods for scoring students’ work.  DDMs can be scored by the educators themselves, groups of teachers within the district, external raters, or commercial vendors.  For districts concerned about the quality of scoring when educators score their own student’s work, processes such as randomly re-scoring a selection of student work to ensure proper calibration or using teams of educators to score together, can improve the quality of the results.  When an educator plays a large role in scoring his/her own work, a supervisor may also choose to include the scoring process into making a determination of a Student Impact.  NOTE: Teacher Instructions may be necessary

Growth Score FAQs from DESE  Do the same numbers of students have to be identified as having high, moderate, and low growth? There is no set percentage of students who need to be included in each category. Districts should set parameters for high, moderate, and low growth using a variety of approaches.  How do I know what low growth looks like? Districts should be guided by the professional judgment of educators. The guiding definition of low growth is that it is less than a year’s worth of growth relative to academic peers, while high growth is more than a year’s worth of growth. If the course meets for less than a year, districts should make inferences about a year’s worth of growth based on the growth expected during the time of the course.  Can I change scoring decisions when we use a DDM in the second year? It is expected that districts are building their knowledge and experience with DDMs. DDMs will undergo both small and large modifications from year to year. Changing or modifying scoring procedures is part of the continuous improvement of DDMs over time.  Will parameters of growth be comparable from one district to another? Different assessments serve different purposes. While statewide SGPs will provide a consistent metric across the Commonwealth and allow for district-to-district comparisons, DDMs are selected

Getting On Line  Or (Pages are listed on the right.)  All materials are available to download on this site.  Any material can be used, modified, shared with your colleagues  Problems?

Quality Assessments  “Substantive”  Aligned with at least 2 standards of Frameworks  And/or local standards  Rigorous  Consistent in substance, alignment, and rigor  Consistent with the District’s values, initiatives, expectations  Measure growth (to be contrasted with achievement) and shifts the focus of teaching toward the individual learner’s change over time

Calculating Growth ScoresMCAS SGP and Local What you need to understand as you are creating assessments

MCAS Student Growth Percentile Local Manipulation of Scores (4-8; ELA or Math; not grade 10 SG{)

Sample Plan Part I SGP and Local Common Assessments* ELAMathScienceSocial Studies 12CA/CA 11CA/CA 10CA/CA 9 8MCAS SGP/CA CA/CA 7MCAS SGP/CA CA/CA 6MCAS SGP/CA CA/CA 5MCAS SGP/CA 4 3CA/CA 2 Common Assessments or Locally Determined 1CA/CA KCA

Sample Plan Part II with Specialists, Singletons Singleton, Art, Music, Technology, PE by Grade or grade- spans Special Education Specialists, Co- Teachers, substantially separate Indirect Measures (IM) Central Office, Psychologist Administrators Principals, Assistant Principals, Superintendent, Coordinators 12CA/CA Inclusion Co-Teachers Can “share” scores with General Ed Teachers Or If their students’ goals are substantially different, the assessments can be modified or can focus on the goal of inclusion Specialists Measure Goals Can develop K-12 rubric Speech PE Adaptation of the SMART goal process Measure goal that has an impact on student growth Attendance College Applications Technology Growth MCAS SGP Either ELA or Math 4-10 PLUS IM 11CA/CA 10CA/CA K

to 244/ 25 SGP 230 to 230/ 35 SGP 214 to 225/ 92 SGP Student Growth Percentiles (SGP Calculation for individual students )

MCAS SGP ELA/Math for Grades 4-8 (not 10) Small Classes, Co-Taught Classes  The median SGP must be used when a teacher has 20 or more students (altogether) in a content area (ELA or mathematics 3-8)  Median SGPs for 8-19 students have validity and may be used if the district determines this is appropriate  More than one educator (a classroom teacher, a support teacher, a curriculum supervisor, and a supervising administrator) may be considered responsible for a content-area SGP. Different students may be in their respective rosters.

“Cut Scores” for MCAS SGP Percent of students Lower growth Higher growth Typical growth One year’s growth 65 Classroom 35 Classroom 40 Whole Grade 60 Whole Grade

MCAS SGP Cut Scores LowModerateHigh Teachers35* or lower Greater than 35, but less than or higher Administrators40 or lower Greater than 40, but less than or higher More latitude is given to teachers because of the statistically small numbers in a classroom (at least 20 students). Administrative “cut scores” for Low and High are based upon the entire class’ scores

Median student growth percentile for one teacher Last nameSGP Lennon6 McCartney12 Starr21 Harrison32 Jagger34 Richards47 Crosby55 Stills61 Nash63 Young74 Joplin81 Hendrix88 Jones95 Imagine that the list of students to the left are all the students in your 6 th grade class. Note that they are sorted from lowest to highest SGP. The point where 50% of students have a higher SGP and 50% have a lower SGP is the median. The 47 (median) is the teacher’s score (one DDM data point) if this includes all of her students. Median SGP for the 6 th grade class

Measuring Local Growth Scores Using the DESE’s Recommended Process Pre- and Post-Tests Rubrics

DESE Recommended Process for Calculating DDMs For the entire course or grade level: 1) List the students and their teachers 2) Enter the pre-test scores 3) Enter the post-test scores 4) Calculate the gain score (post minus pre) for each student 6) Determine the MEDIAN for the entire group 7) Determine the "cut scores" for the ranked students NOTE: DESE says the cut scores should be 1.5 Standard Deviations from center, but for simplicity's sake, you can use 10%, then look at the students' scores to determine if these students were low or high performing. This 10% can be adjusted based upon your professional judgment. Also note: The high gain scores should include at risk and typically high achieving students; the low gain scores should also show a mix of achievement levels to indicate that the test provided an opportunity for students to improve. The individual teacher’s score: Select the individual teacher’s students and determine the median 1. If the teacher's median is above the low cut score and below the high cut score, the gain is MODERATAE 2. If the teacher's median is below the low cut score, the teacher's impact rating is LOW 3. If the teacher's median is above the high cut score, the teacher's impact rating is HIGH NOTE: This statement is on the first tab labeled DIRECTIONS.

Process Using Pre-Post and Median First All Students Assessed For the entire course or grade level: 1) List the students and their teachers 2) Enter the pre-test scores 3) Enter the post-test scores 4) Calculate the gain score (post minus pre) for each student 6) Determine the MEDIAN for the entire group 7) Determine the "cut scores" for the ranked students Then Redistribute to Each Teacher Select the individual teacher’s students and determine the median  If the teacher's median is above the low cut score and below the high cut score, the gain is MODERATAE  If the teacher's median is below the low cut score, the teacher's impact rating is LOW  If the teacher's median is above the high cut score, the teacher's impact rating is HIGH

Sample Cut Score Determination (for local assessments) Pre-test Post test Difference Student Scores Sorted low to high Teacher score is based on the MEDIAN Score of her class for each DDM Cut score LOW Growth Lowest ___% median teacher score median Teacher score Top 20% Cut score HIGH GROWTH Highest ___?

Excel File Tour

Teacher ATeacher BTeacher CTeacher DTeacher ETeacher F None Median 6.5Median 9Median 12Median 10Median 16Median 12 Below 6.5 Between 6.5 and 16 LOWModerate HIGHModerate 6.5 Cut Score lowest 15% 12.0 Median for whole Grade 3 DDM 16 cut score highest 15% 103 Third Graders All Classes

To Be High or Low Half of the students have to be above the cut score or Half have to be below the cut score

Measuring Growth Example: Fountas and Pinnell P to Q; N to P; D to K

Fountas and Pinnell Growth for Each Student Is Based on 10 Months of Growth Second Grade Student Level Achievement Level End of Year Levels from beginning to the end of the year Pre-Post F&P Levels Growth HIGH, MODERATE, OR LOW GROWTH (10 MONTHS=YEAR) QAbove BenchmarkPQ 7 MONTHS GROWTHLOW GROWTH PAt BenchmarkNOP 10 MONTHS OF GROWTH MODERATE GROWTH KBelow BenchmarkDEFGHIJK17 MONTHS OF GROWTH HIGH GROWTH

Summary DDM Process for Determining L, M, H for Every Teacher Whole Grade Level or Course  Score the entire grade level or course or take the MCAS Growth Scores for all students  Rank the scores from highest to lowest (post minus pre or MCAS SGP)  Identify the median score for the entire group  Determine the “cut” scores for local assessments; MCAS 35 and 65 for classrooms Individual Teacher  Select students for each teacher  Rank the scores from highest to lowest  Identify the Median score  Is the median below or above the “cut” score? Is it in the middle?  Don’t forget Roster Verification might change the specific scores and, therefore, change the Median  Distribute scores to teachers for each DDM

Beware Rubrics! Holistic Rubric Show Progress across a Scale, Continuum, Descriptors Details No improvement in the level of detail. One is true * No new details across versions * New details are added, but not included in future versions. * A few new details are added that are not relevant, accurate or meaningful Modest improvement in the level of detail One is true * There are a few details included across all versions * There are many added details are included, but they are not included consistently, or none are improved or elaborated upon. * There are many added details, but several are not relevant, accurate or meaningful Considerable Improvement in the level of detail All are true * There are many examples of added details across all versions, * At least one example of a detail that is improved or elaborated in future versions *Details are consistently included in future versions *The added details reflect relevant and meaningful additions Outstanding Improvement in the level of detail All are true * On average there are multiple details added across every version * There are multiple examples of details that build and elaborate on previous versions * The added details reflect the most relevant and meaningful additions Example taken from Austin, a first grader from Answer Charter School in Boise, Idaho. Used with permission from Expeditionary Learning. Learn more about this and other examples at

Preconventional Ages 3-5 Emerging Ages 4-6 Developing Ages 5-7 Beginning Ages 6-8 Expanding Ages 7-9 2Relies primarily on pictures to convey meaning. 2Begins to label and add “words” to pictures. 2Writes first name. Uses pictures and print to convey meaning. Writes words to describe or support pictures. Copies signs, labels, names, and words (environmental print). Writes 1-2 sentences about a topic. Writes names and familiar words.. 2Writes several sentences about a topic. 2Writes about observations and experiences. 2Writes short nonfiction pieces (simple facts about a topic) with guidance. Writes short fiction and poetry with guidance. Writes a variety of short nonfiction pieces (e.g., facts about a topic, letters, lists) with guidance. Writes with a central idea. Writes using complete sentences. Bridging Ages 8-10 Fluent Ages 9-11 Proficient Ages Connecting Ages Independent Writes about feelings and opinions. Writes fiction with clear beginning, middle, and end. Writes poetry using carefully chosen language with guidance. Writes organized nonfiction pieces (e.g., reports, letters, and lists) with guidance. Begins to use paragraphs to organize ideas. Uses strong verbs, interesting language, and dialogue with guidance. 2Begins to write organized fiction and nonfiction (e.g., reports, letters, biographies, and autobiographies). 2Develops stories with plots that include problems and solutions with guidance. 2Creates characters in stories with guidance. 2Writes poetry using carefully chosen language. 1Begins to experiment with sentence length and complex sentence structure. 1Varies leads and endings with guidance. 1Uses description, details, and similes with guidance.  Uses dialogue with guidance. Writes persuasively about ideas, feelings, and opinions. Creates plots with problems and solutions. Begins to develop the main characters and describe detailed settings. Begins to write organized and fluent nonfiction, including simple bibliographies. Writes cohesive paragraphs including reasons and examples with guidance. Uses transitional sentences to connect paragraphs. Varies sentence structure, leads, and endings. Begins to use descriptive language, details, and similes. Uses voice to evoke emotional response from readers. Begins to integrate information on a topic from a variety of sources. 2Writes in a variety of genres and forms for different audiences and purposes independently. 2Creates plots with a climax. 2Creates detailed, believable settings and characters in stories. 2Writes organized, fluent, and detailed nonfiction independently, including bibliographies with correct format. 1Writes cohesive paragraphs including supportive reasons and examples. 1Uses descriptive language, details, similes, and imagery to enhance ideas independently. 1Begins to use dialogue to enhance character development. 1Incorporates personal voice in writing with increasing frequency. 2Writes organized, fluent, accurate, and in-depth nonfiction, including references with correct bibliographic format. 2Writes cohesive, fluent, and effective poetry and fiction. 1Uses a clear sequence of paragraphs with effective transitions. 1Begins to incorporate literary devices (e.g., imagery, metaphors, personification, and foreshadowing). 1Weaves dialogue effectively into stories. 1Develops plots, characters, setting, and mood (literary elements) effectively. 1Begins to develop personal voice and style of writing. ".  J 

4(25)= 100 4(22)= 88 4(18)= 72 4(15)= 60 x x x x = 80% Criterion Referenced Rubric and Raw Scores or % of 100

AP Rubric of Rubrics Prose Analysis (9 levels give students room to improve Holistic) 9-8 Answers all parts of the question completely. Using specific evidence from the work and showing how that evidence is relevant to the point being made. Fashions a convincing thesis and guides reader through the intricacies of argument with sophisticated transitions. Demonstrates clear understanding of the work and recognizes complexities of attitude/tone. Demonstrates stylistic maturity by an effective command of sentence structure, diction, and organization. Need not be without flaws, but must reveal an ability to choose from and control a wide range of the elements of effective writing. 7-6 Also accurately answers all parts of the question, but does so less fully or effectively than essays in the top range. Fashions a sound thesis. Discussion will be less thorough and less specific, not so responsive to the rich suggestiveness of the passage or precise in discussing its impact. Well written in an appropriate style, but with less maturity than the top papers. Some lapses in diction or syntax may appear, but demonstrates sufficient control over the elements of composition to present the writer’s ideas clearly. Confirms the writer’s ability to read literary texts with comprehension and to write with organization and control. 5 Discusses the question, but may be simplistic or imprecise. Constructs a reasonable if reductive thesis. May attempt to discuss techniques or evidence in the passage, but may be overly general or vague. Adequately written, but may demonstrate inconsistent control over the elements of composition. Organization is attempted, but may not be fully realized or particularly effective. 4-3 Attempts to answer the question, but does so either inaccurately or without the support of specific evidence. May confuse the attitude / tone of the passage or may overlook tone shift(s) or otherwise misrepresent the passage. Discussion of illustrations / techniques / necessary parts of the prompt may be omitted or inaccurate. Writing may convey the writer’s ideas, but reveals weak control over diction, syntax, or organization. May contain many spelling or grammatical errors. Essays scored three are even less able and may not refer to illustrations / techniques at all. 2-1 Fails to respond adequately to the question. May misunderstand the question or the passage. May fail to discuss techniques / evidence used or otherwise fail to respond adequately to the question. Unacceptably brief or poorly written on several counts. Writing reveals consistent weakness in grammar or other basic elements of composition. Although may make some attempt to answer the question, response has little clarity and only slight, if any, evidence in its support. Although the writer may have made some attempt to answer the prompt, the views presented have little clarity or coherence; significant problems with reading comprehension seem evident. Essays that are especially inexact, vacuous, and /or mechanically unsound should be scored 1. 0 A blank paper or one that makes no attempt to deal with the question receives no credit. Rubric from Sharon Kingston

Rubric “Cut Scores”  Create a “growth” rubric and describe a typical year’s growth  Translate into 100% 

Post Test Only Use District’s History to predict scores on AP Calculus Exam, for example Previous Grade’s Math Score Low Growth Moderate Growth High Growth A345 B234 C123 D12

Important Perspective It is expected that districts are building their knowledge and experience with DDMs. DDMs will undergo both small and large modifications from year to year. Changing or modifying scoring procedures is part of the continuous improvement of DDMs over time. We are all learners in this initiative.

Next Steps  Gather data/tests themselves for analysis  MCAS SGP on line  Local Assessments  Transfer to Excel  Students  Pre and Post or MCAS SGP or Post test only  Cut scores  Determine whole group’s median  Determine teacher’s median  Roster Verification  Determine Low, Medium, or High Growth score for each teacher for each DDM

Protocols That Can Be Used with Implemented Assessments  Floor and Ceiling Effects (Can everyone demonstrate growth?)  Validating the Quality of Multiple Choice Questions  Inter-Rater Reliability with Rubrics and Scoring guides  Low-Medium-High Looking at Student Work Protocol (calibration, developing exemplar, developing action plan)