1. King Philip Day 2 District Determined Measures June 25, 2015 Dr. Deborah Brady dbrady3702@msn.com

2. Do Now Getting Online You may want to download from http://ddmsbrady.wikispaces.com  The Excel file (for calculating local DDMs)  The Grade 4 file (for looking at student work)  The Agenda, Mock Assessment Protocol, Checklist

3. Agenda I. Collecting DDMs, Assuring High Quality II. Scoring DDMs: Calibration and Calculations Group Work: Calibration Protocols—Calibrating with Rubrics Group Work: Excel Calculation of pre-post, rubrics, and MCAS SGP Lunch III. Time to work your DDMs

4. Consistency in Directions for Administrating Assessments  Directions to teachers need to define rules for giving support, dictionary use, etc.  What can be done? What cannot?  “Are you sure you are finished?”  How much time?  Accommodations and modifications?

5. Scoring Reminder  Validity—Does it test what it says it tests?  Are the assessors’ ratings calibrated?  Floor and ceiling effects  Rubric concerns  Validity assessment after test is given  What happens to these scores and assessments?  Stored as an L, M, H for district  Used as a discussion topic with evaluator

6. Rubric “Cut Scores”  Create a “growth” rubric and describe a typical year’s growth  Translate into 100%  www.roobrix.com www.roobrix.com

7. Calibration Protocol

8. Considerations for Scoring Student Work  Districts will need to determine fair, efficient and accurate methods for scoring students’ work. (Use consistent directions for teachers.)  DDMs can be scored by the educators themselves, groups of teachers within the district, external raters, or commercial vendors.  For districts concerned about the quality of scoring when educators score their own student’s work, processes such as randomly re-scoring a selection of student work to ensure proper calibration or using teams of educators to score together, can improve the quality of the results.  When an educator plays a large role in scoring his/her own work, a supervisor may also choose to include the scoring process into making a determination of a Student Impact.

9. Mock Calibration 1. All of the readers come together and are provided student compositions for calibration. These compositions can be pre-selected by the facilitator from all of the writing submitted or can be provided by the teachers. To assure fairness in assessment, teachers will not evaluate their own students’ compositions, but will be provided with their students’ work once compositions are graded and the scores are entered for the entire class. The purpose of the calibration meeting is to make sure that all evaluators are assessing student work on the same scale. In addition, the papers used for calibration become the exemplars that all teachers will use during scoring. Rubrics alone are not sufficient for precise assessment. Encourage each scorer to make notations on these compositions. Differences are expected and will be discussed until it is clear that all compositions are assessed with the same standards. 2. Time is given for each scorer to read and to use the rubric provided (local, textbook, MCAS, PARCC, or 6-Trait rubrics are all appropriate) to score the first composition and to enter their score on a chart like the one illustrated below. After each scorer has entered his or her score, the facilitator discusses the reasons for the scores. Scorers may need to find examples for their scores within the compositions. 3. After there is consensus for the first paper, the scorers go on to a second and third until there is a composition that represents each level of the rubric. Generally, after the first one or two compositions are calibrated, the process goes quickly.

10. Mock Calibration  Ask teachers to select (or you can pre-select 2 low compositions, 2 average, and 2 high compositions (altogether) Try to select a clear range.  Photocopy all of the compositions and ask teachers to evaluate the top one and using the 6-level rubric and the 4-level rubric to enter their scores under the first composition. Suggest that they can comment on their copies. Continue with the calibration until you have an exemplar for all levels.  Then the discussion needs to begin to discuss the specific reasoning for each scoring  Assume each person has a good reason, but the purpose is to work toward consensus. Composition #1 st Exemplar 2nd3rd4th56 Content2,2,3,2,4,2 Conventions1,1,1,1,2,1

11. MCAS ( 2 Holistic) Rubrics 123456 ContentLittle topic/idea development, organization, and/or details Little or no awareness of audience and/or task Limited or weak topic/idea development, organization, and/or details Limited awareness of audience and/or task Rudimentary topic/idea development and/or organization Basic supporting details Simplistic language Moderate topic/idea development and organization Adequate, relevant details Some variety in language Full topic/idea development Logical organization Strong details Appropriate use of language Rich topic/idea development Careful and/or subtle organization Effective/rich use of language ConventionsErrors seriously interfere with communication AND Little control of sentence structure, grammar and usage, and mechanics Errors interfere somewhat with communication and/or Too many errors relative to the length of the essay or complexity of sentence structure, grammar and usage, and mechanics Errors do not interfere with communication and/or Few errors relative to length of essay or complexity of sentence structure, grammar and usage, and mechanics Control of sentence structure, grammar and usage, and mechanics (length and complexity of essay provide opportunity for student to show control of standard English conventions)

12. 4 th Grade Prompt http://www.doe.mass.edu/mcas/student/2014/question.asp x?GradeID=4&SubjectCode=ela&QuestionID=33207# You are finally old enough to baby-sit, and your first job is this afternoon! You will be spending the entire afternoon with a one- year-old. When you open the door you realize that instead of watching a one-year-old child, you will be watching a one-year-old elephant! Write a story about spending your afternoon with a baby elephant. Give enough details to show readers what your afternoon is like baby-sitting the elephant

15. 2014 MCAS Grade 4 English Language Arts Composition Topic/Idea Development - Score Point 3 This composition is rudimentary in topic development and organization. The straightforward introduction moves immediately to the surprise of discovering that the "baby" is a baby elephant. From here, though, only basic supporting details are demonstrated as this composition of five paragraphs unfolds. There is an interesting "snoring scenario" which briefly captures the babysitter's personality in his or her impatience with the elephant: "It was as if there were 100 bells surrounding the house and all ringing at the same time. I covered my ears with pillows, it didn't work. I put ear muffs on, it still didn't work. Finally I just woke him up. He was pretty upset." The job ends as the mom comes home and there is a brief exchange of mildly humorous dialogue. The conclusion is simplistic, reiterating that the experience was not enjoyable

16. Calculating Growth ScoresMCAS and Local What you need to understand as you are creating assessments

17. Growth Score FAQs from DESE  Do the same numbers of students have to be identified as having high, moderate, and low growth? There is no set percentage of students who need to be included in each category. Districts should set parameters for high, moderate, and low growth using a variety of approaches.  How do I know what low growth looks like? Districts should be guided by the professional judgment of educators. The guiding definition of low growth is that it is less than a year’s worth of growth relative to academic peers, while high growth is more than a year’s worth of growth. If the course meets for less than a year, districts should make inferences about a year’s worth of growth based on the growth expected during the time of the course.  Can I change scoring decisions when we use a DDM in the second year? It is expected that districts are building their knowledge and experience with DDMs. DDMs will undergo both small and large modifications from year to year. Changing or modifying scoring procedures is part of the continuous improvement of DDMs over time.  Will parameters of growth be comparable from one district to another? Different assessments serve different purposes. While statewide SGPs will provide a consistent metric across the Commonwealth and allow for district-to-district comparisons, DDMs are selected

18. Scoring and Storing Excel File (on line) Who gets the file? Who keeps the file, the student work?

19. Summary DDM Process for Determining L, M, H for Every Teacher Whole Grade Level or Course  Score the entire grade level or course or take the MCAS Growth Scores for all students  Rank the scores from highest to lowest (post minus pre or MCAS SGP)  Identify the median score for the entire group  Determine the “cut” scores for local assessments; MCAS 35 and 65 for classrooms Individual Teacher  Select students for each teacher  Rank the scores from highest to lowest  Identify the Median score  Is the median below or above the “cut” score? Is it in the middle?  Don’t forget Roster Verification might change the specific scores and, therefore, change the Median  Distribute scores to teachers for each DDM

20. Mock Scoring, Storing, Determining Cut Scores Protocol 1. Assess all students for the course or grade level. 2. Enter student names, teacher names pre-test scores, post-test scores, and the gain from pre-test to post-test. 3. Using the SORT function, rank all assessments from highest to lowest. Determine the local cut scores. DESE recommends 1.5 Standard Deviations from 50; local districts have selected 10%, 15%, and 20% as their “cut scores.” 4. Look at samples of student work just above and just below the cut scores. Professional judgment about the appropriateness of this number is then used to determine the local cut scores to determine Low, Moderate, and High Growth. This number may vary from assessment to assessment and from year to year based upon the district’s determination and upon the professional judgment of the district.

21. Sample Cut Score Determination (for local assessments) Pre-test Post test Difference Student Scores Sorted low to high Teacher score is based on the MEDIAN Score of her class for each DDM 2035155 Cut score LOW Growth Lowest ___% 2530515 305020 356025 356025 median teacher score 40703525 median Teacher score 406525 50752530 50803035 Top 20% 508535 Cut score HIGH GROWTH Highest ___?

22. Measuring Growth Example: Fountas and Pinnell P to Q; N to P; D to K

23. Teacher ATeacher BTeacher CTeacher DTeacher ETeacher F 57.56.53.59 5.57.56.5710 6.58.57.5710 6.51108.5710 6.531097 6.5310 12 7610 12.5 106.510 11.512.5 126.510 12.512 None712101213 712 1613 912 1613.5 91313.51613.5 101313.51613.5 101313.51613.6 12161716.515.5 121716.519 121716.5 1217 1617 Median 6.5Median 9Median 12Median 10Median 16Median 12 Below 6.5 Between 6.5 and 16 LOWModerate HighModerate All Classes 1 3 3 3 3.5 5.5 6 6.5 7 7 7 7 7 7 7 7 7.5 8.5 9 9 9 9 9 9.5 10 10.7 10.9 11.0 11.2 11.4 11.5 11.7 11.9 12.0 12.2 12.5 12.6 12.7 12.9 13.1 13.2 13.4 13.6 13.8 13.9 14.1 14.3 14.4 14.6 14.8 15.0 15.1 15.3 15.5 15.6 12.5 13 13.5 13.6 15.5 16 16.5 17 19 26 6.5 Cut Score lowest 15% 12.0 Median for whole Grade 3 DDM 16 cut score highest 15% 103 Third Graders

24. To Be High or Low Half of the students have to be above the cut score or Half have to be below the cut score 3.5 7 7 7 7 10 11.5 12 16 16.5 17 5 5.5 6.5 7 10 12

25. Excel File Tour

27. Fountas and Pinnell Growth for Each Student Is Based on 10 Months of Growth Second Grade Student Level Achievement Level End of Year Levels from beginning to the end of the year Pre-Post F&P Levels Growth HIGH, MODERATE, OR LOW GROWTH (10 MONTHS=YEAR) QAbove BenchmarkPQ 7 MONTHS GROWTHLOW GROWTH PAt BenchmarkNOP 10 MONTHS OF GROWTH MODERATE GROWTH KBelow BenchmarkDEFGHIJK17 MONTHS OF GROWTH HIGH GROWTH

28. Using Excel Excel (really simple method) 1. Enter data 2. Simple pre-post formula 3. Cut and Paste Values 4.Sort Highest to Lowest Tabs include (quick tour):  Fountas and Pinnell Sample for 6 teachers, 103 students with median for full assessment and for each teacher and with High, Moderate, and Low determinations  Pre-Post-Test Calculations (for three teachers)  Rubric Pre-Post (for three teaches)  MCAS SGP (Student Growth Percentile) Calculations for three teachers  A “Test” file that calculates the “gain,” but does not determine cut scores Use the “test” tab to enter your local data Or use the data provided

29. Time to Work with Your Team  On line materials  Excel  Templates