1. Assessment Workshop Creating and Evaluating High Quality Assessments Dr. Deborah Brady

2. Agenda I. Introductions: Overview  Break at about 10, lunch at about 12, session ends at about 3:00  Morning presentation (with frequent processing breaks) and some afternoon time for beginning to plan  High quality Assessments (DESE criteria)  Tools to evaluate assessments  Tools to track all educators’ DDMs  Measuring Student Growth  Direct measures  Local alternatives to determine growth  Pre-/Post, Holistic Rubrics, Measures over time, Post-test only  “Standardization” an alternative, but not required  Indirect measures  Piloting, preparing for full implementation in SY 2015  EXIT SLIPS—questions, priorities, planning for full implementation  My email dbrady3702@msn.com;dbrady3702@msn.com

3. The Steps Necessary to Get Ready for June Report and After Adapting present assessments Creating new assessments Writing to text for HS Developing and Piloting Assessments Alignment of Content Rigorous and appropriate expectations Approval of assessments Assessing Quality and Rigor Security Calibration of standards and of assessors Rubric quality Analysis of results: High-M-Low Growth Piloting 2 DDMs per educator JUNE REPORT Directions for teachers Directions for students Organizing for the actual assessments Storing, tracking the information 2015 Full Implementation Data storage Data Analysis L-M-H Growth Roster Verification Data team time Interpreting the results Student Impact

4. Carousel Walk and Living Likert Scale Carousel Walk  Take a marker for your team (or yourself if you are on your own)  Visit each of the 5 “stages” of DDM development  Put a check to the right of each item that you have completed  After viewing each stage, return to the stage that most represents you and/or your district Living Likert  Go to your “stage”  What are the barriers? What are the strengths that your district has?  Discuss the barriers and strengths with the others at your stage.  Be prepared to report out as a whole group.

5. Potential as Transformative Process When C, I or A is changed…. Elmore, Instructional Rounds, and the “task predicts performance” Curriculum Instruction Assessment

6. Goals for Today  Answer your questions and provide tools/materials to support you  ( Use note cards/exit slips)  By the end of this session  You will understand what needs to be done and explain it to your colleagues  You will have tools to begin to do that work in your district  You may still have unique questions— Please ask them!  dbrady3702@msn.com dbrady3702@msn.com  http://tinyurl.com/lvn5ome materials from this presentation http://tinyurl.com/lvn5ome  This is one page from my Wikispaces site Writing to Text

7. The DESE Requirements Purpose, timeline, requirements, direct and indirect assessments

8. District Determined Measures DEFINITION DDMs are defined as: “Measures of student learning, growth, and achievement related to the Curriculum Frameworks, that are comparable across grade or subject level district-wide” TYPES OF MEASURES  Portfolio assessments  Approved commercial assessments  District developed pre and post unit and course assessments  Capstone projects

9. 2013-2014 District-wide training, development of assessments and piloting June 2014: Report: All educators in the district have 2 DDMs to be implemented fully in SY2015. 2014-2015 All DDMs are implemented; scores are divided into H-M-and Low and stored locally 2015-2016 Second year data is collected and all educators receive an impact rating that is sent to DESE based on 2 years of data for two DDMs Timeline for Piloting and Full Implementation

10. District Determined Measures Regulations  Every educator will need data from at least 2 different measures  Trends must be measured over a course of at least 2 years  One measure must be taken from State-wide testing data such as MCAS if available (grades 4-8 ELA and Math SGP for classroom educators)  One measure must be taken from at least one District Determined Measure which can include Galileo, normed assessments (DRA, MAP, SAT)

11. NEW! Identifying and Selecting DDMs  Establishes a DDM Working Group  Co-chaired by superintendent and president of local bargaining unit or their designees.  Surveys the district for available assessments  Recruits educators to identify assessments and make recommendations  Identifies at least two measures for each educator  Collects feedback on the quality of the DDMs (continuous improvement)  Makes recommendations to the superintendent

12. NEW! Determining Educator Impact on Each DDM  Evaluator and educator meet. Evaluator determines whether students demonstrated high, moderate, or low growth on each DDM.  Evaluator shares the resulting designations of student growth with educator.  Educators confirm rosters.  Must be on roster by 10/1 and remain on roster through last day of testing.  Must be present for 90% of instructional time.

13. NEW! Identifying and Selecting DDMs  Describes process for selecting DDMs  Working group makes recommendations to the superintendent.  If superintendent declines, expedited resolution process is triggered:  Parties petition the Commissioner  Commissioner forwards list of 3 hearing officers with curriculum and/or assessment expertise  Parties choose hearing officer  Hearing officer renders final decision  Educators are informed of their DDMs by fourth week of school.  Educators receive appropriate professional development.

14. Performance & Impact Ratings Performance Rating Ratings are obtained through data collected from observations, walk- throughs and artifacts  Exemplary  Proficient  Needs Improvement  Unsatisfactory Impact Rating Ratings are based on trends and patterns in student learning, growth and achievement over a period of at least 2 years Data gathered from DDM’s and State- wide testing  High  Moderate  Low

15. NEW! Determining a Student Impact Rating  Introduces the application of professional judgment to determine the Student Impact Rating  Evaluator assigns rating using professional judgment.  Evaluator considers designations of high, moderate, or low student growth from at least two measures in each of at least two years.  If rating is low, evaluator meets with educator to discuss  If rating is moderate or high, evaluator/educator decide if meeting is necessary.

16. Summative Rating Exemplary 1-yr Self- Directed Growth Plan 2-yr Self-Directed Growth Plan Proficient Needs Improvement Directed Growth Plan UnsatisfactoryImprovement Plan LowModerateHigh Rating of Impact on Student Learning Massachusetts Department of Elementary and Secondary Education 16 Impact Rating on Student Performance

17. NEW! Intersection of Ratings  Reinforces independent nature of the two ratings.  Exemplary or Proficient matched with Moderate or High = 2-Year Self-Directed Growth Plan  Exemplary/ Moderate and Exemplary/ High = recognition and rewards, including leadership roles, promotions, additional compensation, public commendation, and other acknowledgements.  Proficient/Moderate and Proficient/ High = eligible for additional roles, responsibilities, and compensation.  Exemplary or Proficient matched with Low = 1-Year Self-Directed Growth Plan  Evaluator’s supervisor confirms rating.  Educator and evaluator analyze the discrepancy.  May impact Educator Plan goals.  Student Impact Rating informs the self-assessment and goal setting processes.

18.  Indirect measures of student learning, growth, or achievement provide information about students from means other than student work.  These measures may include student record information (e.g., grades, attendance or tardiness records, or other data related to student growth or achievement such as high school graduation or college enrollment rates).  To be considered for use as DDMs, a link (relationship) between indirect measures and student growth or achievement must be established.  For some educators such as district administrators and guidance counselors, it may be appropriate to use one indirect measure of student learning along with other direct measures;  ESE recommends that at least one of the measures used to determine each educator’s student impact rating be a direct measure.

19. Indirect Measure Examples  Consider SST Process for a team:  High school SST team example  Child Study Team example  RTI team example  High school guidance example  Subgroups of students can be studied (School Psychologist group example)  Social-emotional growth is appropriate (Autistic/Behavioral Program example)  Number of times each student says hello to a non-classroom adult on his or her way to gym or class (Direct)  Number of days (or classes) a student with school anxiety participates  Assess level of participation in a class (Direct)  Increase the “in-depth” studies of at risk students  Make sure students go through the referral process to decrease the number of students who are unnecessarily assessed  Improve applications to college  IEP goals can be used as long as they are measuring growth (academic or social-emotional)

20. Table Talk (5 minutes) Using the 6-phase overview, what are your priorities?

21. Adapting present assessments Creating new assessments Writing to text for HS Developing and Piloting Assessments Alignment of Content Rigorous and appropriate expectations Assessing Quality and Rigor Security Calibration of standards and of assessors Rubric quality Analysis of results: High- M-Low Growth Piloting 2 DDMs per educator JUNE REPORT Directions for teachers Directions for students Organizing for the actual assessments Storing, tracking the information 2015 Full Implementation Data storage Data Analysis L-M-H Growth Interpreting the results Student Impact

22. Assessment Quality Requirements and Definitions from DESE Alignment, Rigor, Comparability, “Substantial,” Modifications

23. What are the requirements?  1. Is the measure aligned to content?  Does it assess what is most important for students to learn and be able to do?  Does it assess what the educators intend to teach?  Bottom Line: “substantial” content of course  At least 2 standards  ELA: reading/writing  Math: Unit exam  Not necessarily a “final” exam (unless it’s a high quality exam) 23

24.  2. Is the measure informative?  Do the results of the measure inform educators about curriculum, instruction, and practice?  Does it provide valuable information to educators about their students?  Does it provide valuable information to schools and districts about their educators? Bottom Line: Time to analyze is essential 24

25. Five Considerations (DESE) 1. Measure growth 2. Employ a common administration procedure 3. Use a common scoring process 4. Translate these assessments to an Impact Rating 5. Assure comparability of assessments (rigor, validity). 25

26. Comparability  Comparable within a grade, subject, or course across schools within a district  Identical measures are recommended across a grade, department, course  Comparable across grade or subject level district-wide  Impact Ratings should have a consistent meaning across educators; therefore, DDMs should not have significantly different levels of rigor 26

27. Two Considerations for Local DDMs, 1. Comparable across schools  Example: Teachers with the same job (e.g., all 5 th grade teachers)  Where possible, measures are identical  Easier to compare identical measures  Do identical measures provide meaningful information about all students?  Exceptions: When might assessments not be identical?  Different content (different sections of Algebra I)  Differences in untested skills (reading and writing on math test for ELL students)  Other accommodations (fewer questions to students who need more time )  NOTE: Roster Verification and Group Size will be considerations by DESE 27

28. 2. Comparable across the District  Aligned to your curriculum (comparable content) K-12 in all disciplines  Appropriate for your students  Aligned to your district’s content  Informative, useful to teachers and administrators  “Substantial” Assessments (comparable rigor):  “Substantial” units with multiple standards and/or concepts assessed. (DESE began talking about finals/midterms as preferable recently) See Core Curriculum Objectives (CCOs) on DESE website if you are concerned http://www.doe.mass.edu/edeval/ddm/example /  Quarterly, benchmarks, mid-terms, and common end of year exams  NOTE: All of this data stays in your district. Only HML goes to DESE with a MEPID for each educator.

29. Rigor Alignment Rigorous  2011 Massachusetts Frameworks  Common Core Shifts  Complex texts  Complex tasks  Writing to text  Shift in Persuasive Essay (Formal Argument)  Shift in Narrative (More substantial and linked to content)  Shift in Informational Text (organization substantiation)  Math, Science, History/SS frameworks Aligned to District curriculum  Shifted to new expectations  Shifted from MCAS expectations  Consider PARCC  This is a district decision  Gradual increments?  Giant steps?

30. Writing to Text and PARCC The Next Step?  The 2011 MA Frameworks Shifts to the Common Core  Complex Texts  Complex Tasks  Multiple Texts  Increased Writing A Giant Step? Increase in cognitive load  Mass Model Units—PBL with Performance-Based Assessments (CEPAs)  PARCC assessments require matching multiple texts

31. Two Forms to Adapt to Your Standards  Handout—DDM Proposal form  Excel file (on wiki) Simple Excel List

32. Last Name First Name Grade/Dept. DDM1DDM2DDM3 (optional) Jones BrigitELA 6MCAS 6 Growth Score ELA ELA 6 DDM (writing to text) Smith Marion9-12 library Library Search Tools DDM Indirect: Increase teachers who do research in library. Watso n Elspeth5 ELA team Fountas and Pinnell DDM MCAS 5 Growth Score History Unit Exam DDM Holme s Sharon2Fountas and Pinnell DDM Galileo DDM June Report Form (Not Yet Released) Educators Linked with DDMs

33. Handout Sample Check all Items that are completedDefinitionYour Answers Here Source of DDM  Locally developed  Standardized test Are you developing the assessment as a department or team, or is your school/district purchasing an assessment? These first four categories can be used for this year’s June report: Educator Grade/Department DDM name Source of DDM  Course What is the title of the course that this DDM will be given in?  Possible educators who will use this DDM Courses and teachers may change, but who at this time will probably teach this course?  Grade(s) of DDMGrade level(s) that this assessment will cover  Alignment to State and/or District Standards At least 2 standards must be assessed to make this assessment a “substantial” assessment For indirect measures, 1) what are the substantial, important, essential areas that you are assessing? 2) How does this indirect measure connect with student growth? Please list the two (or more) standards using standards language. 1. 2.  Rigor: Check the levels of Blooms that are assessed The original Bloom is the first word on the list. The new Bloom (all verbs) is the second. Note, in the new Bloom, Creating is on the highest level, above evaluating. More than one level can be assessed.  Knowledge, Remembering  Comprehension, Understanding  Application, Applying  Analysis, Analyzing  Synthesis, Creating  Evaluation, Evaluating

34.  Type(s) of questions  Multiple Choice, fill in, short answer (recall items from content area)  Multiple Choice, fill in, short answer (text dependent questions)  Open Response (short answer)  Essay (long response). Type:  Narrative  Informational Text  Argument with claims and proof  One text is read  Two texts are read  Performance Assessment (CEPA)  Other_______ (Fill in at right.) Indicate the percentage of the assessment for each question type, for example, multiple choice=50%; 2 open responses=50% (25% each). Multiple Choice _____% Open Response _____% Essay _____________%  Duration of assessment Assessments can take place in a class period or over a period of days. For next year’s scheduling and implementation  When assessment(s) will take place Provide approximate month or window for assessment(s), for example, end of first trimester, September. Provide multiple dates if the assessment is a pre-post or is administered more than once.  Components of assessment that are completed so far.  Directions to teacher for administering  Directions to students  Graphic organizers (optional)  The assessment  Scoring guide  Rubric  Security  Calibration protocol if this assessment has a rubric  Rubric  Not Yet  Does not apply How was the rubric created? For example, adapted from DESE’s CEPA rubric, or developed by the middle school science department. Please include rubric (even in draft form) Begin with: CEPA Or PARCC Or MCAS

35. Table Talk/Team Talk (10 min) How will you develop quality assessments?

36. Calculating Growth Scores Defining growth, measuring growth, calculating growth for a classroom, for a district

37. 4503699 244/ 25 SGP 230/ 35 SGP 225/ 92 SGP Sample Student GROWTH SCORES from the MCAS TEACHER GROWTH SCORES are developed from student growth scores

38. Approaches to Measuring Student Growth  Pre-Test/Post Test  Repeated Measures  Holistic Evaluation  Post-Test Only 38

39. Pre/Post Test  Description:  The same or similar assessments administered at the beginning and at the end of the course or year  Example: Grade 10 ELA writing assessment aligned to College and Career Readiness Standards at beginning and end of year with the passages changed  Measuring Growth:  Difference between pre- and post-test.  Considerations:  Do all students have an equal chance of demonstrating growth? 39

40. Pre-Post Analysis Cut Scores for L-M-H Growth Pre-test Post test Difference %age growth Diff/pre %age growth low to high Sort low to high diff ONE “mock” classroom 20351575%20%5 Cut scoreLOW Growth 2530520%42%15 bottom 20% 30502067%42%20 35602542%50%25 Moderate Growth 35602542%60%25 medianteacher score 40703587%62%25 medianTeacher score 40652562%67%25 50752550%70%30 50803060%75%35 Cut ScoreTop 20%? 50853570%87%35 HIGH GROWTH

41. Determining Growth with Pre- and Post Assessments  Cut scores need to be locally determined for local assessments  Standardized assessments use “The Body of the Work” protocol which easily translates to local assessments  First determine the difference between pre- and post- scores for all students in a grade or course  Then determine what Low Moderate and High growth is. (Local cut scores)  Top and bottom 10% to begin as a test case  Body of the Work check  Then all scores are reapportioned to each teacher  The MEDIAN score for each teacher determines that teacher’s growth score

42. Further measures beyond pre- and post- tests Repeated measures, Holistic Rubrics, Post-Test Only

43. Repeated Measures  Description:  Multiple assessments given throughout the year.  Example: running records, attendance, mile run  Measuring Growth:  Graphically  Ranging from the sophisticated to simple  Considerations:  Less pressure on each administration.  Authentic Tasks 43

44. Repeated Measures Example Running Record 44 Date of Administration # of errors

45. Holistic  Description:  Assess growth across student work collected throughout the year.  Example: Tennessee Arts Growth Measure System  Measuring Growth:  Growth Rubric (see example)  Considerations:  Option for multifaceted performance assessments  Rating can be challenging & time consuming 45

46. Holistic Example 46 1234 Details No improvement in the level of detail. One is true * No new details across versions * New details are added, but not included in future versions. * A few new details are added that are not relevant, accurate or meaningful Modest improvement in the level of detail One is true * There are a few details included across all versions * There are many added details are included, but they are not included consistently, or none are improved or elaborated upon. * There are many added details, but several are not relevant, accurate or meaningful Considerable Improvement in the level of detail All are true * There are many examples of added details across all versions, * At least one example of a detail that is improved or elaborated in future versions *Details are consistently included in future versions *The added details reflect relevant and meaningful additions Outstanding Improvement in the level of detail All are true * On average there are multiple details added across every version * There are multiple examples of details that build and elaborate on previous versions * The added details reflect the most relevant and meaningful additions Example taken from Austin, a first grader from Anser Charter School in Boise, Idaho. Used with permission from Expeditionary Learning. Learn more about this and other examples at http://elschools.org/student- work/butterfly-draftshttp://elschools.org/student- work/butterfly-drafts

47. Post-Test Only  Description:  A single assessment or data that is paired with other information  Example: AP exam  Measuring Growth, where possible:  Use a baseline  Assume equal beginning  Considerations:  May be only option for some indirect measures  What is the quality of the baseline information? 47

48. Post-Test Only A challenge to tabulate growth  Portfolios  Measuring achievement v. growth  Unit Assessments  Looking at growth across a series  Capstone Projects  May be a very strong measure of achievement 48

49. Table/Team Talk Discuss the calculations, security, storage, fairness of determining local cut scores.

50. “Tools” to Support the Process  For determining what is important (Core Curriculum Objectives)  For determining adequacy for use as DDM (Quality Tool)  For making sure each educator has 2 DDMs (Excel Sheet)  For assessing rigor (Cognitive Complexity Rubric, CEPA Rubric)

51. Core Curriculum Objectives (CCOs—partial list for Writing to Text)

52. ELA-Literacy — 9 English 9-12 https://wested.app.box.com/s/pt3e203fcjfg9z8r02si https://wested.app.box.com/s/pt3e203fcjfg9z8r02si Assessment Hudson High School Portfolio Assessment for English Language Arts and Social Studies Publisher Website/Sample Designed to be a measure of student growth over time in high school ELA and social science courses. Student selects work samples to include and uploads them to electronic site. Includes guiding questions for students and scoring criteria. Scoring rubric for portfolio that can be adapted for use in all high school ELA and social science courses. Generalized grading criteria for a portfolio. Could be aligned to a number of CCOs, depending on specification of assignments.

53. Sample DDMs—Local Digital Portfolio Hudson, MA  Buy, Borrow, Build  Each sample DDM is evaluated Hudson’s Evaluation: Designed to be a measure of student growth over time in high school ELA and social science courses. Student selects work samples to include and uploads them to electronic site. Includes guiding questions for students and scoring criteria. Scoring rubric for portfolio that can be adapted for use in all high school ELA and social science courses. Generalized grading criteria for a portfolio. Could be aligned to a number of CCOs, depending on specification of assignments.  Many are standardized assessments

56. Other Tools: MA Model Curricula and Rubrics CEPAs ( Also, Delaware rubrics for specific text types) 123456 Topic development: The writing and artwork identify the habitat and provide details Little topic/idea development, organization, and/or details Little or no awareness of audience and/or task Limited or weak topic/idea development, organization, and/or details Limited awareness of audience and/or task Rudimentary topic/idea development and/or organization Basic supporting details Simplistic language Moderate topic/idea development and organization Adequate, relevant details Some variety in language Full topic/idea development Logical organization Strong details Appropriate use of language Rich topic/idea development Careful and/or subtle organization Effective/rich use of language Evidence and Content Accuracy: writing includes academic vocabulary and characteristics of the animal or habitat with details Little or no evidence is included and/or content is inaccurate Use of evidence and content is limited or weak Use of evidence and content is included but is basic and simplistic Use of evidence and accurate content is relevant and adequate Use of evidence and accurate content is logical and appropriate A sophisticated selection of and inclusion of evidence and accurate content contribute to an outstanding submission Artwork; identifies special characteristics of the animal or habitat, to an appropriate level of detail Artwork does not contribute to the content of the exhibit Artwork demonstrates a limited connection to the content (describing a habitat) Artwork is basically connected to the content and contributes to the overall understanding Artwork is connected to the content of the exhibit and contributes to its quality Artwork contributes to the overall content of the exhibit and provides details Artwork adds greatly to the content of exhibit providing new insights or understandings

57. Use what you have learned from reading “Daedalus and Icarus” by Ovid and “To a Friend Whose Work Has Come to Triumph” by Anne Sexton to write an essay that provides an analysis of how Sexton transforms Daedalus and Icarus. As a starting point, you may want to consider what is emphasized, absent, or different in the two texts, but feel free to develop your own focus for analysis. Develop your essay by providing textual evidence from both texts. Be sure to follow the conventions of standard English. Thus, both comprehension of the 2 texts and the author’s craft are being assessed along with the ability of the student to craft a clear argument with substantiation from two texts. 57

58. Some Examples of Direct Measures High School/Middle School  Mid-terms  Finals  Common Exams  Portfolios  Performances  Writing to text (Reading challenging passage and writing an argument or informational response) Elementary  DRA  Running Records  Benchmark exams  Common exams Frequently “untested” areas  PE—Fitness, Health Concepts  Art—Design, color,  Music—Self-critique of performance or critique of video of performance  SPED: Social-Emotional growth  Media/Technology— Research/Search

59. Protocols to Use Locally for Inter-Rater Reliability; Looking at Student Work  Developing effective rubrics for large-scale assessment  Developing exemplars  Calibrating scores  Looking at Student Work (LASW)  http://Nsfharmony.org/protocol/a_z.html http://Nsfharmony.org/protocol/a_z.html  Sample for Developing Rubrics from an assessment

60. Next Steps  Develop pilot assessments for SY 2014  Assess results; use results to help plan for full implementation in 2015  Develop a plan for all educators to have two DDMs: MCAS growth, purchased, or local  Develop a district process for assessing the quality of assessments (DESE Quality Tool or attachment on last two pages)  Develop an internal process for cut scores and determining low, average, and high growth of students  Track/organize information for June report: Educators/DDMs  Plan for 2015 administration for all educators: Tracking, scheduling, storing year 1 scores, storing year 2 scores 60

61. Table/Team Talk Unique areas and possible solutions. Build on what you have and what you now value

62. Exit Slips  Please give me your feedback on this session. It is so helpful.  What was helpful?  What might improve the presentation?  Additional questions you might have. Please include your email so that I can answer you personally

63. Sample DDMs Good, Not-so-good, and Problematical

64. Demonstrating Growth (when accuracy of computation may be a concern)

65. Essay Prompt from Text Read a primary source about Mohammed based on Muhammad’s Wife’s memories of her husband. Essay: Identify and describe Mohammed’s most admirable quality based on this excerpt. Select someone from your life who has this quality. Identify who they are and describe how they demonstrate this trait. What’s wrong with this prompt?

66. Science Open Response from Text Again, from a textbook, Is this acceptable?

67. Scoring Guides from Text  Lou Vee Air Car built to specs (50 points)  Propeller Spins Freely (60 points)  Distance car travels  1m 70  2m 80  3m 90  4m 100  Best distance (10,8,5)  Best car(10,8,5)  Best all time distance all classes (+5)  235 points total A scoring guide from a textbook for building a Lou Vee Air Car. Is it good enough to ensure inter-rater reliability?

68. Technology/Media Rubric A multi-criteria rubric for technology. What is good, bad, problematical?

69. PE Rubric in Progress. Grade 2 Overhand throw. Looks good?

70. Music: Teacher and Student Instructions

75. World Language Scoring Guide and Rubric

76. World Language Middle School

77. “Don’t let perfection get in the way of good.”

78. Team Planning Time Adapting present assessments Creating new assessments Writing to text for HS Developing and Piloting Assessments Alignment of Content Rigorous and appropriate expectations Approval of assessments Assessing Quality and Rigor Security Calibration of standards and of assessors Rubric quality Analysis of results: High-M-Low Growth Piloting 2 DDMs per educator JUNE REPORT Directions for teachers Directions for students Organizing for the actual assessments Storing, tracking the information 2015 Full Implementation Data storage Data Analysis L-M-H Growth Roster Verification Data team time Interpreting the results Student Impact