ESSENTIAL STRATEGIES TO STRENGTHEN YOUR PROGRAM THROUGH COMMON ASSESSMENTS Dawn Carney, Brookline Public Schools Tim Eagan, Wellesley Public Schools Tiesa Graf, South Hadley Public Schools MaFLA, October 24, 2014

Agenda  Objectives  A message from the DESE (Craig Waterman, Assessment Coordinator)  A few reminders!  Warm up  Developing and strengthening common assessments  Using protocols to analyze student work  Using assessment data to identify feedback for student growth  Learning how the process can work for you – takeaways and next steps

Objectives  This workshop will help you to:  Develop and strengthen DDMs (or common assessments)  Use protocols to examine student work  Use assessment data to identify feedback for student growth  Identify professional learning to incorporate into SMART goals (and evaluation process)  Learn how this process can strengthen your program and work for you!

A message from DESE Craig Waterman Assessment Coordinator Department of Elementary & Secondary Education

A few reminders! Where we are (MaFLA member survey) DDM = common assessments = valuable! Design an action plan Action plan should inform department vision How can these tools help my department with SMART goals/eval process? Essential to know ACTFL Standards, Guidelines and Descriptors

Warm up  Review the workshop objectives  Write down a goal for yourself ~ In terms of where you are with the DDM/common assessment process… What do you need from today and what do you need for the future?  Share with a partner  3 minutes

Developing & Strengthening Common Assessments Improving Teaching & Learning

Assessment is an ongoing process of setting clear goals for student learning and measuring progress toward those goals.

Performance vs Proficiency  Reliant on Instruction  Practiced  Controlled Environment  Separate from Instruction  Unrehearsed  Across a wide range of topics and settings Performance Proficiency ACTFL Performance Descriptors for Language Learners © ACTFL, Inc., 2012

ACTFL Performance Descriptors  Roadmap for teaching and learning,  Help teachers create performance tasks targeted to the appropriate performance range,  Challenge learners to use strategies from the next higher range. ACTFL Performance Descriptors for Language Learners © ACTFL, Inc., 2012

Key Concepts Principles of Assessment

Practicality Resources FundingMaterials# of Staff # of Students Knowledge of Students & of Assessment TimeOther

Impact  Washback +  Washback –  Stakeholders  Students  Parents  Administrators  Teachers  Other

Validity  Does the assessment measure what it is supposed to measure?  Is it used for its intended purposes on the intended population?

Four questions to ask after looking at selected assessments and student work samples. Building a Validity Argument

Validity Argument Question 1:  Does the assessment measure what it is supposed to measure/what I thought it would measure?

Validity Argument Question 2:  Does the assessment reflect real- life language uses? Is it authentic?

Validity Argument Question 3:  Do the students and the teachers take the assessment seriously?

Validity Argument Question 4:  Is the assessment consistent with instruction —the what and the how?

Harvard Data Wise:  “We used to think validity was a property of a test. Modern validity takes a different approach: Inferences based on test scores cannot be perfectly valid, but they can be valid enough to be useful.”

What impact does this validity question have on your action plan? Where is your department on this?  Do your departmental assessments reflect real-life language uses? Are they authentic, and therefore valid?  2 minutes

Developing DDMs: “Best practice…wide range of item difficulties and rigor so that examinees of all abilities can show some capability...” Technical Guide A: Considerations Regarding District-Determined Measures

Presentational Writing Prompt Example Modified Pretest/Post-test Model DDM Examples: Wellesley High School

Modern Languages, Year 3:  A friend of yours has a problem with someone. Describe the problem, explore various solutions, and then advise your friend on a specific course of action. Use all appropriate verb tenses learned. What other information do you need in order to apply the Validity Argument to this prompt? 1 minute.

Performance Targets: Year 3 Students (Typically Sophomores)  Intermediate-Low/Mid Proficiency ACTFL Performance Descriptors for Language Learners © ACTFL, Inc., 2012

Performance Targets: Year 4 Students (Typically Juniors)  Intermediate-Low/Mid Proficiency ACTFL Performance Descriptors for Language Learners © ACTFL, Inc., 2012

Narrowing the Focus Developing Scoring Guides

WPS Presentational Speaking & Writing Task Rubric:

One domain only:  Language Control  “Describes the level of control the learner has over certain language features or strategies to produce or understand language.”  ACTFL Performance Descriptors for Language Learners © ACTFL, Inc., 2012

Why one narrow domain? Practicality:  Build assessment literacy & capacity to talk about student work.  Stay small and focused.  Learn to be Specific and Objective.  Define & refine our performance targets based on data.

Take 2 minutes and discuss:  How can you use the principle of Practicality to support you in building an action plan? For example…

One participant to share with group.

USING PROTOCOLS TO ANALYZE STUDENT WORK/LOOKING FOR GROWTH Consistency of the results of an assessment

Be reliable!  Items (questions/prompts should assess the same skill/knowledge)  Administration (consistency)  Rating (two different scorers should get the same result/rating/score)

Keep it simple! Select a core course objective – goals specific to your curriculum (example) Determine the communicative mode to assess (*hint – presentational!) Develop the prompt Select the domain from your rubric for area of growth Items – developing DDMs

DDM Template from South Hadley inspired by Tim!

Where are you?  Have you developed common assessments?  Do your common assessments focus on core course objectives?  Have you determined departmental rubrics?  Have you determined domains to focus on to determine growth?  SMART goal?  2 minutes to discuss

Departmental decisions about prompt for each level If listening, how many times? If speaking, prompt in TL or English? Prompt provided ahead of time? If writing, time/length expectations etc. Administration reliability: protocol

Example from South Hadley

Scoring Sheet (SHHS)

Where are you?

How do you look at student work to determine quality/growth? Rating – examining student work

Effective Analysis requires trust and commitment  How do you build trust and encourage teamwork in your department?

Developing norms and ACE Habits of Mind  Clear meeting objectives  Develop norms for working together as a group  Focus on evidence and observations – not on judgments

Reliable ratings: Know your rubric **Develop/Select your rubric

Select the domain for growth  Language control, language function  Vocabulary use, text type, level of discourse  Etc.

Determine evidence: rating  What does quality look like? What are you looking for in a sample?  How do we work together to calibrate?

Develop a protocol for analysis

Practice!  Calibrate – select student samples for each department meeting/PD meeting time  Split by language – mix levels  Rate samples and discuss

How to use for SMART goals  Item development  Assessment administration  Ratings and protocols

Where are you?  Item development  Assessment administration  Ratings and protocols  Impact on action plan and/or SMART goals?  What’s your next step?  One response from group to share

One participant to share with group.

Using Assessment Data to Identify Feedback for Student Growth Connecting learning and teaching

What should my students be able to do? Performance targets

What should students be able to do? Proficiency targets

Novice Respond to simple questions Use: isolated words lists of words memorized phrases some personalized re-combinations of words or phrases Ask memorized, formulaic questions Satisfy immediate needs WORD level

Intermediate “Conversation” partner in simple, direct conversations Describe and narrate Ask and answer simple questions Handle survival language “Create” with language SENTENCE level

Advanced  Participate actively in conversations, formal and informal settings  Describe & narrate in major time frames, good control  Use variety of communicative devices to deal effectively with unanticipated complications  Sustain communication with paragraph length connected discourse, accuracy and confidence  Satisfy demands of work/school situations  PARAGRAPH level

Proficiency Families All three levels perform intermediate mid tasks: Novice HighIntermediate LowIntermediate Mid most of the timeall of the timewith ease & confidence All three levels perform advanced mid tasks: Intermediate HighAdvanced LowAdvanced Mid: most of the timeall of the timewith ease & confidence

PSB Proficiency Targets

Take 2 minutes and discuss with your partner.  Does your department have a shared understanding of performance and proficiency?  Do you have a shared understanding of the performance targets for your program?

What do the assessment results tell me about my students? How do I help students progress along the proficiency continuum? Feedback

Individual feedback  directs students to intended learning  points out what student is doing well; offers specific information to guide improvement  given only when students have at least some understanding  doesn’t do the thinking for students  limits corrective information to what the student can act on “How Am I Doing?” by Jan Chappuis in Educational Leadership, September 2012 (Vol. 70, #1, p ),

Internal assessment: rubric as feedback

Feedback: self-assessment & goal setting  Can-do statements  Student goals  Teacher goals

Take 2 minutes and discuss with your partner.  How might the use of common assessments & analysis of student work inform instruction and curriculum in your department?

External Assessment

External Assessment: STAMP 4S

External assessment: feedback for growth

External Assessment: AAPPL

Common assessments, analysis and protocols, feedback for growth, oh my! Application: Making it work for you

Takeaways  Reflect on the initial goal that you developed during the warm up and consider these questions:  What are your next steps?  What’s your action plan?  What do you want/need to learn more about?  SUMMER ACADEMY reminder and handout