1. The Smarter Balanced Assessment Consortium Training 2-Day 1 The Digital Library Review Process Using the Quality Criteria

2. Introductions and Success Stories Introductions by State Share one success from Training 1 Share one lesson learned from Training 1 Slide 2

3. Digital Library Resources Slide 3 Commissioned Professional Learning Modules Resources for educators, students and families Frame Formative Assessment within a Balanced Assessment System Articulate the Formative Assessment Process Highlight Formative Assessment Practices and Tools Assessment Literacy Modules Commissioned Professional Learning Modules Instructional coaching for educators Instructional materials for students Demonstrate/support effective implementation of the formative assessment process Focus on key content and practice from the Common Core State Standards for Mathematics and English Language Arts Exemplar Instructional Modules High-quality vetted instructional resources and tools for educators High-quality vetted resources and tools for students and families Reflect and support the formative assessment process Reflect and support the Common Core State Standards for Mathematics and English Language Arts Create Professional Learning Communities Education Resources

4. Our Successes So Far Formative Assessment Advisory Panel Developed Quality Criteria State Leadership Teams and State Networks of Educators Established, Involving Around 2,000 K-16 Educators Across the Country SLT Training 1 and SNE Training 1 Completed Over 450 Resources Submitted to the Digital Library Blueprints for Exemplar Instructional Modules and Assessment Literacy Modules Blueprints Drafted Matrices for the Exemplar Instructional Modules for ELA and Math and for the Assessment Literacy Modules Drafted Digital Library Software Versions 1.0 and 1.1 Released

5. What’s Coming Next SLT Training 2 in November and December 2013 Software Version 1.15 in February 2014 Short-Term SLT Trainings 3 (March 2014), 4 (May 2014), and 5 (August 2014) Software Version 1.2 in April 2014 First set of 23 Exemplar Instructional Modules and Assessment Literacy Modules in March 2014 Second set of Modules in May 2014 and third set of Modules by September 2014 Long-Term Digital Library Open to Consortium Members in April 2014 Milestone

6. Training 1 Follow-up SLT Webpage Updates – Updated Cover Profile Guide – Tips for Identifying the License Presentation Role Overview in the Digital Library – What can I view as an SLT member? SNE Submissions – What should be submitted by the end of November? Slide 6

7. SNE Individual Report Review Slide 7

8. Training 2: The Digital Library Review Process Using the Quality Criteria for Professional Learning and Instructional Resources Slide 8

9. SLT Training 2 Learning Goal SLT members will understand the Quality Criteria ratings and how to apply them consistently. Success Criterion SLT members will provide evidence of inter-rater reliability among the state SLT members and with other state SLT members in cross-state teams. Slide 9

10. Formative Assessment Advisory Panel Convened the Formative Assessment Advisory Panel to develop the Quality Criteria during three two-day meetings Page 10 Meeting 1 April 17 – 18 Meeting 1 April 17 – 18 Meeting 2 May 8 – 9 Meeting 2 May 8 – 9 Meeting 3 May 22 – 23 Meeting 3 May 22 – 23 Brainstormed initial Quality Criteria Determined structure of Quality Criteria Developed comprehensive list of potential criteria Discussed merits of checklist vs. rubric-based approach Tested criteria using sample resources Refined criteria Synthesized feedback Developed first draft of Quality Criteria Developed 2 nd draft; received feedback from panelists; developed present draft Slide 10

11. Members of the Formative Assessment Advisory Panel 1)Dr. Lynne Anderson-Inman (University of Oregon) 2)Dr. Robert Calfee (Stanford University, UC Riverside) 3)Dr. Bridget Dalton (University of Colorado) 4)Dr. Diane Heacox (St. Catherine University) 5)Dr. Joan Herman (UCLA – CRESST) 6)Dr. John Hill (Purdue University) 7)Dr. Yvette Jackson (National Urban Alliance for Effective Education) 8)Dr. Henry Kepner (University of Wisconsin - Milwaukee) 9)Dr. Katherine McKnight (National Louis University) 10)Valerie L. Mills (National Council of Supervisors of Mathematics) 11)Dr. James Popham (UCLA) 12)Dr. Lucinda Soltero-Gonzalez (University of Colorado-Boulder) Page 11 Twelve experts in the CCSS for ELA, the CCSS for Mathematics, the formative assessment process, adult learning, online professional learning, diverse learners, and urban and rural education comprised the Formative Assessment Advisory Panel. Slide 11

12. Parts of the Quality Criteria Cover Profile Gatekeeping Criteria Quality Criteria for Professional Learning Resources and Quality Criteria for Instructional Resources Governance Criteria Slide 12 N/A

13. Purposes of the Quality Criteria Page 13 Ensure that all resources in the Digital Library are of the highest quality, regardless of source Ensure that all resources reflect the Smarter Balanced vision of effective formative assessment practices Provide consistency in the review process for all resources Ensure that the resources in the Digital Library reflect the intent of the CCSS

14. Resource Posting Work Flow Page 14 Step 1: Resource Submitted Step 3: Quality Criteria Applied Step 4: Decision Step 2: Gatekeeping Criteria Applied SNE 1 SNE 2 SNE 3 SNE 1 Posted Sent to SLT Returned to Submitter Cover Profile

15. A Close Reading of the Quality Criteria Slide 15 Individually Read and Annotate the Quality Criteria – Pages 2-5: Quality Criteria for Professional Learning Resources – Pages 6-9: Quality Criteria for Instructional Resources Think About and Discuss with Your Table Group – What are the differences between the two sets of criteria? – How might these criteria help focus the review process? – Where will your state need more clarity and information for Training 2? Whole Group Discussion

16. Additional Resources SLT Webpage: http://www.amplify.com/smarter- balanced-slt http://www.amplify.com/smarter- balanced-slt – Quality Criteria FAQ – UDL – Professional Learning Slide 16

17. Break and Login Distribution Slide 17

18. Accessing the Digital Library in the Training Environment Page 18 1. Open your Internet browser (Google Chrome typically works best, though Safari, Firefox, and Internet Explorer 9+ are also options). 2. Type the web address: http://training.smarterbalancedlibrary.org http://training.smarterbalancedlibrary.org 3. Type your assigned email address: slt#@smarterbalancedlibrary.org 4. Type the password: password 5. Click “Log into the Digital Library”

19. Common IR Review Read the Cover Profile Read the PDF Materials Watch the Video (whole group) Individually Rate the Resource Discuss Ratings with Your State Team Determine Ratings, Comments, Posting Recommendation, Rationale as a Team Slide 19

20. Common IR Review Debrief On your poster paper: – Resource Title – Criteria Ratings – Posting Rec – Rationale Discuss with the whole group: – Talk us through your team’s process when discussing and deciding on the ratings. Slide 20

21. Reflection 1.What criteria will need more discussion/clarification with your SNE members? 2.What are the areas of confusion about the Quality Criteria for Instructional Resources? 3.What recommendations do you have for technology improvements? Slide 21

22. Cross-State Teams Review the Cover Profile Review Materials Vet the Resource Against the Quality Criteria Slide 22

23. Slide 23 On your poster paper: – Resource Title – Criteria Ratings – Posting Rec – Rationale Discuss with the whole group: What were notable differences to apply to SNE trainings? Cross-State Debrief

24. SLT Training Cycles SNE Payment Last Day of Training Cycle PCG Sends “SNE Member Payment Spreadsheet” to SLT Lead SLT Returns Spreadsheet to PCG SNE Payment Due Training 1 October 30, 2013 November 18, 2013 December 16, 2013 November 26, 2013 December 23, 2013 December 20, 2013 January 24, 2014 Training 2 December 31, 2013 January 20, 2014 February 24, 2014 January 28, 2014 March 4, 2014 February 21, 2014 March 28, 2014 Training 3 April 30, 2014 May 19, 2014May 27, 2014June 20, 2014 Training 4 June 30, 2014 July 24, 2014July 19, 2014August 22, 2014 Training 5 Sept. 30, 2014 October 20, 2014October 28, 2014November 21, 2014 Slide 24

25. The Smarter Balanced Assessment Consortium Training 2-Day 2 The Digital Library Review Process Using the Quality Criteria

26. Day 1 Recap & Your Feedback You all are amazing!! Thank you for your honest conversations and collaboration. We want to address feedback from Tickets Out the Door. Slide 26

27. Follow up from Tickets out the Door Resource Review Queue – Content Expertise Selection in Profile (demo) – Ability to tag ELA Literacy Standards for Social Studies, Science, Technical Subjects Use of Training Environment and Resources – SLT Webpage and Jump Drive Descriptive Feedback – SLT Arbitration Process Exercise Geographic Location in the Cover Profile is an optional field Slide 27

28. Questions 1. How do SLTs best instruct SNEs to use the listed criteria to make a holistic judgment? 2. Can SNEs access the training site? When? Who provides the access information? How can we provide resources to practice? 3. If one area shows no evidence, is that grounds for not recommending a resource? Slide 28

29. Questions 4.Questions about Quality Criteria A.Instructional--#1 i.SBAC Claims, Targets—if this criteria remains, we need to do more training with SLTs, these do not apply as easily to K-2 or to Foundational Skills at all. Could this tag be made after a resource is accepted? ii.Does precursor/successor fit here? iii.Are there criteria from #3 that fit better in #1. B.Instructional--#6-Can an NA rating be included for this section of criteria for resources that may not benefit tremendously from incorporating technology? 5.What algorithms will result in the SLT receiving the resource for arbitration? Slide 29

30. Copyright www.amplify.com/smarter-balanced-slt Updated Copyright Functionality in the Digital Library Copyright Clearance Form Updated Copyright Information in Cover Profile Guide Tips for Identifying the License Presentation Slide 30

31. State share of ideas on how we can help SNE members feel more connected and maintain morale? Slide 31 Send Thank You Cards Highlight Successes Recognizing SNEs with Principals/Providing Certificates Reassurance of completion and payment Recognizing SNE First Approved Resource Badges

32. Common Professional Learning Resource Review Login to the training environment: – training.smarterbalancedlibrary.org Select Common Professional Learning Resource: Formative Assessment Learning Strategies Click and Review: About This Resource Slide 32

33. Common Professional Learning Resource Review 20 minutes – Review About this Resource – Watch Video (whole group) 20 minutes – Review pdf files – Individually vet the resource using the Professional Learning Quality Criteria 15 minutes – Discuss and compare answers with your team 5 minutes – Create your poster Slide 33

34. Debrief What were the notable differences in reviewing a Instructional Resource versus a Professional Learning Resource? Additional questions, comments, concerns? Tips for SNE training? Slide 34

35. Posting Decisions When does an SLT have to make a resource posting decision? Slide 35

36. SNE Recommendation Options Slide 36 Do not recommend Recommend with revisions 1. Recommend with distinction 2. Recommend 3. Recommend with revisions 4. Do not recommend

37. 20 SNE Recommendation Combinations Page 37 DecisionSNE Recommendation Software or SLT Decision? Post with Distinction Recommend with Distinction Software Recommend with Distinction RecommendSoftware Post Recommend with DistinctionRecommend Software Recommend Software SLT Member Judgment Recommend with Distinction Recommend with RevisionsSLT Recommend with Distinction Do Not RecommendSLT Recommend with DistinctionRecommendRecommend with RevisionsSLT Recommend with DistinctionRecommendDo Not RecommendSLT Recommend Recommend with RevisionsSLT Recommend Do Not RecommendSLT Not Post; Return to submitter Recommend with DistinctionRecommend with Revisions Software Recommend with DistinctionRecommend with RevisionsDo Not RecommendSoftware Recommend with DistinctionDo Not Recommend Software RecommendRecommend with Revisions Software RecommendRecommend with RevisionsDo Not RecommendSoftware RecommendDo Not Recommend Software Recommend with Revisions Software Recommend with Revisions Do Not RecommendSoftware Recommend with RevisionsDo Not Recommend Software Do Not Recommend Software Please note: The decision depends on the combination of recommendations; the order of the recommendations does not matter.

38. Recommendation Combinations- SLT Member Decision Page 38 An SLT member will make a posting decision given the following recommendation combinations: SNE Recommendation Recommend with Distinction Recommend with Revisions Recommend with Distinction Do Not Recommend Recommend with Distinction Recommend Recommend with Revisions Recommend with Distinction RecommendDo Not Recommend Recommend Recommend with Revisions Recommend Do Not Recommend

39. SNE Recommendation Recommend with Distinction Recommend with Revisions Recommend with Distinction Do Not Recommend Recommend with Distinction Recommend Recommend with Revisions Recommend with Distinction RecommendDo Not Recommend Recommend Recommend with Revisions Recommend Do Not Recommend SLT Decision Scenario Page 39

40. SLT Member Quality Criteria Review Log into the “real” Digital Library Software Demo Resource Review tab Slide 40

41. Training 2 SLT Arbitration Exercise Materials Resource Cover Profile Three SNE Quality Criteria Reviews – Recommend with Distinction – Recommend – Recommend with Revisions SLT Arbitration Form Page 41

42. When does an SLT have to make a resource posting decision? Find someone you haven’t spoken to from a different table Share your table’s recommendation and rationale. Whole Group Share – What did you hear that was interesting? – Any new thoughts? Slide 42

43. State Team Planning Share any good ideas on the collaboration poster. Slide 43

44. Evaluation https://www.surveymonkey.com/s/SLTTraining 2 Email with the link by 12:00pm Slide 44