1. 1 Teacher Performance Assessment Consortium November 10, 2010 Andrea Whittaker and Nicole Merino Stanford University November 10, 2010 Andrea Whittaker and Nicole Merino Stanford University 1

2. © Stanford Center for Assessment, Learning, and Equity, 2010 Three Sponsoring Organizations AACTE – overall project management, communication with programs Stanford University – assessment development and technical support Council of Chief State School Officers – policy development and support, communication with state education agencies 2

3. © Stanford Center for Assessment, Learning, and Equity, 2010 Sign off from state education agencies Sign off from state education agencies Participation of preparation programs Participation of preparation programs Up to 3 funded through grant Up to 3 funded through grant Alignment and policy studies Alignment and policy studies Adaptation of TPA to address future state program and policy needs Adaptation of TPA to address future state program and policy needs 3 TPAC State Participation Requirements

4. 4

5. © Stanford Center for Assessment, Learning, and Equity, 2010 Lineage of TPAC National Board for Professional Teaching Standards (NBPTS) portfolio assessments – accomplished teachers National Board for Professional Teaching Standards (NBPTS) portfolio assessments – accomplished teachers Connecticut BEST assessment – teachers at end of induction Connecticut BEST assessment – teachers at end of induction Performance Assessment for California Teachers (PACT) – pre-service teachers Performance Assessment for California Teachers (PACT) – pre-service teachers source: SCALE @ Stanford University 5

6. © Stanford Center for Assessment, Learning, and Equity, 2010 APPLE Criteria The National Board for Professional Teaching Standards Administratively feasible Publicly credible Professionally acceptable Legally defensible Economically affordable 6

7. © Stanford Center for Assessment, Learning, and Equity, 2010 7 Discipline Specific Discipline Specific Integrated Assessment Integrated Assessment Student Centered Student Centered Analytic Feedback and Support Analytic Feedback and Support Represents a Complex View of Teaching: Multiple Measures Represents a Complex View of Teaching: Multiple Measures 7 Design Principles: Educative Assessment

8. 888 The TPAC Assessment System Assessments Embedded in Local Programs — examples — Assessments Embedded in Local Programs — examples — Observation/Supervisory Evaluation & Feedback Child Case Studies Analyses of Student Learning Curriculum /Teaching Analyses The Capstone Assessment TPAC Assessment Demonstrates : Planning Instruction Planning Instruction and Assessment and Assessment Engaging Students Engaging Students and Supporting and Supporting Learning Learning Assessing Student Assessing Student Learning Learning ‣ Reflecting ‣ Academic Language TPAC Assessment Demonstrates : Planning Instruction Planning Instruction and Assessment and Assessment Engaging Students Engaging Students and Supporting and Supporting Learning Learning Assessing Student Assessing Student Learning Learning ‣ Reflecting ‣ Academic Language

9. © Stanford Center for Assessment, Learning, and Equity, 2010 9 TPAC Assessment Records of Practice* Instructional and Social Context Learning Segment of 3 to 5 Hours/Days Planning Lesson Plans Lesson Plans Handouts, overheads, student work Handouts, overheads, student work Lesson Commentary Lesson CommentaryEngagement Video clip(s) Video clip(s) Teaching Commentary Teaching CommentaryAssessment Analysis of Whole Class Assessment Analysis of Whole Class Assessment Analysis of learning of 2 students Analysis of learning of 2 students Feedback to 2 students Feedback to 2 students Next steps in instruction Next steps in instruction Evidence of Academic Language Development Evidence of Academic Language Development Daily Reflections and Reflective Commentaries Daily Reflections and Reflective Commentaries 9

10. © Stanford Center for Assessment, Learning, and Equity, 2010 10 Guiding Questions and Analytic Rubrics PLANNING PLANNING ‣ Planning Focused, Sequenced Instruction ‣ Using Knowledge of Students to Inform Teaching ‣ Planning Assessments to Monitor and Support Student Learning ENGAGING ENGAGING ‣ Engaging Students in Learning ‣ Deepening Student Learning During Instruction 10 ASSESSMENT ASSESSMENT ‣ Analyzing Student Work ‣ Using Assessment to Inform Instruction ‣ Using Feedback to Guide Further Learning REFLECTION REFLECTION ‣ Monitoring Student Progress and Adjusting Instruction ACADEMIC LANGUAGE ACADEMIC LANGUAGE ‣ Understanding Language Demands and Resources ‣ Developing Students’ Academic Language Repertoire 10

11. © Stanford Center for Assessment, Learning, and Equity, 2010 Rubric Levels Is the candidate ready for independent teaching (i.e., to be the teacher of record)? Is the candidate ready for independent teaching (i.e., to be the teacher of record)? Rubric Levels Rubric Levels ‣ Level 1 – Some skill but needs more practice to be teacher-of-record ‣ Level 2 – Acceptable level to begin teaching ‣ Level 3 – Solid foundation of knowledge and skills ‣ Level 4 – Stellar candidate, in the top 5% of candidates

12. © Stanford Center for Assessment, Learning, and Equity, 2010 Questions and Answers ??? 12

13. © Stanford Center for Assessment, Learning, and Equity, 2010 13 PACT Scores Inter-rater Reliability Level of Agreement Percent Exact Match 46% ± 1 point 34% ± 2 points or greater 10% Sample Size · 2,580 Spearman-Brown Reliability Estimate · 0.88 13

14. © Stanford Center for Assessment, Learning, and Equity, 2010 14 Content validity Content validity ‣ Development teams, Program directors, Program faculty, & Leadership team ‣ TPE alignment study Concurrent validity Concurrent validity ‣ Evaluation of score validity ‣ Decision Consistency · Holistic vs. Analytic ratings Bias and fairness review Bias and fairness review PACT Validity Studies 14 Construct validity Construct validity ‣ Factor Analysis (2002-03 Pilot Year): Reflection & AssessmentReflection & Assessment InstructionInstruction PlanningPlanning Predictive Validity (Carnegie/CT Study) Predictive Validity (Carnegie/CT Study) Consequential Validity Consequential Validity

15. © Stanford Center for Assessment, Learning, and Equity, 2010 Research Plans Value-added analysis Observations and interviews of candidates completing assessment + first year of teaching Surveys and interviews with faculty and program administrators on use of assessment data 15

16. © Stanford Center for Assessment, Learning, and Equity, 2010 Assessment Development 2009-10 Small-scale tryout tasks & feedback from users 2010-11 Development of pilot prototypes based on feedback and piloting in at least 2 credential areas per institution. User feedback to guide revisions 2011-12 National field test of prototypes, producing a technical report with reliability and validity studies and a bias and sensitivity review. National standard setting. 2012-13 Adoption of validated assessment 16

17. © Stanford Center for Assessment, Learning, and Equity, 2010 Overview of Tasks and Rubrics Artifacts and Commentaries Artifacts and Commentaries ‣ What looks familiar? Level Three Rubric Descriptors Level Three Rubric Descriptors ‣ What gets assessed? Tips for implementation Tips for implementation 17

18. © Stanford Center for Assessment, Learning, and Equity, 2010 Handbook Overview What looks like something we already do? What looks new or different? What will our candidates be good at now? What might be a struggle? 18

19. © Stanford Center for Assessment, Learning, and Equity, 2010 Context Matters Candidates describe what they know about their students as learners: Candidates describe what they know about their students as learners: ‣ Academic development ‣ Academic Language development ‣ Social-emotional development ‣ Family, community and cultural assets ‣ Special needs Candidates use information about students to plan appropriate instruction and assessment. Candidates use information about students to plan appropriate instruction and assessment. Scoring rubrics examine extent to which needs of students are addressed Scoring rubrics examine extent to which needs of students are addressed 19

20. © Stanford Center for Assessment, Learning, and Equity, 2010 20 Guiding Questions and Analytic Rubrics PLANNING PLANNING ‣ Planning Focused, Sequenced Instruction ‣ Using Knowledge of Students to Inform Teaching ‣ Planning Assessments to Monitor and Support Student Learning ENGAGING ENGAGING ‣ Engaging Students in Learning ‣ Deepening Student Learning During Instruction 20 ASSESSMENT ASSESSMENT ‣ Analyzing Student Work ‣ Using Assessment to Inform Instruction ‣ Using Feedback to Guide Further Learning REFLECTION REFLECTION ‣ Monitoring Student Progress and Adjusting Instruction ACADEMIC LANGUAGE ACADEMIC LANGUAGE ‣ Understanding Language Demands and Resources ‣ Developing Students’ Academic Language Repertoire 20

21. © Stanford Center for Assessment, Learning, and Equity, 2010 EM1: Planning Focused, Sequenced Instruction Standards/objectives, learning tasks, and assessments are clearly aligned to a big idea or essential question. The learning tasks and assessments represent differing depths of understanding Standards/objectives, learning tasks, and assessments are clearly aligned to a big idea or essential question. The learning tasks and assessments represent differing depths of understanding Candidate plans how to make clear connections among mathematical facts, computations/procedures, concepts, and reasoning. Candidate plans how to make clear connections among mathematical facts, computations/procedures, concepts, and reasoning. Learning tasks build on each other to promote an understanding of the designated mathematical concepts, computations/procedures, and reasoning skills. Learning tasks (or their adaptation) are justified by explaining their appropriateness for students with references to relevant research and/or theory. Learning tasks build on each other to promote an understanding of the designated mathematical concepts, computations/procedures, and reasoning skills. Learning tasks (or their adaptation) are justified by explaining their appropriateness for students with references to relevant research and/or theory. 21

22. © Stanford Center for Assessment, Learning, and Equity, 2010 EM2: Using Knowledge of Students to Inform Teaching Learning tasks and materials draw upon students’ academic and social/emotional development, including strengths, as well as experiences and interests to help students reach the learning objectives. Learning tasks and materials draw upon students’ academic and social/emotional development, including strengths, as well as experiences and interests to help students reach the learning objectives. Planned support consists of strategically selected or modified tasks/materials and/or scaffolding of instruction that is closely tied to specific learning objectives. It is appropriate for specific individuals or subgroups. Planned support consists of strategically selected or modified tasks/materials and/or scaffolding of instruction that is closely tied to specific learning objectives. It is appropriate for specific individuals or subgroups. 22

23. © Stanford Center for Assessment, Learning, and Equity, 2010 EM3: Planning Assessments that Monitor and Support Student Learning The set of assessments will provide evidence of student learning relative to the standards/objectives for each lesson. At least one lesson’s assessments provide evidence of student learning that extends beyond the formulaic application of computations or procedures. The set of assessments will provide evidence of student learning relative to the standards/objectives for each lesson. At least one lesson’s assessments provide evidence of student learning that extends beyond the formulaic application of computations or procedures. Assessments are aligned to clearly defined benchmarks or criteria for student performance. Assessments are aligned to clearly defined benchmarks or criteria for student performance. Assessments are modified or adapted to be appropriate for students having difficulty demonstrating their learning. Assessments are modified or adapted to be appropriate for students having difficulty demonstrating their learning. 23

24. © Stanford Center for Assessment, Learning, and Equity, 2010 EM4: Engaging Students in Learning Strategies for intellectual engagement seen in the clip(s) offer structured opportunities for students to develop their own understanding of mathematical concepts through discourse. These strategies reflect attention to students’ academic or language development, social/emotional development, and/or cultural and lived experiences. Strategies for intellectual engagement seen in the clip(s) offer structured opportunities for students to develop their own understanding of mathematical concepts through discourse. These strategies reflect attention to students’ academic or language development, social/emotional development, and/or cultural and lived experiences. Candidate identifies successful and unsuccessful teaching practices. The proposed improvements are reasonable and address the learning of a subgroup or individual students. Candidate identifies successful and unsuccessful teaching practices. The proposed improvements are reasonable and address the learning of a subgroup or individual students. 24

25. © Stanford Center for Assessment, Learning, and Equity, 2010 EM5: Deepening Student Learning During Instruction Candidates and/or other students build on what students are saying and/or doing, using reasoning to improve understanding of mathematical concepts. Candidates and/or other students build on what students are saying and/or doing, using reasoning to improve understanding of mathematical concepts. Candidate and/or other students prompt students to make connections between and among mathematical concepts and representations of content. Candidate and/or other students prompt students to make connections between and among mathematical concepts and representations of content. 25

26. © Stanford Center for Assessment, Learning, and Equity, 2010 EM6: Analyzing Student Work Criteria are well-defined and reflect the depth of understanding stated in of the indicated standards/objectives from the learning segment. Criteria are well-defined and reflect the depth of understanding stated in of the indicated standards/objectives from the learning segment. The analysis focuses on patterns of student errors, skills, and understandings in relation to standards and learning objectives. The analysis uses these patterns to understand student thinking. The analysis focuses on patterns of student errors, skills, and understandings in relation to standards and learning objectives. The analysis uses these patterns to understand student thinking. The analysis is supported by work samples and the summary of learning. Specific patterns are identified for individuals or subgroup(s) in addition to the whole class. The analysis is supported by work samples and the summary of learning. Specific patterns are identified for individuals or subgroup(s) in addition to the whole class. 26

27. © Stanford Center for Assessment, Learning, and Equity, 2010 EM7: Using Assessment to Inform Instruction Next steps follow from an accurate analysis of student learning and aim at improving student understanding of important features of the standards/learning objectives. Next steps follow from an accurate analysis of student learning and aim at improving student understanding of important features of the standards/learning objectives. Next steps focus on improving student performance through targeted support to individuals and groups to address specific identified needs. Next steps focus on improving student performance through targeted support to individuals and groups to address specific identified needs. Next steps are based on whole class patterns of performance and some patterns for individuals and/or subgroups and are described in sufficient detail to understand them. Next steps are based on whole class patterns of performance and some patterns for individuals and/or subgroups and are described in sufficient detail to understand them. 27

28. © Stanford Center for Assessment, Learning, and Equity, 2010 EM8: Using Feedback to Guide Learning Specific and accurate feedback helps the student understand what s/he did well, and provides guidance for improvement. Specific and accurate feedback helps the student understand what s/he did well, and provides guidance for improvement. Candidate describes how students will use feedback to improve their work or their understanding. Candidate describes how students will use feedback to improve their work or their understanding. 28

29. © Stanford Center for Assessment, Learning, and Equity, 2010 EM9: Monitoring Student Progress and Adjusting Instruction Daily reflections identify what students could or could not do within each lesson and consider the implications for meeting the standards/objectives at the end of the learning segment. Daily reflections identify what students could or could not do within each lesson and consider the implications for meeting the standards/objectives at the end of the learning segment. Adjustments to instruction are appropriate and focused on addressing some individual and collective learning needs. Adjustments to instruction are appropriate and focused on addressing some individual and collective learning needs. 29

30. © Stanford Center for Assessment, Learning, and Equity, 2010 EM10: Understanding Language Demands and Resources Candidate describes academic language strengths and needs of students at different levels of academic language proficiency. Candidate describes academic language strengths and needs of students at different levels of academic language proficiency. The language genre(s) discussed are clearly related to the academic purpose of the learning segment and language demands are identified. One or more linguistic features and/or textual resources of the genre are explicitly identified. The language genre(s) discussed are clearly related to the academic purpose of the learning segment and language demands are identified. One or more linguistic features and/or textual resources of the genre are explicitly identified. Candidate identifies essential vocabulary for students to actively engage in specific language tasks. Candidate identifies essential vocabulary for students to actively engage in specific language tasks. 30

31. © Stanford Center for Assessment, Learning, and Equity, 2010 EM11: Developing Students’ Academic Language Repertoire The candidate’s use of scaffolding or other support provides access to core content while also providing explicit models, opportunities for practice, and feedback for students to develop further language proficiency for selected genres and key linguistic features. The candidate’s use of scaffolding or other support provides access to core content while also providing explicit models, opportunities for practice, and feedback for students to develop further language proficiency for selected genres and key linguistic features. Candidate articulates why the instructional strategies chosen are likely to support specific aspects of students’ language development for different levels of language proficiency. Candidate articulates why the instructional strategies chosen are likely to support specific aspects of students’ language development for different levels of language proficiency. 31

32. © Stanford Center for Assessment, Learning, and Equity, 2010 Potentially New Competencies Academic language development Interpreting student work and identifying trends across the class Designing next steps in instruction based on assessment results Providing feedback to students 32

33. © Stanford Center for Assessment, Learning, and Equity, 2010 Issues to Think about for Spring 2011 Pilot Match with curriculum Which faculty? Which credential areas? Candidate support Timeline, from introducing assessment through scoring Preparation for videotaping Recruiting scorers Consideration of results 33

34. © Stanford Center for Assessment, Learning, and Equity, 2010 What did successful PACT programs do? Broad and collegial involvement in decision making Inquiry and program improvement rather than compliance Affirm and maintain program values and identity Broad involvement in assessment activities Regular practices and protocols for data analysis Learned from individual “case” studies 34

35. © Stanford Center for Assessment, Learning, and Equity, 2010 Questions and Answers ??? 35

36. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language Overview Three “F” Words… 36

37. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language Academic language development is making the language explicit to expand students’ control over language and improve their language choices according to the purpose/FUNCTION and audience for the message. Academic language also offers structures/FORMS for developing as well as expressing explanations, evaluations, and analyses. Developing students’ FLUENCY in academic language forms and functions provides access to the “language of school” and academic success 37

38. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language The purposes of Academic Language are to clearly and explicitly define, classify, analyze, explain, argue, interpret and evaluate ideas for distant audiences. 38

39. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language The purposes of Academic Language are to clearly and explicitly define, classify, analyze, explain, argue, interpret and evaluate ideas for distant audiences. = FUNCTION 39

40. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language Language for academic purposes differs greatly from every day purposes Language for academic purposes differs greatly from every day purposes These differences include These differences include ‣ a better-defined system of FORMS (genres) with explicit expectations about how texts are organized to achieve academic purposes ‣ precisely-defined vocabulary to express abstract concepts and complex ideas ‣ more complex grammar in order to pack more information into each sentence ‣ a greater variety of conjunctions and connective words and phrases to create coherence among multiple ideas ‣ formatting conventions, graphics and organizational titles and headings to guide understanding of texts 40

41. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language Academic Language also includes instructional language needed to participate in learning and assessment tasks, including discussing ideas and asking questions, summarizing instructional and disciplinary texts, following and giving instructions, listening to a mini-lesson, explaining thinking aloud, giving reasons for a point of view, writing essays to display knowledge on tests. 41

42. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language in TPAC Vocabulary Vocabulary Genre/forms with particular functions Genre/forms with particular functions ▪ Linguistic features (grammar, organization) organization) ▪ Textual resources (headers, table of contents, illustrations, graphs, charts) contents, illustrations, graphs, charts) 42

43. © Stanford Center for Assessment, Learning, and Equity, 2010 VocabularyVocabulary Technical vocabulary: triangle, metaphor, metabolize Technical vocabulary: triangle, metaphor, metabolize Words whose technical meaning is different than everyday language: “balance” in chemistry, “plane” in mathematics, “ruler” in history/social science, “force” in science Words whose technical meaning is different than everyday language: “balance” in chemistry, “plane” in mathematics, “ruler” in history/social science, “force” in science Connector words: and, but, because, therefore, however Connector words: and, but, because, therefore, however 43

44. © Stanford Center for Assessment, Learning, and Equity, 2010 Genres/FormsGenres/Forms Have a general structure – e.g., narratives, explanations, arguments Have a general structure – e.g., narratives, explanations, arguments Structure of an explanation Structure of an explanation - Description of what is being explained - Statements of cause-effect relationships - Sometimes ending with an interpretation judgment judgment 44

45. © Stanford Center for Assessment, Learning, and Equity, 2010 Subject-Specific Genres/FORMS Representing word problems mathematically Representing word problems mathematically Procedures for a science experiment Procedures for a science experiment Literary interpretation Literary interpretation Argument proposing causes of an historical event Argument proposing causes of an historical event 45

46. © Stanford Center for Assessment, Learning, and Equity, 2010 Structures of Argument Genre/FORM Mental verbs used to express opinions: like, believe Mental verbs used to express opinions: like, believe Move from personal to impersonal voice Move from personal to impersonal voice Connectives used for logical relations and to link points Connectives used for logical relations and to link points ‣ Temporal connectives: first, next ‣ Causal conditional connectives: because ‣ Comparative connectives to introduce counterpoints: consequently, therefore 46

47. © Stanford Center for Assessment, Learning, and Equity, 2010 Structure of Arguments (from Knapp & Watkins, 2005) Simple argument: point/proposition, elaboration Simple argument: point/proposition, elaboration ‣ “I like The Simpsons because they make me laugh.” Argument with evidence: Proposition, argument, conclusion Argument with evidence: Proposition, argument, conclusion Discussion: statement of issue, arguments for, arguments against, recommendation Discussion: statement of issue, arguments for, arguments against, recommendation Elaborated discussion: statement of issue, preview of pro/con, several iterations of point/elaboration representing arguments against, several iterations of point/elaboration representing arguments for, summary, conclusion Elaborated discussion: statement of issue, preview of pro/con, several iterations of point/elaboration representing arguments against, several iterations of point/elaboration representing arguments for, summary, conclusion 47

48. © Stanford Center for Assessment, Learning, and Equity, 2010 Characteristics of Advanced Academic Language Abstractions: government, electron, linear equation, acid Abstractions: government, electron, linear equation, acid Nominalizations, verbs or adjectives becoming nouns to enable more dense text or more cohesive text: Nominalizations, verbs or adjectives becoming nouns to enable more dense text or more cohesive text: organize into…→ this organization… …were revealed. The trigger for this revelation was… More precise connector words and phrases, going beyond “and” or “but” to “in contrast” or “Given this, it follows that…”. More precise connector words and phrases, going beyond “and” or “but” to “in contrast” or “Given this, it follows that…”. 48

49. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language Competencies Measured (Rubric 10) Understanding language demands and resources for instructional emphasis Understanding language demands and resources for instructional emphasis ‣ Identification of linguistic features of a genre/form/function addressed within instruction ‣ Relation of vocabulary identified to content and to students’ academic language proficiencies ‣ Description of student language strengths and needs 49

50. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language Competencies Measured (Rubric 11) Expanding students’ academic language repertoires Expanding students’ academic language repertoires ‣ Making key linguistic features related to genre purpose (form and function) visible to students ‣ Modeling vocabulary and linguistic features and providing opportunities for practice (developing fluency) ‣ (at higher levels) appropriateness of models for students at different levels of language proficiency 50

51. © Stanford Center for Assessment, Learning, and Equity, 2010 Analyze video for academic language using graphic organizer Analyze video for academic language using graphic organizer http://www.learner.org/resources/series33.html ActivityActivity 51

52. © Stanford Center for Assessment, Learning, and Equity, 2010 Academic Language Takeaways Academic language is different from everyday language. Some students are not exposed to this language outside of school. Academic language is different from everyday language. Some students are not exposed to this language outside of school. Much of academic language is discipline-specific. Much of academic language is discipline-specific. Unless we make academic language explicit for learning, some students will be excluded from classroom discourse and future opportunities that depend on having acquired this language. Unless we make academic language explicit for learning, some students will be excluded from classroom discourse and future opportunities that depend on having acquired this language. 52

53. © Stanford Center for Assessment, Learning, and Equity, 2010 Upcoming Webinar Tuesday, November 16 1:30 Eastern Time To register contact Rachel Popham at To register contact Rachel Popham at ‣ rpopham@aacte.org 53