1 Using K-12 Assessment Data from Teacher Work Samples as Credible Evidence of a Teacher Candidate’s Ability to Produce Student Learning Presented by Roger Pankratz Tony Norman Judy Pierce of Western Kentucky University Presented at The Making an Impact Conference Kennesaw University March 2007

2 Our Leading Question  Can P-12 assessment data on teacher work samples produced by student teachers provide credible evidence that Western’s graduates have the ability to impact learning with the students they teach?

3 Our Hypothesis  Work sample data can provide credible evidence of ability to impact learning provided unit learning objectives and assessments to measure learning meet credibility standards.

4 Our Research Questions 1) What are the gains in K-6 student achievement reported by Western’s student teachers on teacher work samples? 2) What is the quality of assessments and degree of alignment between unit objectives and assessments used in student teachers’ work samples at Western? 3) To what extent do unit objectives in Western’s student teacher work samples address Kentucky’s core content standards?

5 Our Rationale for Exploring the Use of Work Sample Data as a Measure of Impact  Renaissance Partnership work (Denner et al., 2004) demonstrating relationship between overall TWS performance and quality of learning assessments  Institutional work (e.g., Idaho State and Longwood) suggesting that P-12 scores within the TWS could be used to ascertain impact on learning

6 What is a Teacher Work Sample?  Provides documentation of a teacher candidate’s ability to plan, deliver, and assess a complete standards-based unit of instruction  Provides instruction embedded evidence of the impact of teaching on student learning

7 Validity & Reliability Evidence  Processes parallel targeted INTASC and Kentucky Teacher Standards  Experts affirm that processes describe behaviors/skills that are frequent, critical, and necessary for good teaching.  Multi-institutional studies ( Denner et al., 2004 ) show that TWS can be scored reliably  Two WKU studies show between 81-90% inter-rater agreement. An additional study showed between 67%-95% agreement (faculty tended to score higher) but some training discrepancies were present.

8 The Context of Teacher Work Sample Use at Western Kentucky University  Western Kentucky University has about 400 student teachers each year: 300 K  Western has required TWS as a performance assessment in student teaching since 2001.

9 The Context of Teacher Work Sample Use at Western Kentucky University  Student teachers upload electronic versions of their teacher work samples on the College of Education’s data management system.

10 The Context of Teacher Work Sample Use at Western Kentucky University  Western convenes teacher educators and school practitioners each year to independently score TWS’s (semi-holistic and holistic).  University student teacher supervisors initially score TWS’s for student teaching evaluation.  Performance data of TWS’s are used for program improvement.

11 The Seven Teaching Processes Assessed in Western’s Teacher Work Sample 1) Use of teaching context 2) Unit learning objectives 3) Assessment plan 4) Instructional design 5) Instructional decision making 6) Analysis of learning results 7) Reflection on teaching and learning

12 The Design of our Initial Study of Teacher Work Sample Assessment Data, Quality of Assessments and Quality of Unit Objectives  Analysis of reported pre-post assessment student gains in 22 student teacher work samples produced in fall of 2006 (460 students)  Consensus rating of the quality of assessments developed and administered by each student teacher

13 The Design of our Initial Study of Teacher Work Sample Assessment Data, Quality of Assessments and Quality of Unit Objectives  Description of assessments used for each unit learning objective number and type (short answer, multiple choice, true/false, open response, matching)  “Quick and dirty” rating of the quality of instructional objectives  Alignment with Kentucky’s content standards  Importance of objective related to content standards

14 Study Results: Impact on Student Learning Significant Student Gain (Controlling for Student Teacher Effects) Student Teacher N = 22, P-12 Student N = 464

15 Study Results: Quality of Assessments Rubric

16 Study Results: Quality of Assessments  All TWS had two levels of DOK with most having a lower level goal (DOK 1,2) and higher level goal (DOK 3,4).  Total QA Scores ranged from 9 to 15 (M = 11.89, SD = 1.73).  Holistic QA Scores Frequencies: “2” = 14 (64%), “3” = 3 (27%), and “4” = 2 (9%)  No significant relationships between Faculty Holistic TWS Score and either Total QA Scores (Pearson r = -.25) or Holistic QA Scores (r = -.18)  No significant relationships between Total Gain Scores and either Total QA Scores (Pearson r =.34) or Holistic QA Scores (r =.31)

17 Study Results: Description of Assessments for 10 Work Samples Selected at Random 1 – Kindergarten3 – Science 3 – Grade 14 – Math 2 – Grade 32 – Health 2 – Grade 41 – Social Studies 2 – Grade 6

18 Study Results: Description of Assessments 15.5  Average number of assessment items per unit of instruction – – 30  Range of assessment items for a unit of instruction: 8 – 30 5  Average number of assessment items per instructional objective:  Range of assessment items for an instructional objective:

19 Study Results: Description of Assessments  Types of Assessments Used Overall 53  Short answer: 53  Multiple choice:  Matching:  True/False:  Open Response: 13

20 Study Results: Description of Assessments Types of assessments used by unit objective Depth of Knowledge (DOK) DOK1DOK2DOK3DOK4 Short Answer Multiple Choice Matching61350 True/False7514 Open Response3541

21 Study Results: Quality of Objectives  Alignment with Kentucky Core Content 4 – 0 3 – 4 2 – 3 1 – 2  Importance of objective relative to standards 4 – 1 3 – 7 2 – 1 1 – 0

22 CONCLUSIONS 1)TWS assessment data have potential for credibility provided standards for instructional objectives and assessments are addressed. 2)TWS show impressive average student gains but there is a large variability by individual student.

23 CONCLUSIONS 3)The quality of assessments used looks promising but needs improvement. 4)The number of assessment items per objective for the ten work samples reviewed is too small for any degree of reliability.

24 CONCLUSIONS 5)The quality of instructional objectives for the ten work samples reviewed varies widely with a majority of objectives needing improvement in alignment and focus. 6)More instruction is needed on matching assessments to objectives – especially assessments for DOK 3 and 4 level objectives.

25 CONCLUSIONS 7)With a sustained effort to provide instruction for teacher candidates in the development of high quality instructional objectives and assignments, data from work samples could be one source of evidence of teacher graduate’s potential for impact on student learning.

26 Next Steps  Conducting larger, more precise TWS studies to explore, such as… The extent to which all TWS scores can be accessed and matched to goals The quality of the assessments (depth of knowledge, alignment to standards, adaptations to meet diverse needs of students) The potential for differential impact based on student-teacher program and P-12 student characteristics  Working with state to connect TWS, as well as other program assessments, success to impact on P-12 student learning during teaching internship

27 Discussion Question 1 What are your reactions to and/or comments on Western’s exploration of teacher work sample data as one source of impact evidence?

28 Discussion Question 2 What suggestions do you have for next steps Western might consider in it’s effort to develop credibility in student teacher work sample data?

29 Discussion Question 3 What would you suggest as standards for assessments and learning objectives to improve credibility of work sample data?

30 Discussion Question 4 What actions should Western take to engage other institutions in development of work samples that produce credible data?