Data Wise Data Presentation University of Massachusetts Boston Aimee D’Avignon, Fabian Torres-Ardila, Janna Kellinger, Mike Gilbert, Kevin Ziomek November, 2014

Overview of the Process

Step 1: Organize for Collaborative Work Develop well-functioning, collaborative teams based on the Data Wise Norms: Take an inquiry stance Ground statements in evidence Assume positive intentions Stick to protocol Start and end on time Be here now -Choose focus area as a department based on data at the dept. level -Examine program level data to determine action plan

Step 2: Build Assessment Literacy -Understanding what our working conception of assessment literacy is for this project -Understanding limitations and potential uses for assessment and data -Understanding affordances and constraints of presenting data in various ways -Developing a common language around assessment

Step 2: Build Assessment Literacy

Step 3: Choose a Focus Area A Focus Area is: -Related to instruction -Narrows scope of the inquiry while remaining broad enough to be relevant to many/most of the staff member Possible Focus Areas:

Step 3: Ask a priority question -Arises from a collaborative process -Relates to instruction -Is actionable -Further narrows scope of inquiry -Is genuinely intriguing to faculty and staff Priority Questions:

Step 4: Determine a learner-centered problem -Directly relates to the priority question -Based on digging into multiple data sources -Is within the school’s control -Is a statement about student learning (not a question) -Is specific and small Our students struggle with . . .

Step 5: Examine Instruction Reframe the Learner Centered Problem as a Problem of Practice: -Directly relates to Learner Centered Problem -Is based on evidence from examining instruction -Is within our control Our faculty struggle with . . .

Step 6: Develop an Action Plan -States specifically what teachers will do to address the problem of practice -Action steps in this plan should be research-based, evidence -based, high-leverage, assigned to specific people, and time-bound

Step 7: Plan to Assess Progress The Plan to Assess Progress: -Helps us determine whether our action plans are, in fact, helping students make progress towards addressing the LCP -Should include short-term, medium-term, and long-term assessments -Should include specific and measurable student learning

Step 8: Act and Assess Make it happen Evaluate success

Our Turn

Step 1: Organize for Collaborative Work Develop well-functioning, collaborative teams based on the Data Wise Norms: Take an inquiry stance Ground statements in evidence Assume positive intentions Stick to protocol Start and end on time Be here now Two-pronged approach: 1) Datawise Team inquiry projects 2) Dept. wide inquiry project based on our 4 TEAC claims -Each TEAC claim group examine dept. level data to come up with a learner- centered problem -Then each program examines program level data to examine instruction and develop and carry out an action plan

Step 1: Organize for Collaborative Work Strengths Challenges We have accreditation as motivation Infrastructure is in place We have the support of the Dean and the Chair of C&I to work on this project We are launching a new Undergraduate program which affords us the opportunity to revisit our practices The structure of the University and College establishes and reinforces data silos. Therefore obtaining rich and illustrative points of data can be time consuming. scheduling around work responsibilities means that all necessary stakeholders are not able to be present at the same time Infrastructure (University, College and Department) could be tweaked to improve efficiency Because we are all stretched for time, the decision was made to use Department meeting time as much as possible to do the department wide Data Wise process work.

Step 2: Build Assessment Literacy One new tool that will be at our disposal is EDWIN: State data that follows our students from pre-service to in-service Will include MTELs, PST/PPA data, Teacher Evaluation Scores, Where our students teach, etc.

Step 3: Data Overview: Data Inventory Name of Data Source Content Area Dates of Collection Students Assessed Who Has Access to these data? How are these data currently used? How could these data be more effectively used? MTEL C&I, Curric, Content Varies, prior to Practicum All DWT   Exit Survey Graduation Christine, faculty exercise at a retreat Analyze and share with more persons; programmatic change GPA Program entry Demographics Program Entry EDWIN Exit and Employment Data Aimee Brand new Analyze and share, prepare graduates, support claims PPA Lisa, Christine,  GPDs TEAC, licensure Identify areas where our students struggle. Admissions

Step 3: Data Overview: Assess Assessments Exit Survey (Administered Spring 2013 and Spring 2014) Administration is given twice per year (at graduation): the survey assesses students’ perspectives of the program and their preparation (feelings) Categories do not add up to 100% Some prompts are questions whereas others are statements The data displayed corresponds to a subset of 14 questions. This subset was classified as “Student perception of pedagogical competence” by the data team. N is small (Spring 2013 = 11; Spring 2014 = 28)

Step 3: Data Overview: Choose a Focus Area The focus area for our first inquiry cycle is: TEAC CLAIMS Our rationale for choosing this focus area is: We choose these claims because they are important to our College Mission of preparing urban teachers Our priority question was: How well are we preparing our graduates to meet our TEAC Claims? Our graduates will: - demonstrate cultural competence and address social justice issues in urban and other diverse contexts. (QP 1.3, 1.4.2)   - demonstrate knowledge of content and pedagogical and assessment practices that promote learning. (QP 1.1, 1.2, 1.4.3) -incorporate family and community resources into their practice. (Q.P. 1.1, 1.2, 1.4.3) -demonstrate the skills necessary to engage in professional and life long learning. (QP 1.41, 1.4.3)

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 1

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 1 Student Perception of Pedagogical Competence Question: My teacher education program prepared me to… 1. UMass-Boston teacher preparation faculty structured their course around real problems of teaching practice. 2. Use the state's curriculum frameworks and standards to plan instruction. 3. Teach content knowledge and skills. 4. Use inquiry methods to create effective learning environments 5. Teach problem solving, conceptual understanding, and other aspects of higher order thinking. 6. Motivate students to participate in academic tasks. 7. Use educational technology as a learning tool. 8. Teach students with different ability levels in the same class 9. Reflect on and improve my teaching performance 10.Teach in a high-stakes testing environment. 11. Use classroom research and inquiry strategies 12. Read and understand Individual Education Plans (IEPs) and provide appropriate accommodations for the individual students in my classroom 13. Use classroom management techniques/procedures. 14. Read and understand 504 plans and provide appropriate accommodations for individual students in my classroom.

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 1 Strongest Aspects: “faculty brought a wealth of knowledge to the classroom. The minimum amount of experience the adjunct faculty had was 20+ years, that kept my experience focused on the realities that I will experience.” “Most of the staff actually work in BPS and were great resources to have.” Suggestions for Program: “More hands on instruction opportunities.” “Application of theory into practice” “Spend more time on practical things like classroom management.” “More emphasis on day-to-day classroom situations: behavior management, how to deliver assessments, how to interpret test results” “More attention to practical aspects of teaching; e.g., classroom management, lesson planning” “More practice less theory.” “Focus more on the practical aspects of teaching like pedagogy.”

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 1 Qualitative Data Summary: While instructors have a lot of real-world experience and knowledge of teaching in urban schools, students expressed a need for more practice with real-world situations likely to occur in urban schools. This was reflected in the quantitative data as well. For example, students rated “structuring classes around real teaching problems” high, but some of the practical applications like using standardized assessment results, classroom management, and using IEP and 504 plans rated low.

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 1 Practical Applications Classes structured around real-world problems

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 1 In addition, another quantitative measure on our exit survey supported the aspect of our conclusion relating to instructor strength: The UMass Boston Teacher Preparation faculty know a lot about the reality of contemporary schools. Spring 2013-100% Spring 2014-96%

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 1 Learner-centered problem: While our instructors provide many real life examples from their own experiences, our students struggle to apply them to their own teaching.

Step 5: Examine Instruction DataWise Team DataWise Team Inquiry Cycle 1 Problem of practice: Our instructors impart their own real-life experiences but need to build in more ways for students to practice applying this knowledge

Step 6: Develop an Action Plan DataWise Team Inquiry Cycle 1 Potential solutions: Use instructors real-life experiences to build: in-class role-playing scenarios, case-based teaching, micro-teaching, problem-based learning, analysis and application of real data, simulations, etc.

Step 7: Plan to Assess Progress DataWise Team DataWise Team Inquiry Cycle 1 -Data from future exit surveys -Examining syllabi -Examining student work, e.g. core assignments -Examining PST scores

Step 8: Act and Assess DataWise Team Inquiry Cycle 1 We are exploring the use of simulations, case studies, role-playing, etc.

Our Turn: Inquiry Cycle 2

Step 3: Choose a Focus Area DataWise Team Inquiry Cycle 2 A Focus Area is: -Related to instruction -Narrows scope of the inquiry while remaining broad enough to be relevant to many/most of the staff member Possible Focus Area: What challenges are our teacher candidates facing that we could better prepare them for?

Step 3: Ask a priority question DataWise Team Inquiry Cycle 2 A Priority Question: -Arises from a collaborative process -Relates to instruction -Is actionable -Further narrows scope of inquiry -Is genuinely intriguing to faculty and staff Priority Question: How well prepared are our students to meet the expectations of new state standards and assessment?

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 2 This is a second slide to put the oval and text box in

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 2 High Stakes Testing

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 2 Over the two years of data, the area with the lowest overall rating was “Teach in a High Stakes Testing Environment” The two lowest rating items on the 2014 survey were “teach in a high-stakes testing environment” and “interpret and use standardized test results”

Step 4: Some Qualitative Quotes DataWise Team Inquiry Cycle 2 “more emphasis on day-to-day classroom situations: behavior management, how to deliver assessments (TRC, Dibels, etc.), how to interpret test results” “There were a few teachers (2) who were out of date on the new issues facing teachers such as Race to the Top or the new evaluation system.”

Step 4: Digging into Student Data DataWise Team Inquiry Cycle 2 Learner-centered problem: Our students struggle with understanding and implementing effective assessment strategies as well as interpreting and applying external assessment results within the context of PARCC.

Step 5: Examine Instruction DataWise Team Inquiry Cycle 2 Problem of practice: Given that schools and teachers are increasingly held accountable for student achievement, faculty need to better prepare our students for teaching in this high-stakes testing environment.

Step 6: Develop an Action Plan DataWise Team Inquiry Cycle 2 Potential solutions: -Infusing assessment practices across courses -professional development around Common Core

Step 7: Plan to Assess Progress DataWise Team Inquiry Cycle 2 -Data from future exit surveys -Examining syllabi -Examining student work, e.g. core assignments -Examining PST scores

Step 8: Act and Assess DataWise Team Inquiry Cycle 2 We have asked someone to talk about how Boston Public Schools analyzes and uses data for improvement at our last department meeting.

Your Turn