Assessing the Throughline Between Professional Learning and Student Outcomes Andrew Eyres, Capital Region ESD (WA) Sarita Siqueiros Thornburg, Puget Sound ESD (WA) AESA December 2017

Take-aways See an example of a professional learning system theory of change Learn about the technology infrastructure created to support evaluation and data reporting Hear lessons learned about evaluating a statewide professional learning system

Who is in the room? Round robin introductions of participants

Who are we?

Washington’s Context 200 districts with under 5000 students 1 elected state superintendent of public instruction 9 Educational Service Districts 295 school districts, local control 200 districts with under 5000 students Charter schools new in 2013-14 70,000 educators Over 1 million students 40% free and reduced- qualified students 9% bilingual/migrant population 13% special education qualified Countless stakeholders and partners Andrew Our state superintendent is elected, to serve our 295 school districts, we have nine ESDs. On note is our 200 districts with less than 5000 students .

K-12 student demographics look different across our state Source: OSPI Report Card 2016-17

K-12 student demographics look different across our state Source: OSPI Report Card 2016-17

K-12 student demographics look different across our state Source: OSPI Report Card 2016-17

Opportunity gaps for students of color are common statewide Source: OSPI Performance Indicators

What we are evaluating: Professional Learning System Early Learning Fellows English Language Arts Fellows Mathematics Fellows Science Fellows Regional Early Learning Coordinators Regional Literacy Coordinators Regional Mathematics Coordinators Regional Science Coordinators Regional Early Learning Coordinator Fellows Lead Regional Literacy Coordinator Fellows Lead Regional Mathematics Coordinator Fellows Lead Regional Science Coordinator Fellows Lead This is the system we are evaluating There are two things you should notice about this chart. First, notice that the arrows show that there are relationships within the content-area groups. Fellows are led by Regional ESD Coordinators. One of those Coordinators is the Fellows Lead. The second thing you should notice is that arrows show relationships across the content areas, as well. Although the professional learning in each content area may be very specific, regional coordinators work across content areas to build connections in the leadership experiences Fellows engage in. To summarize: Each content-area group of Fellows are led by Regional Coordinators—1 at each ESD. Each content-area group of Coordinators has a Fellows Lead. There are 4 Fellows Leads. The Fellows Leads work with OSPI content-area leadership and Association of Educational Service Districts champions—directors and Assistant Superintendents who are helping to guide the Fellows’ Network. Fellows’ Advisory Committee (with OSPI and AESD Leadership)

Why evaluate? To understand the effectiveness and influence of our statewide professional learning system to inform improvement and investment.

Goals of the Network Evaluation The goals of the evaluation are to: Provide ESD superintendents, assistant superintendents, and coordinators with useful information to help guide planning efforts and strengthen outcomes of the Professional Learning Network Provide coordinators with useful information that will assist and strengthen their own professional practice Document the impact of the OSPI/AESD Professional Learning Network on improvements in school staff or early learning educator practice Document the influence on student academic and non-academic performance Accomplish the above as easily as possible, since we are all very busy!

Kauffman & Associates, Inc. KAI has a combined experience of 60 years of evaluating educational program at the federal and state levels. KAI conducts high-quality, culturally appropriate research among diverse populations to produce data that create knowledge, explore issues, improve programs, and inform policy making. KAI’s staff also has extensive experience in various forms of social science research. KAI’s team uses both qualitative and quantitative methods, including focus groups, interviews, archival research and surveys, all with an emphasis on applicable results, analyzed and synthesized for maximum program and policy impact. Please visit http://www.kauffmaninc.com/ in or.der to learn more about KAI Drs. Janet Gordon and Joan LaFrance serve as Project Director/Principal Investigator and Senior Evaluator, respectively. 

What we evaluate Participant reaction Participant learning Organizational support and learning Participant use of new knowledge and skills Student learning outcomes our evaluation has data collection related to each of these levels. Ultimately we are most interested in how our professional learning system influences better student outcomes, and this is the most challenging dimension to assess. We will share a bit later on in the presentation about how we are approaching this. Adapted from Guskey, T.R. (2002). Does It Make a Difference? Evaluating Professional Development. Educational Leadership. 59(6), 45-51.

From a 30,000 foot view The process that Janet and Joan facilitated to create the Theory of Change and Logic Model was participatory and iterative.  They held theory of change conversations with each of the Content Area Coordinators’ groups and then brought a smaller cross content team to help finalize these docs. Handout

Logic Model From a 30,000 foot view The process that Janet and Joan facilitated to create the Theory of Change and Logic Model was participatory and iterative.  They held theory of change conversations with each of the Content Area Coordinators’ groups and then brought a smaller cross content team to help finalize these docs. Handout

Data Collection Surveys Focus groups Self-reflection tool Interviews

Relationship between Theory of Action and Surveys Survey items are aligned with Theory of Action For example, the participant changes in understanding and application retrospective survey helps us understand our short term outcome related to participant knowledge and skills

ESD Math, Science, English Language Arts & Early Learning Coordinator Services This is an example of very high level evaluation reporting to our legislature Coordinator-led professional development reached 15,593 education professionals across Washington State during the 2016-17 school year Source: OSPI AESD Legislative Briefing 2016-17

Who are the 2017-18 Fellows? Teachers and educators who represent: Four Areas: English Language Arts, Math, Science, and Early Learning 198 Districts and 25 Community- based Organizations 1082 Fellows Strong! The evaluation pays special attention to the fellows, given that the professional learning experience is ongoing throughout the year and intentionally designed Photo courtesy of Allison Shelley/The Verbatim Agency for American Education: Images of Teachers and Students in Action | CC BY NC

Why is the Fellows’ Network Important to the State? Ongoing Content-area Professional Development Instructional Leadership Expanded Leadership Capacity to Improve Student Learning if we’re going to make a difference in improving student learning and decreasing opportunity gaps, we need to empower more educators to lead change within their classrooms, schools, organizations, and communities. If we can help you build fellows knowledge and leadership skills, then they can help us reach more teachers, more students, more community members.

Where are the 2017-18 Fellows? This slide shows how the Fellows are distributed across the state, and the variations within and across ESD regions. Your takeaway should be that there are at least 10 Fellows in each of the four content areas in each ESD. The reasons for the differences are mostly rooted in which content areas have been part of Fellow the longest. For instance, because math and ELA Fellows were out of the gate first, we can see that the red (ELA) and yellow (math) bars of this chart are generally the tallest. But science is surging in some ESDs past ELA (see 121), past math (see 171), and even past math and ELA (see 105, 114). ESD 113 sports the highest number of math and ELA Fellows. Hey, Andrew, which ESD are you from? Early Learning Fellows, in just their second year, are represented by the blue bars. Although their numbers are smaller, they are steadily moving upward, even nosing past ELA in ESD 123.

COHERENCE Focusing Direction Cultivating Collaborative Cultures Deeping Learning Securing Accountability Pam,

History “Fellows?” Began in 2012-13 Collaboration with OSPI New learning standards for Math, ELA & Science Limited resources with great need for teacher support Julie http://www.k12.wa.us/CurriculumInstruct/Fellows.aspx

Key Components Application process with administrator support Learning= Teacher Leadership, Equity & Content 4 Convenings per year Assurances & Action Plans Willingness to lead change Julie

What we evaluate Participant reaction Participant learning Organizational support and learning Participant use of new knowledge and skills Student learning outcomes It is through the fellows that we begin to understand the througline to student outcomes Adapted from Guskey, T.R. (2002). Does It Make a Difference? Evaluating Professional Development. Educational Leadership. 59(6), 45-51.

Measuring Influence on Student Learning KAI will work with Coordinators to craft a pilot to gauge the influence of professional development participation on student learning A purposeful sample of Fellows and non-Fellows will be used Multiple student artifacts (concept maps, classroom based assessments, SBA, NWEA MAP, performance assessments, etc.) will be examined for feasibility of use Non-academic measures (engagement, collaboration, discourse, attendance, behavior, discipline, etc.) may provide another view How we begin to understand the througline to student outcomes Measuring the influence of Coordinators’ work on student learning will be a pilot this year to allow for the flexibility to explore a variety of ways. Pre/post assessments will not be used; instead we will be working with Coordinators and they, in turn, will be working with Fellows and non-Fellow participants to investigate using student artifacts, such as concept maps, classroom-based assessments, etc. We will also want to look at non-academic measures, such as engagement, collaboration, attendance, etc.

How do we report evaluation results to 40+stakeholder groups statewide? Nine ESDs, four content areas, assistant superintendents, OSPI

OSPI/AESD Database Management System: Features and Benefits Data collection through intake forms (like coordinator survey) and participant surveys Personal logins with customized access to forms, surveys, and reports Accessible through modern desktop browsers Ability to take surveys and review reports on mobile devices Benefits Real-time queries for results of Participant Satisfaction Survey and other surveys Reports can be run with custom specs by ESD and date range Pre-defined lists of choices and drop-downs, which increase data validity (for some course titles) Emerging ability to look at data longitudinally We are investing in the creation of a database that allows for real time queries and reporting that can be accessed by coordinators and leads

Screen Shot from a Similar Database – Reports on Demand

Screen Shot from a Similar Database – Flexible Queries This database will be up and running in the spring

Lessons Learned It is challenging to ensure that everyone statewide has all the information they need at the right time

Professional Learning Network Evaluation Website https://www. waesd The website includes links to the Evaluation Plan Operations Manual, the Evaluation Plan, the overarching TOA and LM, and Fellows Action Plans. There is also a Calendar of Events on the website. This is one way we are trying to address the communications needs, along with monthly newsletters through mailchimp

Lessons Learned Stakeholders have different data needs, and it is difficult for one evaluation to meet everyone’s requirements People are really excited to better understand the influence and impact of professional learning The efforts to implement the evaluation called for a need for the Research, Evaluation and Data Center to provide a backbone communications function about key program decisions and developments

Questions & Discussion

Contact the OSPI-AESD Professional Learning Network Evaluation Team Hilary Loeb hloeb@psesd.org 425-917-7603 Director, Strategy, Evaluation and Learning, PSESD    Cassandra O’Francia cofrancia@psesd.org 425-917-7846 Program Specialist, Strategy, Evaluation and Learning, PSESD    Janet Gordon 509-860-5273 janet.gordon@kauffmaninc.com Evaluation Lead, Kauffman and Associates, Inc.    Julie Cahoon julie.cahoon@kauffmaninc.com 509-789-0661 Project Manager, Kauffman and Associates, Inc.