1. Assessing the Throughline Between Professional Learning and Student Outcomes
Andrew Eyres, Capital Region ESD (WA)
Sarita Siqueiros Thornburg, Puget Sound ESD (WA)
AESA December 2017

2. Take-aways
See an example of a professional learning system theory of change
Learn about the technology infrastructure created to support evaluation and data reporting
Hear lessons learned about evaluating a statewide professional learning system

3. Who is in the room?
Round robin introductions of participants

4. Who are we?

5. Washington’s Context 200 districts with under 5000 students
1 elected state superintendent of public instruction
9 Educational Service Districts
295 school districts, local control
200 districts with under 5000 students
Charter schools new in
70,000 educators
Over 1 million students
40% free and reduced- qualified students
9% bilingual/migrant population
13% special education qualified
Countless stakeholders and partners
Andrew
Our state superintendent is elected, to serve our 295 school districts, we have nine ESDs. On note is our 200 districts with less than 5000 students .

6. K-12 student demographics look different across our state
Source: OSPI Report Card

7. K-12 student demographics look different across our state
Source: OSPI Report Card

8. K-12 student demographics look different across our state
Source: OSPI Report Card

9. Opportunity gaps for students of color are common statewide
Source: OSPI Performance Indicators

10. What we are evaluating: Professional Learning System
Early Learning Fellows
English Language Arts Fellows
Mathematics Fellows
Science
Fellows
Regional Early Learning Coordinators
Regional
Literacy Coordinators
Regional Mathematics Coordinators
Regional
Science
Coordinators
Regional
Early Learning Coordinator
Fellows Lead
Regional
Literacy
Coordinator
Fellows Lead
Regional Mathematics Coordinator
Fellows Lead
Regional
Science
Coordinator
Fellows Lead
This is the system we are evaluating
There are two things you should notice about this chart. First, notice that the arrows show that there are relationships within the content-area groups. Fellows are led by Regional ESD Coordinators. One of those Coordinators is the Fellows Lead.
The second thing you should notice is that arrows show relationships across the content areas, as well. Although the professional learning in each content area may be very specific, regional coordinators work across content areas to build connections in the leadership experiences Fellows engage in.
To summarize:
Each content-area group of Fellows are led by Regional Coordinators—1 at each ESD.
Each content-area group of Coordinators has a Fellows Lead. There are 4 Fellows Leads.
The Fellows Leads work with OSPI content-area leadership and Association of Educational Service Districts champions—directors and Assistant Superintendents who are helping to guide the Fellows’ Network.
Fellows’ Advisory Committee
(with OSPI and AESD Leadership)

11. Why evaluate?
To understand the effectiveness and influence of our statewide professional learning system to inform improvement and investment.

12. Goals of the Network Evaluation
The goals of the evaluation are to:
Provide ESD superintendents, assistant superintendents, and coordinators with useful information to help guide planning efforts and strengthen outcomes of the Professional Learning Network
Provide coordinators with useful information that will assist and strengthen their own professional practice
Document the impact of the OSPI/AESD Professional Learning Network on improvements in school staff or early learning educator practice
Document the influence on student academic and non-academic performance
Accomplish the above as easily as possible, since we are all very busy!

13. Kauffman & Associates, Inc.
KAI has a combined experience of 60 years of evaluating educational program at the federal and state levels. KAI conducts high-quality, culturally appropriate research among diverse populations to produce data that create knowledge, explore issues, improve programs, and inform policy making. KAI’s staff also has extensive experience in various forms of social science research. KAI’s team uses both qualitative and quantitative methods, including focus groups, interviews, archival research and surveys, all with an emphasis on applicable results, analyzed and synthesized for maximum program and policy impact. Please visit in or.der to learn more about KAI
Drs. Janet Gordon and Joan LaFrance serve as Project Director/Principal Investigator and Senior Evaluator, respectively. 

14. What we evaluate Participant reaction Participant learning
Organizational support and learning
Participant use of new knowledge and skills
Student learning outcomes
our evaluation has data collection related to each of these levels. Ultimately we are most interested in how our professional learning system influences better student outcomes, and this is the most challenging dimension to assess. We will share a bit later on in the presentation about how we are approaching this.
Adapted from Guskey, T.R. (2002). Does It Make a Difference? Evaluating Professional Development. Educational Leadership. 59(6),

15. From a 30,000 foot view
The process that Janet and Joan facilitated to create the Theory of Change and Logic Model was participatory and iterative.  They held theory of change conversations with each of the Content Area Coordinators’ groups and then brought a smaller cross content team to help finalize these docs.
Handout

16. Logic Model From a 30,000 foot view
The process that Janet and Joan facilitated to create the Theory of Change and Logic Model was participatory and iterative.  They held theory of change conversations with each of the Content Area Coordinators’ groups and then brought a smaller cross content team to help finalize these docs.
Handout

17. Data Collection
Surveys
Focus groups
Self-reflection tool
Interviews

18. Relationship between Theory of Action and Surveys
Survey items are aligned with Theory of Action
For example, the participant changes in understanding and application retrospective survey helps us understand our short term outcome related to participant knowledge and skills

19. ESD Math, Science, English Language Arts & Early Learning Coordinator Services
This is an example of very high level evaluation reporting to our legislature
Coordinator-led professional development reached 15,593 education professionals across Washington State during the school year
Source: OSPI AESD Legislative Briefing

20. Who are the 2017-18 Fellows? Teachers and educators who represent:
Four Areas: English Language Arts, Math, Science, and Early Learning
198 Districts and 25 Community- based Organizations
1082 Fellows Strong!
The evaluation pays special attention to the fellows, given that the professional learning experience is ongoing throughout the year and intentionally designed
Photo courtesy of Allison Shelley/The Verbatim Agency for American Education: Images of Teachers and Students in Action | CC BY NC

21. Why is the Fellows’ Network Important to the State?
Ongoing Content-area Professional Development
Instructional Leadership
Expanded Leadership Capacity to Improve Student Learning
if we’re going to make a difference in improving student learning and decreasing opportunity gaps, we need to empower more educators to lead change within their classrooms, schools, organizations, and communities.
If we can help you build fellows knowledge and leadership skills, then they can help us reach more teachers, more students, more community members.

22. Where are the Fellows?
This slide shows how the Fellows are distributed across the state, and the variations within and across ESD regions. Your takeaway should be that there are at least 10 Fellows in each of the four content areas in each ESD. The reasons for the differences are mostly rooted in which content areas have been part of Fellow the longest.
For instance, because math and ELA Fellows were out of the gate first, we can see that the red (ELA) and yellow (math) bars of this chart are generally the tallest. But science is surging in some ESDs past ELA (see 121), past math (see 171), and even past math and ELA (see 105, 114). ESD 113 sports the highest number of math and ELA Fellows. Hey, Andrew, which ESD are you from?
Early Learning Fellows, in just their second year, are represented by the blue bars. Although their numbers are smaller, they are steadily moving upward, even nosing past ELA in ESD 123.

23. COHERENCE Focusing Direction Cultivating Collaborative Cultures
Deeping Learning
Securing Accountability
Pam,

24. History “Fellows?” Began in 2012-13 Collaboration with OSPI
New learning standards for Math, ELA & Science
Limited resources with great need for teacher support
Julie

25. Key Components Application process with administrator support
Learning= Teacher Leadership, Equity & Content
4 Convenings per year
Assurances & Action Plans
Willingness to lead change
Julie

26. What we evaluate Participant reaction Participant learning
Organizational support and learning
Participant use of new knowledge and skills
Student learning outcomes
It is through the fellows that we begin to understand the througline to student outcomes
Adapted from Guskey, T.R. (2002). Does It Make a Difference? Evaluating Professional Development. Educational Leadership. 59(6),

27. Measuring Influence on Student Learning
KAI will work with Coordinators to craft a pilot to gauge the influence of professional development participation on student learning
A purposeful sample of Fellows and non-Fellows will be used
Multiple student artifacts (concept maps, classroom based assessments, SBA, NWEA MAP, performance assessments, etc.) will be examined for feasibility of use
Non-academic measures (engagement, collaboration, discourse, attendance, behavior, discipline, etc.) may provide another view
How we begin to understand the througline to student outcomes
Measuring the influence of Coordinators’ work on student learning will be a pilot this year to allow for the flexibility to explore a variety of ways. Pre/post assessments will not be used; instead we will be working with Coordinators and they, in turn, will be working with Fellows and non-Fellow participants to investigate using student artifacts, such as concept maps, classroom-based assessments, etc. We will also want to look at non-academic measures, such as engagement, collaboration, attendance, etc.

28. How do we report evaluation results to 40+stakeholder groups statewide?
Nine ESDs, four content areas, assistant superintendents, OSPI

29. OSPI/AESD Database Management System: Features and Benefits
Data collection through intake forms (like coordinator survey) and participant surveys
Personal logins with customized access to forms, surveys, and reports
Accessible through modern desktop browsers
Ability to take surveys and review reports on mobile devices
Benefits
Real-time queries for results of Participant Satisfaction Survey and other surveys
Reports can be run with custom specs by ESD and date range
Pre-defined lists of choices and drop-downs, which increase data validity (for some course titles)
Emerging ability to look at data longitudinally
We are investing in the creation of a database that allows for real time queries and reporting that can be accessed by coordinators and leads

30. Screen Shot from a Similar Database – Reports on Demand

31. Screen Shot from a Similar Database – Flexible Queries
This database will be up and running in the spring

32. Lessons Learned
It is challenging to ensure that everyone statewide has all the information they need at the right time

33. Professional Learning Network Evaluation Website https://www. waesd
The website includes links to the Evaluation Plan Operations Manual, the Evaluation Plan, the overarching TOA and LM, and Fellows Action Plans. There is also a Calendar of Events on the website.
This is one way we are trying to address the communications needs, along with monthly newsletters through mailchimp

34. Lessons Learned
Stakeholders have different data needs, and it is difficult for one evaluation to meet everyone’s requirements
People are really excited to better understand the influence and impact of professional learning
The efforts to implement the evaluation called for a need for the Research, Evaluation and Data Center to provide a backbone communications function about key program decisions and developments

35. Questions & Discussion

36. Contact the OSPI-AESD Professional Learning Network Evaluation Team
Hilary Loeb Director, Strategy, Evaluation and Learning, PSESD    Cassandra O’Francia Program Specialist, Strategy, Evaluation and Learning, PSESD    Janet Gordon Evaluation Lead, Kauffman and Associates, Inc.    Julie Cahoon Project Manager, Kauffman and Associates, Inc.