Evaluating Professional Development At the School Site Level Cheryl “Li” Walter, Ph.D. CA SIG Evaluator

School Site Teams Focused Engage key players –Principals and Teachers Training & planning time Ongoing coaching Systems change focus Peer-to-peer TA Seeds of larger system changes (districtwide)

Content Areas Reading/Literacy Positive Behavioral Supports Special Ed/General Ed Collaboration

Focus of the Evaluation Degree of Implementation Fidelity of Implementation Sustained Implementation Site level Systems Change

Evaluation as Intervention Useful to Schools and their Coaches Self-Reflection Site Stories (to be shared with other sites) Observation and Feedback Outcomes Focus/Data Tools

3-month Electronic Follow-up –Are you implementing the learning? –How well is it working? –Are you sharing your learning with others? –Barriers and facilitators to implementation? Administrators Site team composition critical –At trainings and at the site

Online Chat Rooms For each program area For each site Coaches have access Evaluators analyze themes of chats –Helps the project assess needs for planning Teleconferences Booster trainings

Site Self-Reflection Team Implementation Checklists (TIC) –Degree of Implementation –Fidelity of Implementation –Systems Change Useful to Sites to see where they are and where they need to go Useful to Coaches in assessing site needs Useful to SIG in assessing training needs

Team Implementation Checklists (TICs) TIC from PBS for behavior programs Using same process, similar instrument with different content with ERIA literacy program –Checklist, simple and quick to use –Focuses sites on the key components, which helps build fidelity –Can quantify implementation

Example TIC- Literacy Program Assessing student’s specific reading skills –Developed criteria for placement Specific reading intervention programs –Staff trained –Being used with fidelity Response to intervention –Multiple levels of intervention provided Systems change –Site team established, scheduling, collab time, etc.

Site Action Plans Paralleling the items in the TIC Operationalize site level practices through answering critical questions Specific info useful to coaches and the SIG See what in place or not with sites successfully implementing or not

Site Observation and Feedback By Coaches By Administrators By Outside Evaluators Paralleling the TIC items Degree and fidelity of implementation cross check/correlation Seeing needs

Interviews and Stories Interviews are conducted with sites after several years of implementation –Have the site team tell their story of change Write ups of the stories are posted on website –Sites earlier in the process can learn from peers –See how different sites have taken different paths Cross site analysis of key elements and factors – development of tools that help developing sites

Leadership Sites Successful sites competition Attend Statewide Institute & Receive TA Keeps these sites engaged/sustained –Growing in other content areas –Learning and refining systems change skills –Learning to present what doing –Peer-to-peer TA with other sites –Lead scale-up within their districts and regions

Clear Outcomes Critical to institutionalizing and sustaining Outcomes must be what the sites care about –Need to feed back results in timely manner Improves compliance/response rates and quality of data Keeps sites focused on why doing what doing –Know whether making progress –Helps sustain when see changes in the outcomes Gain buy-in from administrators and districts

Data Tools to facilitate site level use of data Highlights Reporting –Visual, colorful, user-friendly, key points only CST Charting Program –CA Standards Test Charting Program

CST Charting Program Excel file using macros Minimal Data Entry Automatically Generates a Graph –Visual representation of data –Single-page graphic

Emergent Issues As we’re sustaining and scaling up –Increasing numbers of sites being monitored over extended periods of time –Changes at school sites (key players, programs) –Defining schoolwide or districtwide

Cheryl “Li” Walter, Ph.D. CA SIG/SPDG Evaluator