Supporting Implementation Fidelity: Coaching Early Educators Lise Fox, University of South Florida Mary Louise Hemmeter, Vanderbilt University Pat Snyder and Tara McLaughlin, University of Florida

Two Intervention-Based Projects Embedded Instruction for Early Learning Tools for Teachers (TfT) Evaluating the Efficacy of the Teaching Pyramid Model: Using the TPOT Helping early childhood practitioners use classroom prevention and intervention strategies that promote young children's social-emotional development and prevent challenging behavior Helping preschool teachers in early learning programs learn how to use effective supports and instruction for young children with disabilities Projects focus on Professional Development Research to support implementation of these interventions.

According to NPDCI, the key components of professional development include:   (a) the characteristics and contexts of the learners (i.e., the "who" of professional development, including the characteristics and contexts of the learners and the children and families they serve); (b) content (i.e., the "what" of professional development; what professionals should know and be able to do; generally defined by professional competencies, standards, and credentials); and (c) the organization and facilitation of learning experiences (i.e., the "how" of professional development; the approaches, models, or methods used to support self-directed, experientially-oriented learning that is highly relevant to practice).

Rationale for Coaching Approaches High-quality materials and PD workshops not sufficient for use of evidence-informed practices in practice settings Difficult to define impacts of coaching in EC PD Variants and descriptions of coaching not well distinguished in the literature Some empirical support for efficacy of performance feedback in relation to implementation fidelity Few studies focused on self-monitoring as a form of “coaching” and in relation to implementation fidelity

Two forms of coaching On-site Coaching Self-Coaching

Similar components; different delivery On-site Coaching (Teacher and Coach) Self-Coaching (Teacher and Web-based Supports) Entry and relationship building (First 3 sessions) Learn about the teacher, the classroom and the teachers’ goals, share information about the coaching process, and provide supportive feedback. Goal setting Work with teacher to identify personal development goals and weekly implementation goals. Observation Observe in the classroom, engage in coaching strategies as needed. Feedback Discuss observation, provide supportive and corrective feedback , engage in coaching strategies as needed, support teacher reflection and future planning. Action plan (problem solving) Identify weekly goals, develop a plan for successful implementation , and provide resources and supports as needed. Entry and orientation to the website (1-hour orientation) Learn about the website and self-coaching process, explore website features and how to use the web-based supports to coach. Goal setting Use the online forms to identify personal development goals and graphing tool plan weekly implementation goals. Self-monitoring Work toward personal development goals and collect and record data on implementation goals. Self-evaluation Use self-coaching checklist to guide evaluation and reflection of goals, and use graphing tool to evaluate implementation . Action plan (problem solving) & Resource seeking Select weekly goals, develop an action plan for weekly goals, and use available resources on web site to problem solve when situations arise.

On-site Coaching

Evaluating the Efficacy of the Teaching Pyramid Intervention Who Preschool teachers in classrooms that include children with and without disabilities Teachers in preschool classrooms What Teaching Pyramid practices How High quality workshops Provision of classroom materials Coaching with performance feedback in classroom

The Pyramid Model: Promoting Social and Emotional Competence and Addressing Challenging Behavior Few Children Some Children All Children 9 9

Coaching Teachers in Implementing the Pyramid Implementing it in daily practice Hearing about it in training 10

Teaching Pyramid Coaching Process Preparing for Coaching Conduct baseline observations using the TPOT Meet with teacher to review TPOT and develop action plan Observe teacher with a focus on action plan items Use specific TPOT items Use other observational measures Model, feedback, and provide suggestions as planned Debrief with teacher Provide positive performance feedback Provide corrective performance feedback Discuss issues related to individual children Develop implementation steps Plan for next observation Use a variety of strategies (e.g., video, problem solving)

Follow-up: Provide email feedback following meetings Provide materials as needed Support teacher around developing plans for children at the top of the Pyramid

Let’s look at it—written debrief Hi ----, I enjoyed observing in your classroom today. I can¹t believe how well Mary is doing with her writing. She picked several words from the word wall today and wrote them on the dry erase board. She did a great job! I was able to read all of the letters and she was able to tell me each word. I continue to be amazed at her progress. You did a great job of embedding many learning trials into small group and centers. Almost all of the trials were complete with an antecedent, behavior, and consequence or error correction. I observed 4 trials for engagement, six for counting/matching numerals, and two social bids. I also observed Ms. Rogers providing two engagement trials. You have done an excellent job of instructing your paraprofessionals in delivering complete learning trials. You make such good use of their expertise in the classroom. Having them as support I am sure makes your job a little easier. I noticed that today you delivered several trials for the counting/matching numeral goal. This was great and Mary seemed to respond well. I did notice, however, that by the time she got to the center where she was counting chicks she did not seem as interested in counting. I don¹t think there is anything wrong with the activity, but perhaps because it came right after the counting at the sensory table she seemed to lose interest quickly. One suggestion might be to separate those activities so that they are not back to back. This will give her time to complete other activities in between and increase her interest level when she works on counting again. Just to follow-up on our conversation on Wednesday regarding embedding sequencing trials, you do have many natural opportunities to do this throughout your day. Anytime Mary completes a multi-step activity you could have her tell you how she completed the activity. I have observed you doing this with puzzles, art projects, and in dramatic play. Other activities that require a sequence of steps occur multiple times throughout the day. One strategy I used to use is having children tell me the centers they visited each day in the correct sequence. This is just one idea. It is just a matter of taking advantage of naturally occurring opportunities and this is something you do very well. You might try planning some activities with this focus. I will be in around 9:00 on Wednesday. I am planning to stay to meet with you at 1:15. I would like to discuss your progress on data collection and any other issues or concerns that you have.. You are doing a fabulous job. Please feel free to email or call if you have any questions. I look forward to seeing you next week. open the e-mail provide supportive performance feedback provide corrective performance feedback provide targeted support support planned actions end the e-mail

14

The Teaching Pyramid Observation Tool (TPOT) The TPOT was developed to measure the extent to which the Teaching Pyramid practices are being implemented in a classroom Provides information that can be used to identify training needs of teachers

Using the TPOT Observations Interviews Conducted for a minimum of 2 hours Must observe centers or free play and at least one teacher-directed activity Focus of observation is lead teacher’s behavior Interviews For those practices that cannot be observed in a 2-hour observation

Format of the TPOT Three types of Items Environmental items (Items 1-7) - yes/no based on observation Ratings of practices (items 8-22) - ratings based on observation and/or teacher report Red Flags (items 23-38) - yes/no based on observation

Environmental Arrangement Items - Clear boundaries - Move easily around room - Lack of large open spaces - Adequate number of centers - Materials support play - Preparation of centers - Classroom rules

TPOT Items Based Primarily on Observations Schedule and Routines Transitions between activities Teachers engage in supportive conversations with children Promoting children’s engagement Teaching children behavior expectations Providing directions Effective strategies to respond to problem behavior Teaching social skills and emotional competencies Teaching children to express emotions Teaching problem solving Supporting friendship skills

TPOT Items Scored Based on Teacher Report Supporting children with persistent problem behavior Communication with families to promote involvement Involving families to support social emotional development and addressing problem behavior Strategies to build collaborative teaching with other adults

Red Flag Items Represents issues related to teacher training, support, or program practices Indicates areas for immediate attention and training/support

Sample Item 11. Promoting children’s engagement Notes: 1 2 3 4 5 1 2 3 4 5 A classroom receives a score of “0” if all behaviors under a score of “1” are not observed __Y __N (1.1) Teacher offers general guidance to children to select activities or use materials to promote engagement* __Y __N (1.2) Teacher provides fun activities that will support the engagement of almost all of the class * __Y __N (1.3) Teacher communicates with children on eye level almost all of the time * __Y __N (3.1) Structured large-group (e.g., circle) activities are structured so that children are actively engaged (responding, interacting) almost all of the time __Y __N (3.2) Teacher assists individual children in selecting center activities and becoming actively engaged* __Y __N (3.3) Children are provided with multiple opportunities to make meaningful choices within activities (choosing center, choosing how to use materials, choosing where to sit, etc.) __Y __N (3.4) Teacher frequently comments positively on children who are engaged in activities __Y __N (5.1) Teacher assists individual __N/O children who are exhibiting problem behavior within an activity become actively engaged (score N/O if no opportunity to observe) * __Y __N (5.2) Teacher modifies instruction or activity when children lose interest in large- and small-group activities Notes:

11. Promoting children’s engagement Sample Item 11. Promoting children’s engagement 1 2 3 4 5 A classroom receives a score of “0” if all behaviors under a score of “1” are not observed __Y __N (1.1) Teacher offers general guidance to children to select activities or use materials to promote engagement* __Y __N (1.2) Teacher provides fun activities that will support the engagement of almost all of the class * __Y __N (1.3) Teacher communicates with children on eye level almost all of the time * __Y __N (3.1) Structured large-group (e.g., circle) activities are structured so that children are actively engaged (responding, interacting) almost all of the time __Y __N (3.2) Teacher assists individual children in selecting center activities and becoming actively engaged* __Y __N (3.3) Children are provided with multiple opportunities to make meaningful choices within activities (choosing center, choosing how to use materials, choosing where to sit, etc.) __Y __N (3.4) Teacher frequently comments positively on children who are engaged in activities __Y __N (5.1) Teacher assists individual __N/O children who are exhibiting problem behavior within an activity become actively engaged (score N/O if no opportunity to observe) * __Y __N (5.2) Teacher modifies instruction or activity when children lose interest in large- and small-group activities Notes:

Rochelle ENV = 7 Anchor= 2.87 Red Flags = 0

Rochelle ENV = 7/ 5 Anchor= .733 / 1.0667 Red Flags = 4 / 6

What have we learned about coaching? It takes a significant amount of coaching to get teachers to criterion levels of using the Pyramid practices Teachers need help knowing where to start in terms of implementing the Pyramid practices Teachers need help accessing and developing materials Getting teachers focus on the bottom of the pyramid can be difficult We may need a process for rolling out coaching

Self-Coaching

“Coaching” in Tools for Teachers Embedded Instruction for Early Learning Tools for Teachers (TfT) “Coaching” in Tools for Teachers Impact of Professional Development on Preschool Teachers’ Use of Embedded Instruction Practices Institute of Education Sciences Project Number R324A070077

Embedded Instruction for Early Learning: Tools for Teachers Who Preschool teachers in classrooms that include children with disabilities Certified teachers, at least 2 years experience What Embedded instruction Plan, implement, and evaluate embedded instruction practices with fidelity How High quality workshops and support materials Coaching with performance feedback in classroom Self-coaching with web-based supports

Two forms of coaching On-site Coaching Self-Coaching

Research Design Phase II: Feasibility study Multiple baseline across teachers 3 variations Phase III: Potential efficacy trial Preschool teachers randomly assigned to conditions Three conditions Business-as-usual (no intervention) Tools for Teachers workshops plus on-site coaching Tools for Teachers workshops plus self-coaching

www.embeddedinstruction.net

Getting Started: Watch videos that will show you how navigate and use the website. This will be a useful section for your to refer to as a refresher after this training session as you enter the self-coaching condition. Self-Coaching: Steps of the self-coaching process are described, and resources to help you self-coach to improve your embedded instruction practices are provided. Goals and Data: Connect to the Tools for Teachers Graphing Tool to plan your embedded instruction trials, track your implementation of trials, and monitor child progress. Refresh Your Memory: Revisit the materials used in the embedded instruction workshops. Troubleshooting: Offers tips and scenarios to improve your embedded instruction practice and assist you with problem-solving about embedded instruction. Video Library: Watch videos of examples and non-examples of embedded instruction practices. Other Resources: Gain access to other resources that will help you with embedded instruction. Help: Access technical support for the website.

Selecting Personal Goals

Creating an Action Plan

Self-Coaching Checklist

The Graphing Tool The graphing tool can be an excellent resource for embedded instruction. With the graphing tool you can: Plan embedded instruction and track data on multiple children. For each child, you can: Specify the learning targets and criteria that you are targeting. Plan how many trials you would like to embed and in which activities Print a system generated Activity Matrix . Input how many trials you actually implemented. Input child performance data for every trial implemented. The data you input into the system can be transformed into graphs. Graphs can be printed for your own records or to share in school team meetings and with parents. The graphing tool was designed with a variety of teachers in mind. The system was created for teachers to use the application flexibly. Some teachers may choose to use all aspects of the graphing tool (planning embedded instruction, entering and graphing data on teacher implementation of embedded instruction, entering and graphing data on child performance), while other teachers may choose to use parts of the application. Furthermore, teachers can use various parts of the application differently. We encourage you to use the application in any way that meets your needs.

Features of the Graphing Tool You will see this menu on the left hand side of the graphing tool. You will use this menu to access the different features of the application Under the schedule option there are activities already available for you to select from. Listed activities may have different names than terms you normally use in your classroom. If possible try to use the terms listed, however, you can also request that the project team add new terms to the list for you. The child roster is where you will keep information on your children’s profiles. This is where you will enter the goals and learning targets that you are currently working on with the child. It is also optional to specify criteria for the child’s performance. If you plan on tracking child progress through the website, then you must specify criteria to obtain this data. Under child roster you will also plan the activities in which you plan to embed trials. The active week function allows you to return to past weeks to make changes or review past entries. The trial menu is where you access trial plans and trial implementations: Trial plan is where you enter how many trials you plan to implement in the different activities per activity. You can do this for each learning target and for each child. After you have implemented your plan, you will return to the website and enter your implementation data. Trial Implementations is where enter how many trials you actually implemented in each activity for each learning target and also how the child performed in each of the trials. Report is where you will visit to generate tables and graphs on your implementation and the child’s performance. There are several ways that you can choose to display this information. Lastly, you would visit Settings to return to the start page, change your password, or sign out. When you first start using the graphing tool you will need set up your account. To do this you will set your daily schedule and enter all the needed information for the child Roster. We expect initial set-up will take between 30-50 minutes based on your experience with web-based applications. After the initial set-up for your account you can use the graphing tool weekly to update your embedded instruction planning and enter data that you have collected on trials implemented and child performance.

Daily Schedule In order to use the graphing tool, you must set your classroom schedule. Find “daily schedule” in the left menu. Simply click on “Add Activity to Daily Schedule,” and one-by-one, select all of the activities that occur during the course of the school week. Remember that some activities have different names, and so the term provided may not be the one you normally use in your classroom. If you find that none of the categories apply to something that is included in the schedule, through the website, you can request that the project team add it to the list for all of the teachers.

Child Roster The child roster is where you will enter all of the children whose progress on embedded instruction you would like to track. At the top of the page, click on “Add a new child.” You will be asked for the child’s name and the number of days attend school a week.

Adding New Learning Targets Just as you have learned in the workshop, you will be breaking down your child’s goals into learning targets that can be embedded in your classroom. After entering all of the child’s goals, return to the main child roster page and click on “Child Learning Targets.” Learning Targets will be associated with selected goals. Similar to what you did for the child’s goal, click on “Add new child learning target.” On this page, you must select the goal that corresponds to that learning target. The goals will be listed by the goal nickname. You must also create a learning target nickname. After entering the learning target description, you have the option to specify the child’s performance criteria. For the purpose of this website, we have categorized criteria by color. You can select what each color stands for, but these are the colors that will correspond to the graphs that are generated. When using this option, you do not have to use all of the color categories provided. Trainer: refer participants to Color Coded Criteria information in the help section of the graphing tool for more information on using the color coded system.

Trial Plan When you click on “Trial Plan,” you will first see a list of the children in your classroom. You must click on “Trial plan for child” to create and access the activity matrix. After selecting the child you are planning for, select a learning target that you plan to embed. At the top of the page, click “Add new activity to the trial plan.” Select one of the activities you plan on embedding trials, and then enter the number of daily trials that you would like to plan. The activity options are limited to those that you specified when entering your learning target in the child roster. Repeat this process for each activity within each learning target. When you have finished entering all of your trial plans for the week, return to “Trial plan for child” and click on “View Activity Matrix” to view your child’s activity matrix for the week. A print button is located at the bottom of the page, if needed.

Trial Implementation At the end of the week, you can then enter the outcome of your plan in order to generate tables and graphs for reports. To access enter data for each child, begin by clicking on “Trial Implementation for Child” then “Trial Implementation for Child’s Learning Target.” You will be brought to a page that contains all of the information on the learning target and reminders of the descriptors for the color criteria. Click on “Add new trial implementations for child’s learning target” For each activity that you have planned for, enter number of trials that you actually implemented. Including this information will generate reports that compare your number of plan trials vs. implemented trials to monitor your progress. After you have entered this information, you have the option to specify the child’s performance for each trial. For each trial, enter the color that corresponds to the child performance. The descriptors are provided on the page as reminders. This information is needed in order to generate reports on the child’s performance in response to embedded instruction. Let’s look at some of the graphs you can generate.

Teacher Performance Report Looking at this graph, you can compare the number of planned vs. implemented trials for each learning target. This graph can be used as your self-evaluation phase of self-coaching. You can use this graph to examine whether you are meeting your planned performance. This graph indicates learning targets that you may be having difficulty with. For the following week, you are now aware that you need to increase the number of trials for a particular goal because you are not following through as well as you are in comparison to other goals. Remember, this can impact the child performance as well.

Reports for Child’s Learning Target This is one of the examples of the type of report that you can generate to show the child’s performance. The types of graphs that are available are: Performance Report: This option generates a report the child’s overall performance on a single learning target for the active week. Performance by Activity Report: This report breaks down the child’s performance on a single learning target by the planned activities. You can see how the child is performing on a learning target across the entire day. Cumulative Performance Report: This is similar to the performance report, but totals the child’s progress from the first week you collected data to the current active week. Cumulative Performance by Activity Report: This report breaks down the child’s performance on a single learning target by the planned activities. You can see how the child has been performing on a learning target since you began collecting data. Progress Report: This option allows you to look at the child’s performance across week. This is the best option to use to illustrate the child’s improvement over a period of time. The graphs are color coded according to the colored criteria you specified for the learning target. When you print out the graph, a description of the goal, learning target, and criteria are included.

“Using the self-coaching process you have the tools for improving your practice, however, it is up to you to take the time to use the tools.”

What are we learning about on-site and self-coaching coaching? It may be important for teachers to have some experiences with on-site coaching before self-coaching. It takes time for teachers to learn embedded instruction practices and implement them with fidelity. The importance of a foundational curriculum in the classroom and the development of functional and generative learning targets.