James Hendricks Denise Skarbek Mind the Gap The Need for Quality Management of Assessments that Measure Teacher Candidates’ and Completers’ Impact on PK-12 Student Learning James Hendricks Denise Skarbek

Applying DMAIC model to teacher education programs New Director of Program Approval, July 2015 Small, private education department State approved Seeking model to determine effectiveness of teacher education program Review model during data summit AACTE promotes quality

Bronfrenbrenner’s ecological model Macro Exo Meso Micro Accreditation Body (Quality Assurance) Employer Satisfaction Teacher Satisfaction Child-Teacher Candidate Child-Teacher Completer Teacher Completer-impact on student-university response, community response

Micro Level and DMAIC Micro Meso Exo Macro Define: Measurements: Analysis: Improve: Control: Micro Meso Exo Macro

DMAIC Model Define Measure Analyze Improve Control A Data-driven, customer-focused, structured problem-solving framework which builds on learning from previous phases to arrive at a permanent solution for difficult problems

DMAIC Project Problem Definition: Addresses a teacher preparation program issue. The assessment issue must be laser focused--micro level—teacher completer and PK-12 student. Goals: SMART goals are directly tied to a well-defined set of metrics (APPR) Business Benefit: The project should culminate in a measurable cost, schedule or quality benefit (quality APPR) Tools: data/information analyses Roles and Responsibilities/Timelines: for each team member

Project Form

Define. Identify problem that is not too large that you believe you can achieve an improvement that has an impact on the outcome. The customer for higher education, in this case study, is the teacher candidate/teacher completer. The SIPOC model means is an acronym for suppler/input/process/output/customer. You describe your current process of assessment in teacher education for the process. In other words, you identify who the supplier is that is the university and other major stakeholders. This is typically determined by the accreditation body such as CAEP or your state requirements for accreditation. Next, define your customer, the teacher candidate, the school district where they are employed and the accreditation body. This means you will always have more than one supplier in the process and you will need to define them. The process is the positive impact on P-12 student. Certain factors must be taken into consideration such as how the teacher candidate is being trained, for example, online or face-to-face a team or committee must be established to determine this. The outputs are teacher completer’s impact on p-12 student learning. The teacher completer must ensure that the k-12 student successfully completes the state exams. What are the drop-out rates or retention for teacher completers. Always a living model and can change but need to have an understanding of what is currently being done to be able to assess and determine outcomes and what changes, if any, need to occur. The university that is involved in this process is doing a data days that will include part of the process of quality management and which is also espoused from AACTE. For example, we are looking at evaluation of effectiveness of our teacher education programs in any area that was not at proficient or excellent level. Measurement is to collect the scores and review the trends and patterns over a period of time and look at how the trends were effected. What other measurements can you find. Can you examine the faculty that are teaching the courses. For example, did you have a faculty that did not perform proficient in student evaluations. Look at semester trends are the classes effected? Examine data in different ways. Training faculty on how to examine to data will be occurring during the data days. This will provide the faculty an opportunity to look at this data and determine what other types of data to determine what is going on? Are classes to easy to hard? What are curriculum trends that are aligned to the standards and determining where to measure and analyze and that you have an “ah ha” moment. Throughout this process it is never to blame a person or point to a person. Cost analysis-determining how much time is spent with students can be less time spent on preparing. Less students wanting to come into teacher preparation programs many universities have experienced this. Because of the market loss wanting to have students with lower GPAs entering the program instead of making it competitive and desirable to enter into the program. Administration wanting to do more with failed and retention of students so faculty who serve as academic advisors are spending more time working with the students. Improvement: Use different measurement tools such as strengths, opportunities and weakness to help determine what types of risk factors may be

Supplier(s) = University Inputs = Teacher education program Process = how the teacher completers were trained (online-face-to- face), where are the field training, what types of diversity in the school Outputs = performance of teacher completer, how and what they teach Customers= k-12 student, employer, and FL DOE

Cause and Effect Diagram

5 Whys – An Approach to Finding Root Cause The 5 Whys can be used individually or as a part of the fishbone diagram. The fishbone diagram helps you explore all potential or real causes that result in a single failure or defect. Once all inputs are established on the fishbone, you can use the 5 Whys technique to drill down to the root causes. Example: Problem Statement: On your way home from work your car stops in the middle of the road. 1. Why did your car stop? – Because it ran out of gas. 2. Why did it run out of gas? – Because I didn’t buy any gas on my way to work. 3. Why didn’t you buy any gas this morning? – Because I didn’t have any money. 4. Why didn’t you have any money? – Because I lost it all last night in a poker game. 5. Why did you lose your money in last night’s poker game? – Because I’m not very good at “bluffing” when I don’t have a good hand. As you can see the final Why leads the team to a root cause that the team can take action upon. It is much quicker to come up with a system to teach a person to “bluff” a hand than it is to try to directly solve the stated problems above without further investigation.

Pareto Diagram Example

Other Considerations Data and Information must be accurate, reliable, relevant and useful Use a Program Analyst for data analysis (visual transformation of data) Laser focus the scope for problem solving – can’t solve “world peace” but should scope for project completion in 3 to 6 months Use a Facilitator familiar with Problem Solving techniques Have administration champion the project – periodic reporting to ensure progress and support Typically, there is a cultural business bias toward achieving the biggest goals in the shortest time (quick fix). Successful results of a manageable project will be the impetus for further projects. The project does not end with the implementation of the solution. Constant monitoring and re-addressing issues is required to maintain control. (The DMAIC process is a continuous cycle)

Micro Level and DMAIC Micro Meso Exo Macro Define: Child-Teacher Candidate and Child Teacher completer Measure: Observations Valid/Reliable Must use district observation form (cost analysis-having teachers serve as university supervisors) use the district evaluation form Analysis: Cost effectiveness and data Improve: Control: Micro Meso Exo Macro