Jacklyn Altuna, M.Ed. Hannah Betesh, M.P.P.

Evaluation Context  Multi-year, random-assignment evaluation of a teacher professional development program that consists of an impact study and an implementation study  Focus of this presentation: Data from Implementation Study  Larger study focuses primarily on quantitative data  3-year study  52 middle schools randomly assigned to either:  treatment group: teachers receive professional development  control group: business as usual  Role of qualitative data in the context of an experimental evaluation  Fidelity to model  Describe implementation context  Treatment Contrast 2

Components of Professional Development Intervention  7-Day Summer Institute  4 Individualized Coaching Sessions per year  4 Afterschool Teacher Collaboration Meetings 3

Research Questions  Fidelity to Model  Is the intervention being implemented as planned?  What are the variations and adaptations to the implementation of this professional development model?  Does implementation vary across schools and districts? If so, why and how?  What are the implications of the variations on program outcomes?  Implementation Context  What conditions support or hinder the implementation of this professional development program?  Treatment Contrast  What is the treatment condition being compared to? 4

Data Sources PROFESSIONAL DEVELOPMENT COMPONENT DATA SOURCES Summer Institute(1)Detailed field notes using semi- structured observation protocol (2)Interviews with program developers (3)Teacher focus groups Individualized Coaching(1)Coaching records (2)Interviews with coaches (3)Teacher focus groups Afterschool Teacher Collaboration Meetings (1)Detailed field notes using semi- structured observation protocol (2)Interviews with coaches (3)Teacher focus groups 5

Data Collection Process Focus Groups & Interviews  Teacher focus groups were conducted:  In-person, once a year by two researchers where possible  In both treatment and control schools  Tape recorded and transcribed  Program developer and coaching interviews conducted:  Over the phone, twice a year  Tape recorded and transcribed Direct Observations  All in-person observations conducted by two researchers taking detailed field notes via laptop using semi-structured observation protocol  Observation training conducted by senior researcher  All write-ups reviewed for quality and clarity by senior researchers; revised as needed Coaching Records  Written and developed by coaches as documentation of each coaching session with teachers  Sent directly to BPA bi-monthly 6

Observer and interview training as a way of increasing rigor in qualitative research Observer Training 1. Apprenticeship model: Junior researchers observe with senior researcher 2. Senior researcher reviews protocol with team to discuss expectations with regard to level of detail, provides examples for each item in observation protocol, discuss relationship of notes to write-up and larger study’s research questions Interview/Focus Group Facilitator Training 1. Senior researcher goes over protocol and discusses probes, types of responses, relationship of question to larger study’s research questions 2. Apprenticeship model 7

Data Cleaning Process Focus Groups & Interviews  All transcriptions reviewed and revised by interviewer/facilitator for accuracy Direct Observations  All raw notes and write-ups reviewed for quality and clarity by senior researchers; revised as needed  Templates and write-ups formatted for auto-coding  Use of heading styles (e.g. Heading 1, Heading 2, etc.) Coaching Records  Formatting revised for easy input into Nvivo Qualitative Analysis software  Tables removed from template 8

Nuts & Bolts of Our Team Based Data Analysis Process 1. Reviewing data individually 2. Identified broad themes 3. “Chunked” data as a team using auto-coding 4. Conducted inductive coding 5. Participated in co-development of codebooks 6. Team-based coding and negotiation of codes and hierarchies 7. Created a walkway between coding hierarchies and research questions 8. Co-development of analytic memos 9

Team Coding & Multi-stage Analysis: A Look at the Afterschool Teacher Collaboration Meeting Data Set 10

Lesson Design Meeting Data Set  Data Sources:  Notes from 5 observations of after school teacher collaboration meetings  Transcripts from of each lesson design meeting facilitator  Focus on content, connections to classroom, teachers ’ issues and concerns, engagement, facilitator qualities, successes and challenges of implementation, and variations in implementation. 11

Nvivo Jargon  Reasons for choosing Nvivo  Nodes correspond to a topic or category  Two types of ‘ nodes ’ : ‘ free nodes ’ and ‘ tree nodes ’  Free Nodes are dropping off points for data and do not presume any relationships or connections.  Tree Nodes are hierarchical structures that organize that free nodes. 12

Becoming familiar with data set  Step 1: Become familiar with data set individually and as a team, discuss emerging themes.  Year 1 – unraveling the “black box” – exploratory  Read raw notes and transcripts  Have a short team meeting to debrief and discuss 13

Reorganize Notes & Transcripts Into Broad Topic Areas  Step 2: Reorganize text into broad topic categories based on observation and interview protocols and emerging themes from team discussion (one person).  Revisit protocols and insert salient topic categories as headings into raw notes and transcripts  Use paragraph headers (more on this later) for auto- coding 14

Reorganize Notes & Transcripts Into Broad Topic Areas 15

Auto Code in Nvivo  Step 3: Auto-code to “chunk” the data into more manageable pieces  Automatically codes all text under the topic area heading  Lets you view the data by category without disturbing original text  Is linked back to original data source; toggle to get context 16

Teacher Collaboration Observation Topic Categories  room arrangements /logistics  notes on participants  narrative/script of the meeting  connections to the classroom/standards  teacher engagement, satisfaction, and morale  issues and concerns raised by the teachers  notes on presenter qualities  anything else interesting, important, or out of the ordinary  post-interview notes 17

18 Teacher Collaboration Observation Topic Categories

Delving Deeper Into the Text: Assigning Free Codes’  Tree Codes  Step 5: Examining text attached to each categories across all the pieces of data (notes and transcripts) Read through each node from the auto coding and create child nodes or free nodes as necessary Each member assigned to their own nodes to read through and summarize 19

Delving Deeper Into the Text: Assigning ‘Free Codes ’ 20

Delving Deeper Into the Text: Assigning ‘Free Codes’ 21

Analytic Memo Structure 1 1. Overview of Lesson Design Meetings  A. Structure and format  B. Goals & vision for lesson design meetings  C. Attendance  D. Notes on participants 2. Level of Engagement 3. Connections to classroom standards 4. Challenges in Implementation 5. Successes in Implementation 6. Modifications to Implementation 7. Perspectives  A. Coaches’ perceptions  B. Teachers’ perceptions 22

Taking A Step Back: Restructuring the Code List Two members of the research team coded independently, generating two sets of free nodes Step 6: Identifying themes that emerged from the codes and re-grouping codes into logical, hierarchical tree structures 23

Analytic Memo Structure 2 1. Overview of Lesson Design Meetings  A. Structure and format  B. Goals & vision for lesson design meetings  C. Attendance 2. Contextual Issues  A. Barriers in implementation  B. Successes in implementation 3. Modifications for Future Implementation  A. Coaches ’ Responses to Varying Needs, Opportunities and Limitations  B. Modifications for future implementation 4. Perspectives 24

Restructuring the Code List Step 7: Taking a step back - restructure coding hierarchies to respond to research questions  Restructure analytic memo 25

Analytic Memo Structure 3 1. Fidelity to Model  A. Structure and format  B. Goals & vision for lesson design meetings  C. Attendance 2. Implementation Context  A. Barriers in implementation  B. Successes in implementation  Perspectives 3. Modifications for Future Implementation  A. Coaches ’ Responses to Varying Needs, Opportunities and Limitations  B. Modifications for future implementation 26