Amid New, Different, and Traditional Ways of Knowing: Bringing a Networked Improvement Approach to Scale in EdD Curriculum Assessment Joy C. Phillips, PhD Kathy D. Geller, PhD Kenneth J. Mawritz, PhD

Session Learning Objectives Illustration of Improvement Science (IS) Usage Specifically, Networked Improvement Communities (NIC) As mechanism to facilitate EdD program curriculum assessment and continuous program improvement. (Bryk, Gomez, & Grunow, 2011; Bryk, Gomez, Grunow, & Lemahieu, 2015; Russell et al., 2017) Demonstration of Lewin’s (1951a) force field analysis method To guide faculty conversations About curriculum assessment and improvement. | 2

Session Overview Introduction 5” Application of NIC to Our Work 5” Overview of Our Work 10” Use of force field analysis 20” Table discussions 20” Full group reflections 15”

Networked Improvement Community (NIC), Defined “An intentionally designed social organization with a distinctive problem-solving focus; roles, responsibilities and norms for membership; and the maintenance of narratives that detail what they are about and why it is important to affiliate with them” (Bryk, Gomez, & Grunow, 2011)

NIC: Essential Elements 1) Focused on a well specified common aim, 2) Guided by deep understanding of the problem, the system that produces it, and a shared working theory on how to improve it, 3) Disciplined by the rigor of improvement science, and 4) Coordinated to accelerate the development, testing, and refinement of interventions, their rapid diffusion out into the field, and effective integration into varied educational contents. | 4

Our Work Drexel University School of Education EdD faculty have engaged in a comprehensive program assessment process that: Expands existing program learning outcomes, Keystones, into mastery statements, Aligns the EdD program with national, regional, and university standards, Identifies measurable formative and summative outcome data/evidence, Describes/proposes how measurable data/evidence wil be reviewed, Describes/proposes reporting and follow-up processes.

Description: Our Process Call for EdD Advisory Committee Faculty, Jan 2017, to conduct Prep work for CAEP visit; Volunteer work team, 3 Clinical Associate Professors; Worked via web conferencing and email from 3 states (NC, CA, PA) from Jan 2017-to-date; Worked in an organic fashion to respond to SoE template; Engaged in rich discussion, critical reflection, questioning, debate, reconsideration, finally agreement; From tacit to explicit norms: shared understanding that the group would work collaboratively, honor each other’s contributions respectfully, and allow unlimited “push-back” from any member on any point at any time. SECTION TITLE | 2

EdD Program Learning Objectives: Keystones EdD graduates possess the abilities to create and support communities that are the bases for sustainable change Leaders develop the habits of mind and competencies to lead complex organizations shaped by global forces Leaders develop the abilities to sustain their own leadership growth EdD graduates utilize the full range of emerging technologies to reach across generations, communicate effectively, and engage others in meaningful change EdD graduates exemplify the curiosity, inquiry skills, and scholarly competencies needed to investigate an idea and transform it into meaningful action. HEADLINE SECTION TITLE | 2

Program Assessment Planning Tooling Tool School of Education: Program Assessment Planning Tooling Tool Program Learning Outcomes (PLOs) DSLPs/ Standards/ SoE Themes Data/Evidence Review of Evidence Reporting and Follow-up WHAT are the PLOs? PLOs are learning expectations, expressed in active, measurable terms WHICH [university] Student Learning Priorities (DSLPs), Standards (InTASC, CAEP, SPA) and SoE themes align with each outcome? (See list below) WHAT data or evidence will be used to determine whether or not the outcome is met and/or level of impact on P-12 learning? Provide: --Type of measure --Type of evidence --# of students assessed --Where in the curriculum or program is the outcome assessed? --When and how often will the outcome be assessed? HOW will program data/ evidence be reviewed? Identify: --the evaluation tool (rubric; evaluation summary form; statistical tool) --performance target, achievement level, or benchmark --Who will review the evidence? --What will be the process? --When and how often will the evidence be reviewed? WHAT is the plan for reporting the results and findings (strengths and gaps)? I dentify: --To whom will the quantitative and/or qualitative results be reported? How often? -- Who will be responsible for reporting the results and findings? For tracking the follow-up? SECTION TITLE | 2

Articulated Program Learning Outcome (PLO) for EdD Keystone 2a Program Learning Outcomes Standards: CAEP, CPED, DSLPs, SOE Themes Data/Evidence Review of Evidence Reporting and follow-up Keystone 2:   Leaders develop the habits of mind and competencies to lead complex organizations shaped by global forces. CAEP 1 Content & Pedagogical Knowledge 2 Clinical Partnerships & Practice 4 Program Impact 5 Provider Quality Assurance & Continuous Improvement EdD students participation/performance as evidenced by the following: Utilize systems thinking to address leadership standards through knowledge base development and application (EDUC 800, 845, 801, 802) Leadership: individual and team projects Theory U projects Create an e-portfolio of student academic and professional accomplishments (e.g., writing assignments, sample discussion boards, team projects, course activities, IRB approval letter, SOE/GS funding letters, juried conference presentations, community presentations, juried publications, etc. ) Proposed - Create a review process for the e-portfolio.

Articulated Program Learning Outcome (PLO) for EdD Keystone 2b Program Learning Outcomes Standards: CAEP, CPED, DSLPs, SOE Themes Data/Evidence Review of Evidence Reporting and follow-up Learning Outcome: Demonstrate mastery for leading systemic change drawing from a range of theoretical frameworks: (a) Systems Thinking (b) Leadership & Adaptive Leadership & Theory U (c) Creative Problem Solving and Design Thinking (d) Communication CPED 1 Ethics/Social justice 2 Positive Difference 3 Collaboration/ communication Theory w/systemic inquiry DSLP 1 Communication 2 Creative & Critical Thinking 4 Information Literacy 6 Technology Use 7 Global Competence 8 Leadership (b) Identify & develop leadership style (EDUC 800, 845, 801) Academic papers & team projects Technology tools (c) Creative Problem Solving & Design Thinking (EDUC 800, 804, 845, 801, 802, 803, 810, 818) Utilize leadership and research to design dissertation Comprehensive examination rubric Leadership; Action oriented research and evaluation; Keystones, Writing and APA as scored independently by two professors with a rating of 2.0 (scale 0 – 3) or higher on all categories   CITI Certification Create a central repository for CITI Certifications (obtained during EDUC 810) Semi-annual report (Fall/Spring) Comprehensive Examination outcomes by Program Director Program Manager provides annual listing of CITI certifications for active students

Articulated Program Learning Outcome (PLO) for EdD Keystone 2c Program Learning Outcomes Standards: CAEP, CPED, DSLPs, SOE Themes Data/Evidence Review of Evidence Reporting and follow-up Keystone 2:   Leaders develop the habits of mind and competencies to lead complex organizations shaped by global forces. SOE 1-Innovation & Creativity SOE 2-Diversity SOE 5-Leadership (d) Effective communication principles (EDUC 800, 845, 801, 802, 802) Academic papers & team projects Discussion boards & voice threads (e) Summative evidence includes the following: Comprehensive CITI Dissertation Proposal Final Dissertation Dissertation Proposal Defense Committee Approval (Forms D-2 & D-2a) Dissertation Final Defense (Forms D – 3 to D- 5) Dissertation Approval Form Completion Form Work with HRPP to create IRB approvals annual report for EdD research Create (from D2) Annual report by quarter on doctoral proposal Create (from D5s) Annual report by quarter on doctoral defenses approved

Lewin’s Force Field Analysis SECTION TITLE | 2

Step 1: Working as a Team Identify each driving force and each restraining force in your particular situation Driving Forces Restraining Forces CAEP Accreditation CPED Alignment Standards Leadership Team Effort Program Excellence Effective Marketing Faculty Time Faculty Knowledge - EdD Program Difficult Conversations Resistance to Change Create New Reporting Systems Data Analysis Time Philosophical Differences – EdD and PhD

Step 2a: Working as a team Provide a “brief” definition for each driving force Driving Forces CAEP – Council for the Accreditation of Educator Preparation CPED – Carnegie Project on the Education Doctorate: Alignment Activity Standards – University, Regional, State, and National Norms Leadership Team Effort – Steering Committee to Design and Facilitate Process Program Excellence – Scholarly Commitment to EdD Mission Effective Marketing – Design a Dynamic Marketing Approach

Step 2b: Working as a Team Provide a “brief” definition for each restraining force Restraining Forces Faculty Time – Window of Time and Responsibilities Faculty Knowledge of EdD – Expectation(s) Awareness of the EdD Program Design Difficult Conversations – Time and Willingness to Discuss the EdD Program Change – Resistance Associated with a Change in Heart, Mind, & Will Create New Reporting Systems – Designing New Systems to Provide Evidence Data Analysis Time – A Staff Window of Time Opportunity to Provide Evidence Philosophical Differences – a Confluence of PhD, EdD, Personal Beliefs, & 21st C Ideas

  Step 3: Working as a Team Identify the priority for each driving force by assigning a percentage to it. Initially, the total percentage may be over 100%. Identify the priority for each restraining force by assigning a percentage to it. Initially, the total percentage may be over 100%. Ultimately, ensure that the calculation of each set—driving and restraining forces—equals percentages 100%.

Table 1: Sample Percentages Driving and Restraining Forces Driving Forces % Restraining Forces CAEP Accreditation 22 Faculty Time 24 CPED Alignment Faculty Lack of Knowledge of the EdD Program 16 Standards 6 Difficult Conversations 15 Leadership Team Effort Resistance to Change 19 Program Excellence 18 Create New Reporting Systems 3 Effective Marketing 8 Time for Data Analysis (Existing Data and Emerging Data) 4   Philosophical Differences in EdD Program Intent Total 100 % SECTION TITLE | 2

Step 4: Working as a Team Design a priority pie to symbolize the driving force priorities Design a priority pie to symbolize the restraining force priorities

Priority Pies: Comparison Driving Forces Restraining Forces

Step 5: Working as a team, “next steps” include the following: Analyze the priorities established in the driving force column and the restraining force column Considering this analysis, design a dynamic ongoing plan to review all aspects of the Force Field Priority Protocol   Step 5 provides the launch for effective next steps. Good Luck!

Table Talk If attending as a team, work together. Using the Force Field Analysis Process Identify your EdD program’s driving and restraining forces. Discuss at your table various institutional/program response.

Reflection Questions? Want to discuss further? Joy Phillips joy.phillips@drexel.edu Kathy Geller kdg39@drexel.edu Ken Mawritz kjm97@drexel.edu