EVALUATION TO IMPROVE—AS WELL AS PROVE Susan Shevitz & Larry Bailis Prepared for LICSE Symposium February 25, 2010

TECHILAH” (“The end of the action begins with the thinking”) “SOF MA’ASEH B’MACHSHAVAH TECHILAH” (“The end of the action begins with the thinking”)

WHILE STANDING ON ONE FOOT Most people think of evaluation as an after-the-fact effort to PROVE that something worked (or didn’t) We argue that evaluation can also be used as a way to IMPROVE programs if they are used as a before-the-fact planning tool

EVALUATION . . . SHOULDN’T BE “Gotcha” Pro-forma Haphazard Divisive Painful SHOULD BE What works What actually helps Planned & well timed Team building “Facts are our friends”

EVALUATION AS A WAY OF THINKING/PLANNING What do we want to happen as a result of our lesson, activity, program, curriculum, policy, school? Over the short, medium, long-term? How can we reliably learn whether we have achieved these goals?

TODAY’S GOALS Understanding a logic model Generating questions and approaches for evaluation from the logic model Trying to do a logic model on something from your schools Exploring ways LICSE evaluation has influenced (and is influencing) professional leaders Receiving a some practical tools

LICSE AS A CASE STUDY

OUR ASSERTION Developing a logic model is a good way to sharpen up planning processes as well as prepare for evaluation Developing a logic model is thus a good way to start im-proving as well as proving

SO– WHAT IS A LOGIC MODEL? A logic model provides a concise visual representation of activities that are the core of your program and how they fit together. Logic models consist of: (a) inputs, (b) activities, (c) output, (d) intermediate outcomes, and (e) end outcomes.

WHAT DOES A LOGIC MODEL LOOK LIKE? F O R W H M → A S U P T I N C V E SHORT-TERM LONG-TERM

FOR WHOM

FOR WHOM Full-time principals of Reform, Conservative, and Reconstuctionist congregational schools in the NY metropolitan area

ASSUMPTIONS- THEORY OF CHANGE

ASSUMPTIONS- THEORY OF CHANGE A well-designed 2 year prof. development program can provide principals with the knowledge, skills, attitudes, and understanding of leadership necessary to make the changes that are needed to revitalize congregational education The knowledge, skills, attitudes, and understandings will be sufficient to allow principals to forge partnerships with other congregational constituencies in order to develop strategies to overcome the serious challenges that prevent congregational education from functioning as intended and achieving its desired outcomes

ASSUMPTIONS- THEORY OF CHANGE LICSE leaders have the time and resources to adequately plan and implement each component of the program The process of promoting change in schools can be carried out quickly enough so that some desired outcomes will become visible within a year or two of the completion of the program

STRATEGIES/ACTIVITIES

STRATEGIES/ACTIVITIES An integrated series of professional development in Judaics, pedagogy and curriculum, and leadership presented through different modalities,including: Retreats, two summer seminars, ten day Israel seminar, twelve one-day symposita Individual and group mentoring Individualized components such as 360 degree assessment, individualized learning plans , action research projects, site visits, access to on-line resources, and follow-up support

STRATEGIES/ACTIVITIES Careful ongoing review and assessment of the way the program is implemented and its outcomes [to make room for mid-course corrections]

measurable by the end of the study (or some other short-term target) OUTCOMES measurable by the end of the study (or some other short-term target)

OUTCOMES Principals participate in LICSE activities and find them helpful Principals demonstrate achievement of sufficient Judaic knowledge and understanding of pedagogy to become leaders in reforming their schools’ curricula and supporting their teachers’ professional performance and growth

OUTCOMES Principals develop sufficient leadership skills to forge working relationships with key constituencies and to diagnose the strengths and weaknesses of the education program at their schools Principals work collaboratively with their constituencies to develop and implement meaningful strategies to promote needed changes

OUTCOMES Principals work collaboratively with their constituencies to formulate plans for school change and to take the first steps to achieving the changes

(long-term goals that begin to be achieved by the end of the study) IMPACT (long-term goals that begin to be achieved by the end of the study)

IMPACT Principals will become forces for revitalizing congregational education Changes will be made that make congregational educ. the kind of experience that lay leaders, educators, and parents hope it can be

IMPACT High quality Jewish education becomes a trademark of the participating congregations and a reason why people choose to them Young people will grow up with a love for and understanding of their Jewish heritage and want to live meaningful Jewish lives

TO EVALUATE Evidence Methodology/Instrumentation What is acceptable? What is reasonable? Methodology/Instrumentation What makes sense? (Time, budget, context, etc.) Will it get the information that is needed? What instruments are best used?

FROM LOGIC MODEL TO EVALUATION EVIDENCE OF HOW EACH COMPONENT AND THE INTERACTION AMONG THE COMPONENTS AFFECT THE PROFESSIONAL PRACTICE OF FELLOWS METHODS That will get the right info; Qualitative / Quantitative Surveys, inter- views, participant observation, focus groups

Symposia Summer Seminars Israel Seminars How are they working? Helping to move towards outcomes, goals Materials, observation, discussion, feedback survey, interviews,survey Summer Seminars Israel Seminars

Mentoring 360 Contact with thinkers / leaders

Special Tools: IMAPs, IAPs Intentional Community / Peer Network Intentional Community / Relationships w Staff & Faculty

ISSUES IN APPLYING THIS FRAMEWORK Q: What kind of evidence is acceptable? A1:This is not a PhD dissertation, and so the answer is that it needs to be persuasive not perfect. A2:”The best you can, given time and resource constraints”

ISSUES IN APPLYING THIS FRAMEWORK Q: What kind of evidence is acceptable? A3: Much of it is documentation, were the right people served, did assumptions pan out, was curriculum delivered as planned. A4: Off-the-shelf or do-it-yourself assessment tools

SO HOW DOES IT REALLY WORK?? QUESTIONS from group REFLECTIONS from Evie and group about LICSE experience: how information is used NEXT STEPS