1. EVALUATION TO IMPROVE—AS WELL AS PROVE
Susan Shevitz & Larry Bailis
Prepared for LICSE Symposium
February 25, 2010

2. TECHILAH” (“The end of the action begins with the thinking”)
“SOF MA’ASEH B’MACHSHAVAH
TECHILAH” (“The end of the action begins with the thinking”)

3. WHILE STANDING ON ONE FOOT
Most people think of evaluation as an after-the-fact effort to PROVE that something worked (or didn’t)
We argue that evaluation can also be used as a way to IMPROVE programs if they are used as a before-the-fact planning tool

4. EVALUATION . . . SHOULDN’T BE “Gotcha” Pro-forma Haphazard Divisive
Painful
SHOULD BE
What works
What actually helps
Planned & well timed
Team building
“Facts are our friends”

5. EVALUATION AS A WAY OF THINKING/PLANNING
What do we want to happen as a result of our lesson, activity, program, curriculum, policy, school?
Over the short, medium, long-term?
How can we reliably learn whether we have achieved these goals?

6. TODAY’S GOALS Understanding a logic model
Generating questions and approaches for evaluation from the logic model
Trying to do a logic model on something from your schools
Exploring ways LICSE evaluation has influenced (and is influencing) professional leaders
Receiving a some practical tools

7. LICSE AS A CASE STUDY

8. OUR ASSERTION
Developing a logic model is a good way to sharpen up planning processes as well as prepare for evaluation
Developing a logic model is thus a good way to start im-proving as well as proving

9. SO– WHAT IS A LOGIC MODEL?
A logic model provides a concise visual representation of activities that are the core of your program and how they fit together.
Logic models consist of: (a) inputs, (b) activities, (c) output, (d) intermediate outcomes, and (e) end outcomes.

10. WHAT DOES A LOGIC MODEL LOOK LIKE?F O R W H M → A S U P T I N C V E
SHORT-TERM
LONG-TERM

11. FOR WHOM

12. FOR WHOM
Full-time principals of Reform, Conservative, and Reconstuctionist congregational schools in the NY metropolitan area

13. ASSUMPTIONS- THEORY OF CHANGE

14. ASSUMPTIONS- THEORY OF CHANGE
A well-designed 2 year prof. development program can provide principals with the knowledge, skills, attitudes, and understanding of leadership necessary to make the changes that are needed to revitalize congregational education
The knowledge, skills, attitudes, and understandings will be sufficient to allow principals to forge partnerships with other congregational constituencies in order to develop strategies to overcome the serious challenges that prevent congregational education from functioning as intended and achieving its desired outcomes

15. ASSUMPTIONS- THEORY OF CHANGE
LICSE leaders have the time and resources to adequately plan and implement each component of the program
The process of promoting change in schools can be carried out quickly enough so that some desired outcomes will become visible within a year or two of the completion of the program

16. STRATEGIES/ACTIVITIES

17. STRATEGIES/ACTIVITIES
An integrated series of professional development in Judaics, pedagogy and curriculum, and leadership presented through different modalities,including:
Retreats, two summer seminars, ten day Israel seminar, twelve one-day symposita
Individual and group mentoring
Individualized components such as 360 degree assessment, individualized learning plans , action research projects, site visits, access to on-line resources, and follow-up support

18. STRATEGIES/ACTIVITIES
Careful ongoing review and assessment of the way the program is implemented and its outcomes [to make room for mid-course corrections]

19. measurable by the end of the study (or some other short-term target)
OUTCOMES
measurable by the end of the study
(or some other short-term target)

20. OUTCOMES
Principals participate in LICSE activities and find them helpful
Principals demonstrate achievement of sufficient Judaic knowledge and understanding of pedagogy to become leaders in reforming their schools’ curricula and supporting their teachers’ professional performance and growth

21. OUTCOMES
Principals develop sufficient leadership skills to forge working relationships with key constituencies and to diagnose the strengths and weaknesses of the education program at their schools
Principals work collaboratively with their constituencies to develop and implement meaningful strategies to promote needed changes

22. OUTCOMES
Principals work collaboratively with their constituencies to formulate plans for school change and to take the first steps to achieving the changes

23. (long-term goals that begin to be achieved by the end of the study)
IMPACT
(long-term goals that begin to be achieved by the end of the study)

24. IMPACT
Principals will become forces for revitalizing congregational education
Changes will be made that make congregational educ. the kind of experience that lay leaders, educators, and parents hope it can be

25. IMPACT
High quality Jewish education becomes a trademark of the participating congregations and a reason why people choose to them
Young people will grow up with a love for and understanding of their Jewish heritage and want to live meaningful Jewish lives

26. TO EVALUATE Evidence Methodology/Instrumentation What is acceptable?
What is reasonable?
Methodology/Instrumentation
What makes sense? (Time, budget, context, etc.) Will it get the information that is needed?
What instruments are best used?

27. FROM LOGIC MODEL TO EVALUATION
EVIDENCE
OF HOW EACH COMPONENT
AND THE INTERACTION AMONG THE COMPONENTS AFFECT THE PROFESSIONAL PRACTICE OF FELLOWS
METHODS
That will get the right info;
Qualitative /
Quantitative
Surveys, inter-
views, participant observation, focus groups

28. Symposia Summer Seminars Israel Seminars
How are they working? Helping to move towards outcomes, goals
Materials, observation, discussion, feedback survey, interviews,survey
Summer Seminars
Israel Seminars

29. Mentoring
360
Contact with thinkers /
leaders

30. Special Tools:
IMAPs, IAPs
Intentional Community / Peer Network
Intentional
Community /
Relationships w
Staff & Faculty

31. ISSUES IN APPLYING THIS FRAMEWORK
Q: What kind of evidence is acceptable?
A1:This is not a PhD dissertation, and so the answer is that it needs to be persuasive not perfect.
A2:”The best you can, given time and resource constraints”

32. ISSUES IN APPLYING THIS FRAMEWORK
Q: What kind of evidence is acceptable?
A3: Much of it is documentation, were the right people served, did assumptions pan out, was curriculum delivered as planned.
A4: Off-the-shelf or do-it-yourself assessment tools

33. SO HOW DOES IT REALLY WORK??
QUESTIONS from group
REFLECTIONS from Evie and group about LICSE experience: how information is used
NEXT STEPS