Evaluating Learning in the CALM Network A narrative of Appreciative Evaluation Jo Tait

Aim o To evaluate the impact of coaching, action learning and mentoring activities on participants learning - and their organisations.

Anticipated Outcomes o Framework for further appreciative research and evaluation studies. o Identified characteristics of good practice in action learning in HE. o Development for Associates and the CALM network. o A cohort of participants who have evaluated their own learning and can apply AL principles in practice.

Initial focus and challenges o Associates are all independent consultants, and staff developers working in higher education o Challenges - engagement, motivation and distance o 6 (currently only 3) Associates working as coaches, action learning set facilitators and mentors with participants o Challenges - timing and context o Associates seem best placed to work with participants to evaluate their learning, because of their closeness to the process and because of confidentiality issues o Challenges - distance and confidentiality

The Evaluation Community? Associates Participants Researchers

First Evaluation Community tasks o Define evaluation criteria o In uncertain and unformed network o Identify activities which will inform the evaluation  Learning agreements  Structured discussions and/or interviews within action learning process  Personal reflective diaries  Case studies  Learning history But all dependent on engagement and commitment of Associates

Formative plans for evaluation o Network events - never fully attended o Facilitating learning through dialogue and reflection with Associates - requires connection and commitment o Enhance Associates’ appreciation of own and participants’ learning - needs activity, dialogue and interaction o Illuminate organisational situations - anticipates trust and confidentiality

Plans for summative evaluation o Researchers synthesise data gathered from evaluation community o Evaluation team makes an overall assessment of impact of the learning of Associates and participants in relation to learning agreements o Evaluation community explores unanticipated learning outcomes o Review whole system development and evaluation events o Identify ways in which organisational impacts could be explored

Appreciative approaches to evaluation aim to o inquire into, identify and develop the best of what is, in order to create a better organisation and / or better practice o look at organisational issues challenges and concerns o focus on what seems to be working well, rather than on identifying or fixing problems o reflect with participants on their perceptions and shared understandings o be both an intervention and a lever for change o allow underpinning by a poetic principle – we choose where to rest our attention.

Appreciative inquiry and the unconditional positive - my preferences o Learning from experience o Person-centred principles (Rogers) in adult learning o Commitment to practical outcomes - usefulness o Importance of context for learning - situated (ecological) approach to professional development

Seven principles for participative and appreciative evaluation 1. Context and maturation 2. Leadership 3. Values & expectations 4. Evaluation capacity 5. Engagement and action 6. Exploring & extending outwards 7. Looking for where learning is disabled

1. Context and maturation o Take account of situated learning o Recognise principles of organisational development and maturation o Map stakeholders, values and relationships o Allow time for process and action o Keep reviewing and checking

2. Leadership o Direction and sense-making, when dealing with ‘unknowable and uncertain’ (Weick) appreciative territory o Cultivate dialogue and encourage negotiation o Tend to the emergent o Keep trying out the ‘story’ of the evaluation

3. Values and expectations o Evaluation values need to be explicit and shared from the beginning o Allow time and space for co- evaluators to join in expressing and negotiating those values o Appreciate that values may be espoused, practised and embodied differently o Attend to participative values throughout

4. Evaluation capacity o Develops through participation o Involves all in self-reflection and critical questioning o Take account of current practices and understandings o Appreciate the challenges of this way of working when tacit expectations may anticipate older, more powerful models.

5. Engagement and action o Relationships are core to appreciative approaches o Constantly attend to the invisible, subtle and undervalued skills of community-building o Recognise and appreciate practical limitations - work with what is.

6. Exploring outwards o Look beyond the current project or activity o Be open to other views and different ways of working o Articulate what you think you already know so that it can be challenged

7. Look for where learning is ‘disabled’ o Anticipate and appreciate ‘disabling’ resistance to change o Attend to the affective environment o Notice individual and collective discomfort as part of learning o Make it safe to risk voicing tacit assumptions

Dialogue - the hub of appreciative evaluation o Conditions and modes may vary but face to face is our preference o Dialogue is a spark and an adhesive o Helps us to gain insight and understand each other o Uncovers tacit assumptions o Allows us to question our world views o Enables shared meaning to emerge from divergent ideas

Questions and directions? o How to create a supportive framework from our principles - beyond sparks and glue! o Strengthening theoretical models for appreciative approaches o Drafting the evaluation report for different stakeholders o An academic paper that is also useful for evaluators

References Coghlan, A. T., Preskill, H., & T., T. C. (2003). An overview of appreciative inquiry in evaluation. In H. Preskill H. and A. T. Coghlan (Eds.), Using appreciative inquiry in evaluation. New directions for evaluation. San Francisco: Jossey Bass. Ludema, J. D., Cooperrider, D. & Barrett, F. J. (2001). Appreciative inquiry: The power of the unconditional positive question. In P. Reason & H. Bradbury (Eds.), Handbook of action research: Participative inquiry and practice (pp ) London: Sage.

Links o Appreciative Inquiry Commons intro/whatisai.cfm intro/whatisai.cfm o New Paradigm paradigm.co.uk/Appreciative.htm