TEC 2013 Hyderabad The role of programme evaluation in curriculum development OR ‘How are we doing?’ Richard Kiely University of St Mark & St John, UK

My life in programme evaluation 1.Medium of instruction evaluation (Pupil) 2.Native-speaker teacher evaluation (Teacher) 3.Evaluation for development consultancy (Consultant) 4.International CLIL project (Evaluator) 5.Teacher education programme impact study (Researcher)

Some principles 1.Evaluation cannot just test the theory 2.Evaluation has to understand learning processes as well as outcomes 3.Evaluation has to engage with all stakeholders 4.Evaluation has to facilitate action 5.Evaluation should make sense of the programme for everyone.

This presentation Overview of language programme evaluation and its potential in curriculum and professional development. Evaluation purposes, designs and methods Identities and roles of programme stakeholders Using the processes and outcomes of evaluation for improvement of programmes, and Management of change issues.

Overview Programme evaluation is ‘a set of strategies to document and understand the programme. It involves research activity (conventional studies or action research by which teachers learn about and transform aspects of their practice) and assessment data (conventional measures of outcomes). In addition to these, evaluation has to engage with the social, cultural and historical identity of the programme, as a product of the institution, as a phase in the biographies of participants, and as a context of personal investments of individual stakeholders’. Kiely 2009: 114

Programme Evaluation TYPE 1TYPE 2 Experimental/ comparative Case study ProbabilityPossibility Policy-orientedPractice-oriented Programme theory (explicit) Programme theory (implicit) Product-focusProcess-focus ExternalInternal

Programme Evaluation TYPE 1TYPE 2 Experimental/ comparative Case study ProbabilityPossibility Policy-orientedPractice-oriented Programme theory (explicit) Programme theory (implicit) Product-focusProcess-focus ExternalInternal Discuss: Any of these features new to you?

Evaluation purposes, designs and methods Purposes: accountability, development and quality assurance Designs: the links between data, theory and action Methods: ways of getting data.

Evaluation designs – Type 1 Template 1Template 2 Measurement of outcomes (language tests; teacher performance; teacher qualifications; Surveys of attitudes; preferences; aspirations; wants; needs

Evaluation designs – Type 2 Template 1Template 2Template 3 Quality Assurance Programme support Teacher research

Evaluation designs – Type 2 Template 1Template 2Template 3 Quality Assurance Programme support Teacher research Buzz group discussion: What kind of activities do you think would contribute to these designs?

Evaluation designs – Type 2 Template 1Template 2Template 3 Quality Assurance Programme support Teacher research End of course questionnaires MentoringAction research SurveysPeer observationReflective Practice Focus GroupsReading circlesExploratory practice AuditsAssessment workshops Study for research degrees Assessment dataMeetings

Evaluation designs Template 1Template 2Template 3 Quality Assurance Programme support Teacher research Design issues How do we find the right amount of activity? How do we get a focus on learning? How do we add value to learning opportunities?

Identities and roles of programme stakeholders Teachers

Identities and roles of programme stakeholders Students Teachers

Identities and roles of programme stakeholders Students Teachers Teacher educators

Identities and roles of programme stakeholders Students Teachers Teacher educators Leaders and managers

Identities and roles of programme stakeholders Students Teachers Teacher educators Leaders and managers Remote stakeholders

Identities and roles ImmediatestakeholdersRemote stakeholders StudentsTeachersManagers CustomersTransformersSponsors LearnersTransmittersParents ParticipantsAdvisorsEmployers PractitionersAssessors

Student role - learning Using the processes and outcomes of evaluation for improvement of programmes WORKSHOP

Student role - learning Using the processes and outcomes of evaluation for improvement of programmes WORKSHOP Buzz group discussion: How can teachers workshop evaluation process with students?

Student role - learning Processes Awareness raising Focus groups Structured discussions Reflections as part of assessment

Student role - learning Engaging learners in communicative encounters, especially if their aim is to explore emotional content and experiences, can become too bound up in itself unless this activity also reaches an evaluation stage. Trying to understand what has happened while undertaking a particular task, why it was suggested by the teacher, and contributing actively to the evaluation of learning arrangements, sequences, resources and input materials by means of reflection and meta-communicative discourse – all these are considered indispensable learner activities in ELT. Legutke and Thomas (1991:65)

Student role - learning Engaging learners in communicative encounters, especially if their aim is to explore emotional content and experiences, can become too bound up in itself unless this activity also reaches an evaluation stage. Trying to understand what has happened while undertaking a particular task, why it was suggested by the teacher, and contributing actively to the evaluation of learning arrangements, sequences, resources and input materials by means of reflection and meta-communicative discourse – all these are considered indispensable learner activities in ELT. Legutke and Thomas (1991:65) Autonomy

Student role - learning Engaging learners in communicative encounters, especially if their aim is to explore emotional content and experiences, can become too bound up in itself unless this activity also reaches an evaluation stage. Trying to understand what has happened while undertaking a particular task, why it was suggested by the teacher, and contributing actively to the evaluation of learning arrangements, sequences, resources and input materials by means of reflection and meta-communicative discourse – all these are considered indispensable learner activities in ELT. Legutke and Thomas (1991:65) Autonomy Agency

Student role - learning Engaging learners in communicative encounters, especially if their aim is to explore emotional content and experiences, can become too bound up in itself unless this activity also reaches an evaluation stage. Trying to understand what has happened while undertaking a particular task, why it was suggested by the teacher, and contributing actively to the evaluation of learning arrangements, sequences, resources and input materials by means of reflection and meta-communicative discourse – all these are considered indispensable learner activities in ELT. Legutke and Thomas (1991:65) Autonomy Agency Motivation

Using the processes and outcomes of evaluation for improvement of programmes Teachers in a community of practice

Using the processes and outcomes of evaluation for improvement of programmes Collaborative development (peer observation; mentoring; coaching) Workshops on materials; ICT; assessment formats; test data

Using the processes and outcomes of evaluation for improvement of programmes Collaborative development (peer observation; mentoring; coaching) Workshops on materials; ICT; assessment formats; test data Buzz group discussion: Do these happen in your context? If not, why not?

Teacher role – professional learning Sponsored professionalism: based on qualification and recognition Independent professionalism: ‘a commitment to careful and critical examination of the assumptions and practices embedded in sponsored professionalism’ (Leung 2009:53).

Teachers and Change Int: So after the evaluation […..] Will you do it differently next time? Millie: I don’t think I’ll do it differently. […] we have developed some good strategies and some quite good materials, that will continue to develop. So it is not a change in direction, but perhaps going further in the same direction. Kiely 1998: 194

Teachers and Change (Kiely 2001:257) Feedback Resistance Reflection Innovation CYCLE 1 Anna gets feedback from students suggesting less attention to explaining words in class    CYCLE 2    Teachers and Change (Kiely 2001:257)

Feedback Resistance Reflection Innovation CYCLE 1 Anna gets feedback from students suggesting less attention to explaining words in class  Anna interprets this as selfish thinking on the part of students   CYCLE 2    Teachers and Change (Kiely 2001:257)

Feedback Resistance Reflection Innovation CYCLE 1 Anna gets feedback from students suggesting less attention to explaining words in class  Anna interprets this as selfish thinking on the part of students  Anna rationalises that this might not be the best way of using classroom time  CYCLE 2    Teachers and Change (Kiely 2001:257)

Feedback Resistance Reflection Innovation CYCLE 1 Anna gets feedback from students suggesting less attention to explaining words in class  Anna interprets this as selfish thinking on the part of students  Anna rationalises that this might not be the best way of using classroom time  Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words CYCLE 2    Teachers and Change (Kiely 2001:257)

Feedback Resistance Reflection Innovation CYCLE 1 Anna gets feedback from students suggesting less attention to explaining words in class  Anna interprets this as selfish thinking on the part of students  Anna rationalises that this might not be the best way of using classroom time  Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words CYCLE 2 Anna gets feedback from students suggesting more attention to explaining words in class    Teachers and Change (Kiely 2001:257)

Feedback Resistance Reflection Innovation CYCLE 1 Anna gets feedback from students suggesting less attention to explaining words in class  Anna interprets this as selfish thinking on the part of students  Anna rationalises that this might not be the best way of using classroom time  Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words CYCLE 2 Anna gets feedback from students suggesting more attention to explaining words in class  Anna resists suggestions that her focus on ‘broad swathes of meaning’ should change   Teachers and Change (Kiely 2001:257)

Feedback Resistance Reflection Innovation CYCLE 1 Anna gets feedback from students suggesting less attention to explaining words in class  Anna interprets this as selfish thinking on the part of students  Anna rationalises that this might not be the best way of using classroom time  Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words CYCLE 2 Anna gets feedback from students suggesting more attention to explaining words in class  Anna resists suggestions that her focus on ‘broad swathes of meaning’ should change  Anna: ‘This group say they want it, so I try to do it for them’, and includes a short activity in Week 6  Teachers and Change (Kiely 2001:257)

Feedback Resistance Reflection Innovation CYCLE 1 Anna gets feedback from students suggesting less attention to explaining words in class  Anna interprets this as selfish thinking on the part of students  Anna rationalises that this might not be the best way of using classroom time  Anna develops a pedagogy which focusses on comprehension of ideas rather than individual words CYCLE 2 Anna gets feedback from students suggesting more attention to explaining words in class  Anna resists suggestions that her focus on ‘broad swathes of meaning’ should change  Anna: ‘This group say they want it, so I try to do it for them’, and includes a short activity in Week 6  Anna spends more time on vocabulary in Weeks 9 & 10, and provides vocabulary tasks to texts in Weeks 11 & 12 Teachers and Change (Kiely 2001:257)

Teachers and Change Change is gradual, perhaps not always visible to the naked eye; Change occurs at the teacher’s pace; Change is negotiated; Change happens.

Key guidelines from principles 1.Evaluation cannot just test the theory 2.Evaluation has to understand learning processes as well as outcomes 3.Evaluation has to engage with all stakeholders 4.Evaluation has to facilitate action 5.Evaluation should make sense of the programme for everyone.

Use evaluation to …. 1.…. explain what is going on 2.… answer the ‘How are we doing?’ question 3.… allow all voices to be heard 4.… act for improvement 5.… raise awareness and explain.

References Kiely, R. (2012) Designing evaluation into change management processes. Overview chapter in Tribble, C. (Ed) Managing Change in Language Education. London: The British Council pp Kiely, R. & P. Rea-Dickins (2009) Evaluation and learning in language programmes. In Knapp, K. and B. Seidlhofer with H. Widdowson (eds) Handbooks of Applied Linguistics: Volume 6: Handbook of foreign language communication and learning. Amsterdam: Mouton de Gruyter. Pp Kiely, R. (2011) Understanding CLIL as an innovation. Studies in Second Language Learning and Teaching. Vol 1, No 1, pp Kiely, R. (2009) Small answers to the big question: Learning from language programme evaluation. Language Teaching Research Vol 13, No 1: Kiely, R. (2006) Evaluation, innovation and ownership in language programs. Modern Language Journal, Vol 90, No 3 pp: Kiely, R. & P. Rea-Dickins (2005) Program Evaluation in Language Education. Basingstoke: Palgrave Macmillan (Series editors Chris Candlin and David Hall) [Second edition – 2013 – in preparation]. Legutke, M. & H. Thomas (1991) Process and Experience in the Language Classroom. Harlow: Longman

Thank you