P OSITIONING EVALUATION AS RESPONSIVE TO INFLUENCES AND AS BEING INFLUENTIAL Cheryl Ballantyne NSW Department of Education and Communities Western Sydney Region Australasian Evaluation Society International Conference 29 August – 2 September 2011 Hilton, Sydney, Australia

A IMS OF PRESENTATION To explore the notion of evaluation as an intervention by: Describing a current evaluation in progress Identifying influences to date Outlining future directions for the evaluation

T HEORETICAL CONTEXT Evaluation as an intervention has an end goal of social betterment (Henry, 2005; Henry and Mark, 2003) Evaluation as an intervention has potential to influence at individual, interpersonal and collective levels (Henry and Mark, 2003) Integrated theory of evaluation influence incorporates three dimensions, source of influence, intention and time frame (Kirkhart, 2000)

K EY CLAIM Evaluation, when incorporated as a regular intervention within program implementation can increase the program’s effectiveness and its responsiveness to contextual influences.

S TRUCTURE 1. Description of program 2. What have been the contextual features that have influenced the program development and implementation? 1. What influences has the evaluation had to date?

S TRUCTURE 4. Why has this evaluation had these influences? 5. How was the evaluation data generated? 6. What are the implications for developing future evaluation instruments?

1. W ESTERN S YDNEY N INGBO C HINESE V OLUNTEER T EACHER - RESEARCHER P ROGRAM Partnership involving DEC Western Sydney Region, Ningbo Municipal Education Bureau, PRC, and University of Western Sydney

P ROGRAM AIM To promote the teaching and learning of Chinese language and culture in WSR schools by generating interest in schools and embedding that interest through a sustained program of support Long term – development of students’ intercultural competence in Chinese

P ROGRAM FEATURES participating schools, 10 primary and 8 high schools selected by expression of interest, majority of students are non-background speakers of Chinese Up to 10 graduates from Ningbo, China arrive at the end of June each year Volunteer in schools two days per week to support the teaching and learning of Chinese language, culture and society Additional human resource to participating schools equal to approximately $25,0000 per year

P ROGRAM FEATURES Volunteers complete Master of Education (Honours) at University of Western Sydney Research informs and is informed by school experience Return to China in December the following year

2. C ONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION Globalisation Expansion of China in the world Approx 1.2 billion English speakers in China Approx 700 billion Chinese speakers in other countries Implications for students’ future communities and employment

C ONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION Commonwealth Government commitment to increasing Asia literacy of Australian students 2020 target National Curriculum Languages learning area Mandarin Chinese to be one of two target languages to be taught in NSW Across curriculum dimension – Asia and Australia’s engagement with Asia

C ONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION NSW DEC memorandum of understanding with China NSW Confucius Institute in partnership with China Confucius classroom in Western Sydney Region

C ONTEXTUAL FEATURES THAT HAVE INFLUENCED PROGRAM DEVELOPMENT AND EVALUATION Western Sydney Region memorandum of understanding with Ningbo Municipal Education Bureau annual cultural exchange visits sister schools

E VALUATION RESPONSE 2010 – 2012 evaluation of Volunteer Teacher-researcher Program to inform ongoing program development and future directions Focus question: What can a Chinese native-speaker volunteer program contribute to the teaching and learning of Chinese language and culture in schools?

A NALYSIS OF E VALUATION INFLUENCES : Individual Interpersonal Collective (Henry and Mark, 2003)

3. W HAT INFLUENCES HAS THE EVALUATION HAD TO DATE ? Individual Increased the importance individual principals and regional staff attribute to the Program Renewed individual principals’ commitment to the Program Shaped Regional leaders’ attitudes towards importance of Chinese language learning and the partnership with Ningbo

W HAT INFLUENCES HAS THE EVALUATION HAD TO DATE ? Interpersonal Principals share ideas about their Chinese programs with one another Principals emphasise importance of Chinese programs to staff and community

W HAT INFLUENCES HAS THE EVALUATION HAD TO DATE ? Interpersonal Additional schools have expressed interest in participating in the Program Inquiries about/interest in the Program from interstate and national levels Meeting of partners to be held in September 2011 to discuss extension of partnership

W HAT INFLUENCES HAS THE EVALUATION HAD TO DATE ? Collective Signing of new Memorandum of Understanding planned for late September 2011 Increased allocation of funding to program Regional planning to target outreach programs delivered through Confucius Classroom to high school learners Planning of professional learning to focus on areas for further program development

E VALUATION INTERVENTION AND SOCIAL BETTERMENT In 21 st century Australia, Asia literacy and development of intercultural competence in Chinese has potential to lead to social betterment (increase students’ world view, improve employment prospects). Evaluation has identified student, parent and teacher interest in Asia literacy through Chinese Evaluation has revealed that students, parents and teachers regard Asia literacy through Chinese as important Evidence has influenced further evaluation strategies, to enable a deeper analysis of influence

4. W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? ( EVIDENCE )

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? ( EVIDENCE )

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? P RIMARY SCHOOL SURVEYS 702 student responses from eight schools 67% - learning Chinese language important 80% - learning Chinese language interesting 63% - learning about China important 78% - learning about China interesting 65% would like to continue learning Chinese in 2011

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? P RIMARY SCHOOL PARENT SURVEYS 117 parent responses from seven schools More than 80% - learning Chinese language and culture important and valuable for students

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? H IGH SCHOOL SURVEYS 245 student responses from five schools 178 student responses from one school only 10 parent responses all from one school

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? ( EVIDENCE )

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? S TAFF SURVEYS 50 staff responses representing 12 schools 38 from 6 primary schools 12 from 6 high schools 95% - learning Chinese language important 100% - learning Chinese culture important 93% - Chinese Program has a positive influence on schools

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? S TAFF SURVEYS

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? P RINCIPAL FOCUS GROUPS Volunteers’ rapport with students has stimulated interest in Chinese language Chinese language is accepted as part of the curriculum ( no longer exotic – High School Principal) Planning for Chinese language and culture programs occurring in primary schools and high schools

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? P RINCIPAL FOCUS GROUPS Chinese language has replaced other language programs - two high schools Links between Chinese language and culture and other learning areas - three high schools Chinese classroom allocated - one high school

W HY HAS THIS EVALUATION HAD THESE INFLUENCES ? P RINCIPAL FOCUS GROUPS Collaboration across learning communities to support transition Year 6 to 7. System support important to the long term effectiveness of school programs

5. H OW WAS THE EVALUATION DATA GENERATED DEC annual census information on languages programs Focus group of Principals term Surveys of students, parents and staff distributed to 18 participating schools term

E VALUATION INFLUENCE : SOURCE, INTENTION, TIME FRAME Source – process (focus groups) and reporting of evidence (results) Intention – intended (focus groups, reporting of evidence) Time frame – immediate, evaluation embedded in program implementation (Kirkhart, 2000)

6. W HAT ARE THE IMPLICATIONS FOR FUTURE EVALUATION INSTRUMENTS ? F UTURE DIRECTIONS Student self-assessment of intercultural competence to enable deeper analysis of Program’s impact on student learning Parent and teacher assessment of student intercultural competence to enable deeper analysis of impact on student learning

6. W HAT ARE THE IMPLICATIONS FOR FUTURE EVALUATION INSTRUMENTS ? F UTURE DIRECTIONS School self-evaluation (based on program standards) to enable deeper analysis of school programs and influence school planning Document analysis of school plans, policies and annual school reports to analyse extent to which Chinese programs are valued, planned for and embedded in whole school operation

C ONCLUSION This evaluation (in early stages) has had influence at school, regional, state, interstate and national levels Further planning of evaluation instruments is intended to deepen influence This evaluation, as an intervention embedded in program planning and implementation, has potential to make a difference for Western Sydney students

T HANK YOU FROM C HERYL B ALLANTYNE FOR CONSIDERING - P OSITIONING EVALUATION AS RESPONSIVE TO INFLUENCES AND AS BEING INFLUENTIAL Australasian Evaluation Society International Conference 29 August – 2 September 2011 Hilton, Sydney, Australia