Evaluating and assessing the impact of working with young people in regional and minority language revitalisation Rhys Jones, Elin Jones (Prifysgol Aberystwyth) and Fiona O’Hanlon (Edinburgh University)

Strwythur/Structure Cyd-destun allanol: gwaith ieuenctid mewn iaith leiafrifol External context: youth work in a minority language Cyd-destun mewnol: heriau a chyfleon Internal context: challenges and opportunities Theori newid Theory of change Cymorth a chamau nesaf Support and next steps

Cyd-destun allanol/External context Polisi ar sail tystiolaeth (Test, Learn, Adapt) Evidence-based policy (Test, Learn, Adapt) Mae ymyrraethau’n methu yn aml Interventions often fail The need to justify public spending Mae angen cyfiawnhau gwariant cyhoeddus

Cyd-destun allanol/External context “The strategy recognises... the need to strengthen the evidence base on the impact of youth work across Wales.” “much more needs to be done to systematically and robustly evidence the contribution of youth work” (both from the National Youth Work Strategy for Wales 2014-18). Youth Work Reference Group (2015): “to develop and implement a National Outcomes Framework”.

Cyd-destun allanol/External context “Self-evaluation processes use the triangulation of evidence-based information and data, people’s views and direct observation of practice to inform outcomes and measure impact on young people. In order to plan improvement effectively we need to know how well we are doing” (Scotland’s National Youth Work Strategy 2014-19), p. 17). Measuring impact as one of 5 principles of youth work

Cyd-destun allanol/External context Develop efficient and user-friendly ways of collecting and analysing quantitative and qualitative data at local, regional and national level to inform service planning, delivery, assessment and continuous improvement (Ireland’s National Youth Strategy 2015-20, p. 35) “Undertake and publish research that demonstrates the positive role of youth work in achieving national outcomes and local priorities” (National Youth Council of Ireland’s Strategic Plan, p. 23).

Cyd-destun allanol/External context “Continue to build on our evidence about the Welsh language and its speakers, as a basis for assessing the effectiveness of our interventions and developing Welsh language policy. Considering, as programmes are being planned, the most appropriate methods of evaluating their effect. By combining qualitative and quantitative data we aim to enhance our understanding of speakers’ linguistic experiences and practices as they move through different periods of their lives” (Cymraeg 2050, p. 77).

Cyd-destun allanol/External context “In order to track progress on the Plan, Bòrd na Gàidhlig will collate and publish information from a range of sources, including public authorities, Gaelic organisations and community groups in an annual National Gaelic Language Plan progress report. Increasing the use of Gaelic is at the core of the National Gaelic Language Plan and ways of measuring this will be developed to give a clear picture as to how the strategies for increasing use are working” (National Gaelic Language Plan 2018-23, p. 55).

Cyd-destun allanol/External context Local Action Plans “supplemented by a language audit process which will identify ongoing strengths and weaknesses and provide time-series evidence of the impact which language-related policies and reforms would have on actual language use” (20 year Strategy for the Irish Language, p.24). Overall… An emphasis on the need to demonstrate outcomes and impact Coming from funders and practitioners Couched in terms of effectiveness rather than value for money New methods, synthesis, a focus on time

Cyd-destun allanol/External context Beth yw’r cyd-destun allanol ar gyfer mesur canlyniadau a thraweffaith eich gwaith? What is the external context for evaluating the outcomes and impact of your work? Beth yw’r cyd-destun pwysicaf (gwaith ieuenctid neu waith iaith)? What is the most important context (youth work or language work)? Sut mae’r angen i arddangos canlyniadau yn dylanwadu ar y berthynas gyda noddwyr? How does the need to show outcomes influence relationships with sponsors?

Cyd-destun mewnol/Internal context “Urdd Gobaith Cymru is a National Voluntary Youth Organisation with over 53,000 members between the ages of 8 – 25 yrs old. It provides Sport, Cultural, Community and Residential opportunities through the medium of Welsh for children and young people in Wales that will enable them to make a positive contribution to the community. The Urdd has nurtured generations of young men and women to be proud of their country, open to the world and living embodiments of our language and culture, along with the universal values which we cherish in Wales. Its contribution to…the lives and confidence, and mental health of our young people over the years has been immense” (Urdd Gobaith Cymru Annual Report 2017-18)

Cyd-destun mewnol/Internal context “Gael Linn is a national organisation whose principal aim is to foster and promote the Irish language and its heritage as a living language and as an expression of identity at policy and at community level.” Feachtas is a young Irish-language organisation that was established in 1980. Objectives of the organisations: a) To promote the Irish language among young people b) To help the social and personal development of a young person c) Help to build an Irish-language community Cumann na bhFiann provides opportunities for young people to meet through the medium of Irish and create networks of Irish speaking friends throughout the country

Cyd-destun mewnol/Internal context Comann na Gàidhlig organises a range of sporting and cultural activities for Gaelic-speaking young people. It aims to: create opportunities for them to use Gaelic out with the classroom so that young people can use Gaelic in informal settings along with their peers. increase the young people’s confidence in using Gaelic and their fluency.   Research shows young people, parents and teachers feel local community initiatives are successful in terms of confidence and language skills (Young 2018) Fèisean nan Gàidheal tutors young people in traditional music, Gaelic song & drama. It aims to ‘bring our culture to our young people.’ -Around 6,000 young people take part in such tuition each year.   -Research shows participation to be associated with positive attitudes to Gaelic and motivation to use the language (Broad & France 2005). Comann na Gàidhlig has a dual approach of (i) supporting young people within 9 specific communities and (ii) organising sporting and youth activities for Gaelic-speaking young people across Scotland. Young (2018) Iomairtean Gàidhlig: Buaidh is Cothroman. Edinburgh: Mereil Young Consulting Broad, S. & France, J. (2005) 25 Years of the Feisean. The participants’ story: attitudinal research on the Feis movement in Scotland. Glasgow: RSAMD

Cyd-destun mewnol/Internal context Hyrwyddo iaith leiafrifol (gallu ac yn enwedig defnydd) Promoting a minority language (ability and especially use) Cysylltiad gyda hunaniaeth a diwylliant A connection with identity and culture Creu cymunedau ieithyddol Creating language communities Ond hefyd… But also… Creu dinasyddion ‘gwell’ Creating ‘better’ citizens Gwella perfformiad academaidd Improving levels of academic achievement Lleihau anhafaliadau economaidd/NEETs Addressing socio-economic inequalities/NEETs

Cyd-destun mewnol/Internal context Beth yw prif amcanion eich mudiad? What are your organisation’s main aims? I ba raddau rydych chi’n asesu canlyniadau a thraweffaith eich gwaith? To what extent do you evaluate the outcomes and impact of your work? Sut ydych yn gwneud hyn? How do you do this? Oes arfer dda? Is there good practice? Beth yw’r heriau neu rwystrau? What are the challenges or obstacles? Oes cyfleon? Are there opportunities?

Cyd-destun mewnol/Internal context Capasiti (amser, sgiliau) Capacity (time, skills) ‘Pellter a deithiwyd’ ‘Distance travelled’ Treigl amser The passage of time Cymdeithas fel system cymhleth Society as a complex system The demands of different funding agencies Gofynion asiantaethau ariannu gwahanol Specific challenges in relation to language: Heriau penodol ym maes iaith: Defnydd iaith Language use Perthynas gyda ffactorau mwy haniaethol Link to more abstract factors Can quantitative data capture this? Ai data meintiol sydd orau?

Cyd-destun mewnol/Internal context Cyfleon Opportunities Rydych chi’n adnabod eich ‘cleientiaid’ yn dda iawn You know your ‘clients’ very well Mae gennych chi storfa fawr o wybodaeth amdanynt You possess a large store of information about them Mae cyfle i lunio agenda gyffredin? There is an opportunity to shape a common agenda?

Cyd-destun mewnol/Internal context Allbwn, canlyniad a thraweffaith Output, outcome and impact Allbwn: gweithgareddau, prosesau, cynnyrch Output: actions, processes, products Canlyniad: canlyniad y gweithareddau a phrosesau Outcomes: the results of actions and processes Impact: broader and longer-term effects Traweffaith: effaith ehangach a mwy hir dymor

Cyd-destun mewnol/Internal context “While the outcome evaluation tells us what kind of change has occurred, an impact evaluation paints a picture as to how a program might have affected participants’ lives on a broader scale” (Andrew Harding, LSE). Path fath o dystiolaeth mae modd ei gasglu?/What kind of evidence is it possible to collect? Pa fath o dystiolaeth a ddisgwylir i chi ei arddangos?/What kind of evidence are you expected to produce?

Cyd-destun mewnol/Internal context

Theori newid/Theory of change Deiagram sy’n dangos dylanwad rhaglen neu ymyrraeth ar gleientiaid A diagram that represents the effect of a programme or intervention on clients Amlinellu yr holl weithgareddau a phrosesau (allbwn) Outline all of the activities and processes (output) How these processes etc lead to outcomes Sut mae’r prosesau ayb yn arwain at ganlyniadau The broader impact of the outcomes Natur traweffaith ehangach y canlyniadau

Dull y NWA/The MSC method

Theori newid/Theory of change “We believe that a clear, concise and convincing explanation of what you do, what impact you aim to have and how you believe you will have it is a vital foundation of any programme, and a prerequisite for effective evaluation” (NESTA, Guidance for Developing a Theory of Change for Your Programme).

Theori newid/Theory of change Cymrwch 10 munud i lunio deiagram theori newid ar gyfer un o amcanion eich mudiad Take 10 minutes to produce a theory of change diagram for one of your organization’s aims A ydyw’n ddefnyddiol? Is it useful? A ydyw’n arwain at gwestiynau/ystyriaethau newydd? Does it lead to new questions/considerations?

Camau nesaf/Next steps Cymorth Help Cyrff yng Nghymru a’r Alban yn barod i helpu Institutions in Scotland and Wales are ready to help Prifysgol Aberystwyth Prifysgol Caeredin Aberystwyth University WISERD Edinburgh University Pa gefnogaeth sydd ei angen? What support is needed?

Camau nesaf/Next steps Ysgrifennu dogfen ar sail y cyflwyniad, yr ymchwil sy’n sail iddo a’r mewnbwn gennych chi Writing a document on the basis of the presentation, the research that lies behind it and your input Parhau i gydweithio? Continuing to collaborate?