1. Evaluating and assessing the impact of working with young people in regional and minority language revitalisation Rhys Jones, Elin Jones (Prifysgol Aberystwyth) and Fiona O’Hanlon (Edinburgh University)

2. Strwythur/Structure
Cyd-destun allanol: gwaith ieuenctid mewn iaith leiafrifol
External context: youth work in a minority language
Cyd-destun mewnol: heriau a chyfleon
Internal context: challenges and opportunities
Theori newid
Theory of change
Cymorth a chamau nesaf
Support and next steps

3. Cyd-destun allanol/External context
Polisi ar sail tystiolaeth (Test, Learn, Adapt)
Evidence-based policy (Test, Learn, Adapt)
Mae ymyrraethau’n methu yn aml
Interventions often fail
The need to justify public spending
Mae angen cyfiawnhau gwariant cyhoeddus

4. Cyd-destun allanol/External context
“The strategy recognises... the need to strengthen the evidence base on the impact of youth work across Wales.”
“much more needs to be done to systematically and robustly evidence the contribution of youth work” (both from the National Youth Work Strategy for Wales ).
Youth Work Reference Group (2015): “to develop and implement a National Outcomes Framework”.

5. Cyd-destun allanol/External context
“Self-evaluation processes use the triangulation of evidence-based information and data, people’s views and direct observation of practice to inform outcomes and measure impact on young people. In order to plan improvement effectively we need to know how well we are doing” (Scotland’s National Youth Work Strategy ), p. 17).
Measuring impact as one of 5 principles of youth work

6. Cyd-destun allanol/External context
Develop efficient and user-friendly ways of collecting and analysing quantitative and qualitative data at local, regional and national level to inform service planning, delivery, assessment and continuous improvement (Ireland’s National Youth Strategy , p. 35)
“Undertake and publish research that demonstrates the positive role of youth work in achieving national outcomes and local priorities” (National Youth Council of Ireland’s Strategic Plan, p. 23).

7. Cyd-destun allanol/External context
“Continue to build on our evidence about the Welsh language and its speakers, as a basis for assessing the effectiveness of our interventions and developing Welsh language policy.
Considering, as programmes are being planned, the most appropriate methods of evaluating their effect.
By combining qualitative and quantitative data we aim to enhance our understanding of speakers’ linguistic experiences and practices as they move through different periods of their lives” (Cymraeg 2050, p. 77).

8. Cyd-destun allanol/External context
“In order to track progress on the Plan, Bòrd na Gàidhlig will collate and publish information from a range of sources, including public authorities, Gaelic organisations and community groups in an annual National Gaelic Language Plan progress report.
Increasing the use of Gaelic is at the core of the National Gaelic Language Plan and ways of measuring this will be developed to give a clear picture as to how the strategies for increasing use are working” (National Gaelic Language Plan , p. 55).

9. Cyd-destun allanol/External context
Local Action Plans “supplemented by a language audit process which will identify ongoing strengths and weaknesses and provide time-series evidence of the impact which language-related policies and reforms would have on actual language use” (20 year Strategy for the Irish Language, p.24).
Overall…
An emphasis on the need to demonstrate outcomes and impact
Coming from funders and practitioners
Couched in terms of effectiveness rather than value for money
New methods, synthesis, a focus on time

10. Cyd-destun allanol/External context
Beth yw’r cyd-destun allanol ar gyfer mesur canlyniadau a thraweffaith eich gwaith?
What is the external context for evaluating the outcomes and impact of your work?
Beth yw’r cyd-destun pwysicaf (gwaith ieuenctid neu waith iaith)?
What is the most important context (youth work or language work)?
Sut mae’r angen i arddangos canlyniadau yn dylanwadu ar y berthynas gyda noddwyr?
How does the need to show outcomes influence relationships with sponsors?

11. Cyd-destun mewnol/Internal context
“Urdd Gobaith Cymru is a National Voluntary Youth Organisation with over 53,000 members between the ages of 8 – 25 yrs old. It provides Sport, Cultural, Community and Residential opportunities through the medium of Welsh for children and young people in Wales that will enable them to make a positive contribution to the community. The Urdd has nurtured generations of young men and women to be proud of their country, open to the world and living embodiments of our language and culture, along with the universal values which we cherish in Wales. Its contribution to…the lives and confidence, and mental health of our young people over the years has been immense” (Urdd Gobaith Cymru Annual Report )

12. Cyd-destun mewnol/Internal context
“Gael Linn is a national organisation whose principal aim is to foster and promote the Irish language and its heritage as a living language and as an expression of identity at policy and at community level.”
Feachtas is a young Irish-language organisation that was established in Objectives of the organisations:
a) To promote the Irish language among young people
b) To help the social and personal development of a young person
c) Help to build an Irish-language community
Cumann na bhFiann provides opportunities for young people to meet through the medium of Irish and create networks of Irish speaking friends throughout the country

13. Cyd-destun mewnol/Internal context
Comann na Gàidhlig organises a range of sporting and cultural activities for Gaelic-speaking young people. It aims to:
create opportunities for them to use Gaelic out with the classroom so that young people can use Gaelic in informal settings along with their peers.
increase the young people’s confidence in using Gaelic and their fluency.  
Research shows young people, parents and teachers feel local community initiatives are successful in terms of confidence and language skills (Young 2018)
Fèisean nan Gàidheal tutors young people in traditional music, Gaelic song & drama. It aims to ‘bring our culture to our young people.’
-Around 6,000 young people take part in such tuition each year.  
-Research shows participation to be associated with positive attitudes to Gaelic
and motivation to use the language (Broad & France 2005).
Comann na Gàidhlig has a dual approach of (i) supporting young people within 9 specific communities and (ii) organising sporting and youth activities for Gaelic-speaking young people across Scotland.
Young (2018) Iomairtean Gàidhlig: Buaidh is Cothroman. Edinburgh: Mereil Young Consulting
Broad, S. & France, J. (2005) 25 Years of the Feisean. The participants’ story: attitudinal research on the Feis movement in Scotland. Glasgow: RSAMD

14. Cyd-destun mewnol/Internal context
Hyrwyddo iaith leiafrifol (gallu ac yn enwedig defnydd)
Promoting a minority language (ability and especially use)
Cysylltiad gyda hunaniaeth a diwylliant
A connection with identity and culture
Creu cymunedau ieithyddol
Creating language communities
Ond hefyd…
But also…
Creu dinasyddion ‘gwell’
Creating ‘better’ citizens
Gwella perfformiad academaidd
Improving levels of academic achievement
Lleihau anhafaliadau economaidd/NEETs
Addressing socio-economic inequalities/NEETs

15. Cyd-destun mewnol/Internal context
Beth yw prif amcanion eich mudiad?
What are your organisation’s main aims?
I ba raddau rydych chi’n asesu canlyniadau a thraweffaith eich gwaith?
To what extent do you evaluate the outcomes and impact of your work?
Sut ydych yn gwneud hyn?
How do you do this?
Oes arfer dda?
Is there good practice?
Beth yw’r heriau neu rwystrau?
What are the challenges or obstacles?
Oes cyfleon?
Are there opportunities?

16. Cyd-destun mewnol/Internal context
Capasiti (amser, sgiliau)
Capacity (time, skills)
‘Pellter a deithiwyd’
‘Distance travelled’
Treigl amser
The passage of time
Cymdeithas fel system cymhleth
Society as a complex system
The demands of different funding agencies
Gofynion asiantaethau ariannu gwahanol
Specific challenges in relation to language:
Heriau penodol ym maes iaith:
Defnydd iaith
Language use
Perthynas gyda ffactorau mwy haniaethol
Link to more abstract factors
Can quantitative data capture this?
Ai data meintiol sydd orau?

17. Cyd-destun mewnol/Internal context
Cyfleon
Opportunities
Rydych chi’n adnabod eich ‘cleientiaid’ yn dda iawn
You know your ‘clients’ very well
Mae gennych chi storfa fawr o wybodaeth amdanynt
You possess a large store of information about them
Mae cyfle i lunio agenda gyffredin?
There is an opportunity to shape a common agenda?

18. Cyd-destun mewnol/Internal context
Allbwn, canlyniad a thraweffaith
Output, outcome and impact
Allbwn: gweithgareddau, prosesau, cynnyrch
Output: actions, processes, products
Canlyniad: canlyniad y gweithareddau a phrosesau
Outcomes: the results of actions and processes
Impact: broader and longer-term effects
Traweffaith: effaith ehangach a mwy hir dymor

19. Cyd-destun mewnol/Internal context
“While the outcome evaluation tells us what kind of change has occurred, an impact evaluation paints a picture as to how a program might have affected participants’ lives on a broader scale” (Andrew Harding, LSE).
Path fath o dystiolaeth mae modd ei gasglu?/What kind of evidence is it possible to collect?
Pa fath o dystiolaeth a ddisgwylir i chi ei arddangos?/What kind of evidence are you expected to produce?

20. Cyd-destun mewnol/Internal context

22. Theori newid/Theory of change
Deiagram sy’n dangos dylanwad rhaglen neu ymyrraeth ar gleientiaid
A diagram that represents the effect of a programme or intervention on clients
Amlinellu yr holl weithgareddau a phrosesau (allbwn)
Outline all of the activities and processes (output)
How these processes etc lead to outcomes
Sut mae’r prosesau ayb yn arwain at ganlyniadau
The broader impact of the outcomes
Natur traweffaith ehangach y canlyniadau

23. Dull y NWA/The MSC method

24. Theori newid/Theory of change
“We believe that a clear, concise and convincing explanation of what you do, what impact you aim to have and how you believe you will have it is a vital foundation of any programme, and a prerequisite for effective evaluation” (NESTA, Guidance for Developing a Theory of Change for Your Programme).

25. Theori newid/Theory of change
Cymrwch 10 munud i lunio deiagram theori newid ar gyfer un o amcanion eich mudiad
Take 10 minutes to produce a theory of change diagram for one of your organization’s aims
A ydyw’n ddefnyddiol?
Is it useful?
A ydyw’n arwain at gwestiynau/ystyriaethau newydd?
Does it lead to new questions/considerations?

26. Camau nesaf/Next steps
Cymorth
Help
Cyrff yng Nghymru a’r Alban yn barod i helpu
Institutions in Scotland and Wales are ready to help
Prifysgol Aberystwyth
Prifysgol Caeredin
Aberystwyth University
WISERD
Edinburgh University
Pa gefnogaeth sydd ei angen?
What support is needed?

27. Camau nesaf/Next steps
Ysgrifennu dogfen ar sail y cyflwyniad, yr ymchwil sy’n sail iddo a’r mewnbwn gennych chi
Writing a document on the basis of the presentation, the research that lies behind it and your input
Parhau i gydweithio?
Continuing to collaborate?