1. Self-Evaluation Toolkit
Rosslyn Monro
CLCQ Conference 2018

2. Emotional Baggage Evaluation – The world’s most [adjective] topic
Fill in the blank:
Evaluation – The world’s most [adjective] topic
Tell you partner your reasons for choosing your adjective
Self-Evaluation Toolkit | 2

3. This workshop CLCQ developed a Self-Evaluation Toolkit
Practical step through the toolkit
Toolkit is on-line –
Self-Evaluation Toolkit | 3

4. Theory of Change
Theory of Change sets out the link between CLC activities and the achievement of medium and long-term outcomes.
Self-Evaluation Toolkit | 4

5. Theory of Change Pairs:
Think of a client that stands out for you that no longer has contact with your service.
Based on what you know about the client, paint a picture for your partner of the client’s best life.
Talk about their:
health,
relationships
sense of safety and security
engagement in community life
Self-Evaluation Toolkit | 5

6. Theory of Change
The story of your client’s best life are the outcomes or impacts.
An evaluation seeks to trace our activities (inputs) with the client (stakeholder) to understand the link between our work and the client’s life (stakeholder outcomes)
The evaluation toolkit tracks outcomes for 4 stakeholder groups:
Clients
Community partners
The justice system
Volunteers
Each table is focussing on one stakeholder group each today
Self-Evaluation Toolkit | 6

7. Theory of Change
“Without our service the Police would have more work to do”
Outcome – savings to the justice system
“Our clients say they feel a weight has been lifted from their shoulders because they know their options”
Outcome – reduced stress, better health, reduced reliance on health care system
Self-Evaluation Toolkit | 7

8. Theory of Change
Toolkit developed a Theory of Change with intermediate and long term outcomes
Self-Evaluation Toolkit

9. Theory of Change – Intermediate Outcomes
The toolkit methodology predominately focusses on intermediate outcomes because they are:
More immediately identifiable
Measurable
Based on a program logic that if we achieve intermediate outcomes they will contribute to the longer-term outcomes which are harder to measure
As we develop our evaluation methodology we will work on developing ways to measure the long term outcomes
Self-Evaluation Toolkit | 9

10. Theory of Change – Intermediate Outcomes
Your turn: Workshop Activity 1 - Develop a summary of intermediate outcomes
Summarise the list of outcomes with no more than 2 phrases or sentence
eg. Clients get the services they need
eg. Volunteers have enhanced career prospects
Using the workbook – page 2 for stakeholder
Self-Evaluation Toolkit | 10

11. Theory of Change – Intermediate Outcomes for clients
Stake- holder
Domains
Intermediate Outcomes
Working with clients
Appropriateness of service delivery
Client has their matter properly heard and adjudicated
Client has confidence in their legal representation
Client knows what they can do to get help/knows what to do next time
Effectiveness in making an impact for clients
Client has more knowledge of their rights and responsibilities
Client has specific issues satisfactorily resolved
Each stakeholder group shares their 2 summary statements before showing the relevant slide.
Self-Evaluation Toolkit | 11

12. Theory of Change – Intermediate Outcomes for the community
Stake-holder
Domains
Intermediate Outcomes
Working with the community
Increased legal knowledge and education for the community and partners
Increased community knowledge and self-confidence in advising on legal situations
Increased community access to services that meet their legal needs
Improved ability to have legal issues addressed
Improved access to advice and support on legal matters
Increased targeted & appropriate referrals into non-legal support services
Increased number of clients accessing referral services
Self-Evaluation Toolkit | 12

13. Theory of Change – Intermediate Outcomes for Justice System
Stakehold er
Domains
Intermediate Outcomes
Working with the justice system
Effectiveness of CLCs in supporting the Justice system
(Self-represented applicants) understand the process, are prepared and have realistic expectations of possible outcomes
(Represented clients) benefit from appropriate and higher quality representation
Reduction in the incidence of non-appearance
Addresses government’s need to provide an effective service for vulnerable individuals and the broader community
Improved productive contact, engagement and collaboration with CLCs
Improved speed and efficiency in processing cases
Effectiveness of contribution to legal policy and practice
Increased exposure to advocacy on public and legal policy
Department has access to informed advice and representation on legislative issues/changes
Self-Evaluation Toolkit | 13

14. Theory of Change – Intermediate Outcomes for Volunteers
Stake- holder
Domains
Intermediate Outcomes
Engaging volunteers
Ability to leverage volunteer support
Greater awareness and empathy within law firms
Impact on volunteers engaged
Individuals gain experience in new and different areas of law
Individuals increase their understanding of clients’ circumstances and broader community issues
Self-Evaluation Toolkit | 14

15. Gathering Data
Effective outcomes measurement is dependent on collecting the right data at the right frequency so that change can be tracked over time.
Self-Evaluation Toolkit | 15

16. Gathering Data – Toolkit overview
Toolkit has specific guidance about the data to collect and how to link it to the intermediate outcomes/domains
3 main sources of data:
Hard Data – CLASS and other CLC statistical records
Survey Data – pro forma surveys for each stakeholder group and how to implement the survey process
Case studies – resources to guide writing case studies
Self-Evaluation Toolkit | 16

17. Gathering Data – Hard Data
Toolkit will update information about CLASS reporting as it becomes available which includes:
client demographics
activity types
description of services etc.
Many CLCs already produce this type of data for annual reports
Self-Evaluation Toolkit | 17

18. Gathering Data - Surveys
Each stakeholder group has a survey that tracks to the intermediate outcomes
Focus on client survey uses the toolkit survey
Queensland CLCs are required to undertake client surveys under their service agreements with the Queensland Government: clause 5.4 states:
'You must complete a client survey in the form developed through a co-design process with the Queensland legal assistance sector and the Commonwealth Government.'
Client survey is the more complex survey process compared to the 3 other stakeholder groups.
Focus on client survey – copies of the survey are the in the toolkit and hoping that it will be basis of a coordinated surveys across the sector
Self-Evaluation Toolkit | 18

19. Gathering Data - Surveys
Ideally CLCs Coordinate client surveys:
Single client survey
Centralised collection of responses and analysis
Conducted in the same month
Individual CLC reports are provided
Benchmark report are provided
State report to DJAG with opportunity for CLCs to provide comment on their data
Self-Evaluation Toolkit | 19

20. Gathering Data - Surveys
Who should complete the survey? (Survey sample)
​an 'experience-based' approach to data collection ie. collecting data when convenient, such as after service delivery or from a random sample of clients during a given time period.
the proportion of responses collected is reflective of the spread of the client demographic serviced by your CLC and the types of services provided. 
​25% as an overall figure to aim for, bearing in mind the different challenges in connecting with distinct client groups, for example, those who connected only once for advice.
​a survey sample that avoids any bias, for example, by attempting to ensure that it does not disproportionately represent respondents who may have had predominantly positive experiences.
The survey sample goes to the robustness of the data
Self-Evaluation Toolkit | 20

21. Gathering Data Surveys
How could clients complete the survey?
Four methods you could use to tell clients about the survey:
Option 1: Provide a form after an advice session or other service at your centre 
Option 2: Call former clients and ask them to undertake the survey 
Option 3: former clients and ask them to undertake the survey 
Option 4: Write a letter to former clients and ask them to undertake the survey.
Trial use of client survey – telephoning client provided the best response rate and richness in datga
Self-Evaluation Toolkit | 21

22. Gathering Data - Surveys
3 other stakeholder groups:
Community Partners
Justice System partners
Volunteers
Toolkit has the surveys for the 3 other stakeholder groups that track to the intermediate outcomes
Trial process
and/or telephone process on contacting and responding
Collation through Survey Monkey
Generally a good response rate
Smaller sample size compared to clients
Self-Evaluation Toolkit | 22

23. Gathering Data – Case studies
An effective case study describes how the work of the CLC illustrates the intermediate outcomes/domains for the stakeholder group.
DOs
Choose individual stories that suggest that they are symptomatic of a bigger problem so that the individual is not the problem
Describe the impact of the problem on people’s lives and communities
Provide an overview of the activities that summarises why the CLC was best placed to help solve the problem, rather than another organisation
Include a testimonial
Describe the change from the stakeholder's perspective 
Refer to the major area of impact (intermediate outcome)and how it was specifically achieved in this situation
Spend some time focussing on case studies. Even if the case studies are not used within the whole evaluation toolkit, they can be used for a range of purposes in telling our stories.
Workshop Activity 2
Self-Evaluation Toolkit | 23

24. Gathering Data – Case studies
DON'Ts
Provide so much background information it dilutes the impact of the CLC work on the problem
Provide blow by blow description of the activities undertaken by the CLC
Use a testimonial as the outcome e.g. 'the client was happy that their legal problem was solved' is not an outcome that describes the change created.​
Self-Evaluation Toolkit | 24

25. Gathering Data – Case studies
Example case study:
A client presented at Local CLC. He had been to see a number of government and non-government agencies as he thought he was entitled to a disability support payment but was struggling with the paperwork involved. At Local CLC our solicitor took his instructions and provided him with advice about his entitlements. The solicitor worked with the client to find and collate all the necessary paper work and contacted Centrelink to negotiate on the client’s behalf, including providing Centrelink with the client’s medical records. The client was very happy when his disability support payment was confirmed by Centrelink.
Self-Evaluation Toolkit| 25

26. Gathering Data – Case studies
3 elements of a case study: PROBLEM | SOLUTION | RESULT/OUTCOME
PROBLEM | SOLUTION | RESULT/OUTCOME
Describe the problem within the broader social context and how the problem was a threat to commonly held values e.g. access to justice, voice, security, fairness, opportunity
Situate the problem as part of a bigger social problem eg. homelessness, violence against women, police targeting marginalised groups, mortgage stress
Describe whether and/or how a community partner was part of identifying the problem.
Self-Evaluation Toolkit | 26

27. Gathering Data – Case studies
3 elements of a case study: PROBLEM | SOLUTION | RESULT/OUTCOME
PROBLEM | SOLUTION | RESULT/OUTCOME
Describe the problem within the broader social context and how the problem was a threat to commonly held values e.g. access to justice, voice, security, fairness, opportunity
Situate the problem as part of a bigger social problem eg. homelessness, violence against women, police targeting marginalised groups, mortgage stress
Describe whether and/or how a community partner was part of identifying the problem.
Self-Evaluation Toolkit | 27

28. Gathering Data – Case Studies
PROBLEM – ​SOLUTION – RESULT/OUTCOME Problem: Access to social security benefits ensures that people who have a disability can live with dignity and have their basic needs met. For people with a disability it can be challenging to negotiate a complex social security system to obtain social security benefits to which they are entitled. After approaching a number of government and non-government agencies for assistance to obtain disability support payment, Michael sought advice from Local CLC. He was experiencing high levels of stress about how to pay for his daily needs, as he did not have regular income.
Self-Evaluation Toolkit | 28

29. Gathering Data – Case Studies
Read your example case study and re-write the problem area
PROBLEM | SOLUTION | RESULT/OUTCOME
Describe the problem within the broader social context and how the problem was a threat to commonly held values e.g. access to justice, voice, security, fairness, opportunity
Situate the problem as part of a bigger social problem eg. homelessness, violence against women, police targeting marginalised groups, mortgage stress
Describe whether and/or how a community partner was part of identifying the problem.
Your Turn: Workshop activity 2 – Problem
Share the problems
Serf-Evaluation Toolkit| 29

30. Gathering Data – Case studies
PROBLEM | SOLUTION | RESULT/OUTCOME
Describe the activities that the CLC undertook and how they added value in addressing the problem
Highlight the quality and the relevance of the activities and how the activities were best suited to solving the problem
Local CLC assisted Michael by contacting Centrelink and negotiating with them about Michael’s entitlement to income support. This involved providing documents to Centrelink that showed evidence of Michael’s situation.
Your Turn: workshop activity 2 - Solution
Self-Evaluation Toolkit | 30

31. Gathering Data – Case studies
PROBLEM | SOLUTION | RESULT/OUTCOME
Describe the changes or improvements that were made as a result of the CLC activities
The end of the case study should be able finish the sentence, 'Now that the problem is solved…‘
Be able to answer the “so what” question from the stakeholder’s perspective
As a result of Local CLC’s work, Michael is now in receipt of disability income support. Michael has been able to start paying off a back log of bills, and has said that:
‘Thanks to Local CLC, I feel less stressed and more in control.’
Your Turn: Workshop Activity 2 – Result/Outcome
Each group shares their bad case study and how they turned it into a good casestudy
Presentation Title and Footer Information (change on Master page) | 31

32. Gathering Data – Case studies
PROBLEM – ​SOLUTION – RESULT/OUTCOME
​Access to social security benefits ensures that people who have a disability can live with dignity and have their basic needs met. For people with a disability it can be challenging to negotiate a complex social security system to obtain social security benefits to which they are entitled.
After approaching a number of government and non-government agencies for assistance to obtain disability support payment, Michael sought advice from Local CLC. He was experiencing high levels of stress about how to pay for his daily needs, as he did not have regular income. 
Local CLC assisted Michael by contacting Centrelink and negotiating with them about Michael’s entitlement to income support. This involved providing documents to Centrelink that showed evidence of Michael’s situation.
As a result of Local CLC’s work, Michael is now in receipt of disability income support. Michael has been able to start paying off a back log of bills, and has said that:
‘Thanks to Local CLC, I feel less stressed and more in control.’
Each group shares their bad case study and how they turned it into a good casestudy
Self-Evaluation Toolkit | 32

33. Reporting and Review
Self-Evaluation Toolkit | 33

34. Reporting and Review – Impact Reports
Impact Reports put all the data together and makes conclusions about the extent to which intermediate outcomes (domains) have been met or contributed to
Your turn – Workshop Activity 3
Using the pre-collated data for your stakeholder group write a few dot points on the issues that the data raises using the domains as your criteria.
Each group shares the issues they have identified from the data – followed by the next 4 slides that provide a full written summary.
Self-Evaluation Toolkit | 34

35. Reporting and Review – Impact Reports
Our Impact on our clients
Clients of Local CLC agree that we provide appropriate services that can assist with them resolving their legal problems and as a result they feel better equipped with how to seek help in the future. While some clients identified that they felt their legal problem was not resolved to their satisfaction, they did not attribute this to a problem with the services provided by Local CLC. The case studies illustrate that when clients have their legal problems resolved this can lead to greater impacts upon their lives such as improved family relationships, freedom from debt, and access to an income.
In the coming 12 months we look forward to working towards further increasing our access to the community through the development of a Reconciliation Action Plan.
Self-Evaluation Toolkit | 35

36. Reporting and Review – Impact Reports
Our impact in our community
Local CLC works closely with community organisation partners to ensure that members of Local community get the legal services they need when they need them. Referral to and from our community partners are a vital part being responsive to the community’s legal needs and ensuring that the people who most need our help have priority access to our services. There is some future work in enhancing Local CLC’s warm referral process to community partners to ensure clients get the extra support they need.
Local CLC provides community provides community legal education activities and resources to the Local community and community partners. Our legal education program is designed to improve the legal education and knowledge of the community and our partners, so that our community can avoid legal problems in the first place, or get the legal help they need. The evidence indicates that this is best done with community partners that work most closely with the clients that need our services and in the next 12 months we look forward to developing our CLE program in consultation with them.
Self-Evaluation Toolkit | 36

37. Reporting and Review – Impact Reports
Our impact on the justice system
This year, Local CLC has contributed the efficiency and effectiveness of the justice system by delivering services to our community that help people navigate the justice system or avoid entering the justice system in the first place. We do this through targeted service delivery that has direct impact on the effective operation of the Magistrates and Family Courts and by conducting community legal education that helps people avoid legal problems. A majority of our justice colleagues in courts and tribunal have said that our work assists them to ensure that the justice system works more effectively and efficiently.
This year Local CLC has been effective in influencing public and legal policy in areas that affect our community. Half of our justice system colleagues are of the view that we help them to better understand the issues faced by our client and how the justice system addresses them. Through strategic litigation Local CLC changed the interpretation of workplace discrimination law which will assist many more future claimants facing illegal treatment. As part of the Fund Equal Justice campaign, Local CLC contributed to the reversal of the funding cuts to CLCs across the country. In the coming year we will continue to work with Queensland Consumer Alliance on pay day lending and work with decision makers about making tenancy law fairer for tenants.
Self-Evaluation Toolkit | 37

38. Reporting and Review – Impact Reports
Our impact on volunteers
Overall, Local CLC is able to leverage a substantial amount of resources for the community through harnessing the energy, enthusiasm and commitment of our 80 volunteers. Many of the volunteers, through the survey, have indicated that their volunteering experience at Local CLC provided positive benefits to their professional development, particularly for students and professional early in their career. The challenge for Local CLC in the future is to ensure that we continue to engage experienced volunteers and that their volunteering experience at Local CLC is professionally fulfilling and productive.
Distribute the impact sections for each group
Self-Evaluation Toolkit | 38

39. Reporting and Review – Service Review
During the process of looking at the data from the last exercise and summarising the outcomes there were probably some questions or gaps that came up.
Your turn – Workshop Activity 4
Look at the data again and write a list of:
questions or areas for more information that would better understand the impact of the service or
areas for more work in the future as the data is showing some gaps
Self-Evaluation Toolkit | 39

40. Reporting and Review – Service Review
The toolkit has a number of service review questions for each stakeholder group
The service review questions can help with:
gathering better information
strategic and operational planning
developing new services or
changing existing services
Self-Evaluation Toolkit | 40

41. Reporting and Review – Service Review
Stake- holder
Domains
Service review questions
Working with clients
Appropriateness of service delivery
Who accessed the service?
Are the clients who accessed the service our intended client base for this service?
Are there any trends about the types of clients accessing the service?
Are there any client groups missing that we would expect to see accessing the service?
What level of service was provided to the clients?
Is there an underservicing of clients?
Are there any trends about problem types?
Are there problem types that are not represented but should be given the target group?
Are clients being referred to appropriate agencies? 
Are there other agencies that should be referred to, but are not?
Are the services meeting the client's needs/expectations?
Do others in the community make referrals to the service?
Effectiveness in making an impact for clients
Is the service meeting the client's legal need?
Is there a clear link between the service provided and client outcome?
Self-Evaluation Toolkit | 41

42. Reporting and Review – Service Review
Stake-holder
Domains
Service review questions
Working with the community
Increased knowledge and education for the community and partners
Have we got the right mix of CLE activities/resources, given our target audiences?
Is our CLE responsive or reactive?
Do we use our scarce CLE resources to have the greatest impact?
Have we got evaluation in place for each CLE activity/resource, so we understand their impact?
Based on what we know about our CLE activities/resources, should we be piloting any new strategies?
Increased community access to services that meet their legal needs
Are referral agencies referring clients from our target groups?
Do we need more information about how people find us?
How are our collaborative relationships going and is there positive or negative evidence in the data about the state of them?
Where are we referring clients to and what picture does this paint about our client base and their legal and social needs?
Self-Evaluation Toolkit | 42

43. Next Steps
Finish this sentence:
“Now that I know more about the Evaluation Toolkit I will……………….”
Presentation Title and Footer Information (change on Master page) | 43

44. Next Steps
We need you!
Evaluation work helps CLCQ tell the positive stories of CLC to funders, politicians, and the community:
NPA Review
More funding!
Share your impact reports with CLCQ and the community – they can form the basis for your annual report
Self-Evaluation Toolkit) | 44

45. Presentation Title and Footer Information (change on Master page) | 45