Evaluation: Find out what visitors think Improve what we do Nicola Bell, MA culture ~ evaluation ~ learning

What is Evaluation? Evaluation is the systematic collection of information about the content, characteristics, and outcomes of a programme to make judgements about it, improve its effectiveness, and/or inform decisions about future activities. (Kirby, P., and Bryson, S. (2002) Measuring the Magic? Evaluating and Researching Young People’s Participation in Public Decision Making, London: Carnegie Young People's Initiative) Consulting with young people - literature review young-people-13.pdf

What is evaluation? It is Quality Control, to find out: Is it good? How can we make it better?

Why do we want to carry out an evaluation? To find out how to develop a project To find out if the project was successful To find out if the money was well spent To get money for future activities To find out what non-visitors think

If you don’t evaluate, you might: waste time waste money produce something that you can't change lose interest from your target audience lose an opportunity to learn something useful lose an opportunity to tell people about the good work you are doing lose funding

When planning an evaluation, remember: Plan the evaluation at the beginning of the project Allow enough time to collect and analyse the data Allow enough money for the evaluation Consider using a freelance evaluator who can bring an independent view to the project

Who is the evaluation for? Museum staff Project partners Project participants Funders Local or national government, future partner organisations Visitors

Types of evaluation Front end - e.g. planning new exhibit Prototyping - e.g. for a new interactive Baseline - e.g. visitors’ knowledge at the beginning Formative - e.g. how the programme is developing Summative - e.g. was the programme effective? Usually evaluations only cover a short time; longitudinal studies (over several years) would be good

Methodology: Link the evaluation to the project’s aims and objectives Aims and objectives must be made clear at the start Aim = what you are going to do? What changes will happen? (to participants, to the museum, to staff) Objectives = how you are going to do it? Objectives should be SMART

SMART objectives: S - specific M - measurable A - agreed R - realistic T - time-based

Outputs, Outcomes and Impact Logic Framework model: Inputs > activities > outputs > outcomes > impacts Examples: InputsActivitiesOutputsOutcomesImpacts Money Staff time Ideas Activities Events Exhibitions Numbers of visitors at events Number of events Changes which happen: People learn skills People have more confidence Changes which happen beyond the outcomes: Changes for visitors Changes for the museum

Outcomes can be: Positive Negative Expected Unexpected They can happen: During the project At the end of the project

What sort of data do you need to collect? Quantitative data: How much? How many? Qualitative data: What was it like? Opinions and views

Decide how many people you need to gather data from: (assuming 3% error) Population size 90% confidence level 95% confidence level 99% confidence level

Sampling The sample should be representative of the whole population (e.g. everyone who attended the exhibition) Random sampling: e.g. every 5 th or 15 th person who leaves the exhibition Convenience sampling: e.g. people who are available Snowball sampling: e.g. for people who are hard to reach - ask some people, then ask them to suggest other people (e.g. their friends) to interview

Make sure that the data collection method is suitable for the people and the information Triangulation = using several different methods, to get strong evidence partnerships.com/data/files/consulting- young-people-13.pdf Make sure that you have appropriate permissions for photography, audio recording, interviewing children

Methods of collecting data: Mediated (with a member of staff): Questionnaires (with interviewer) Interviews Focus groups Observation or tracking Participant observation Video Personal Meaning Mapping (concept maps)

Need not (necessarily) be mediated: Questionnaires (self-completion) Diaries, journals, scrapbooks Comments books Comments cards Drawings Mind maps On-line surveys (e.g. Blogs SMS, Facebook, Twitter

What do you know, think, feel about: Etnografski Muzej Zagreb Concept map: visitors write what they know etc about the topic, before the visit the exhibition. Afterwards, they write (in a different colour) what they now know, think, feel etc. Analyse comments using e.g. Generic Learning Outcomes.

Make an Evaluation Plan Who are the participants? What is the activity? Who is the evaluation for? What are the aims and objectives? Inputs, activities, outputs, outcomes, impacts? Baseline, formative, summative? Which methods?for each part of the project for each group of people (participants, staff, artists etc) How will you present the evaluation?

Analysing the data The analysis must reflect the evaluation aims and objectives Allow plenty of time (and therefore money) for analysis A written report is usually necessary for managers, funders and other stakeholders The report should give recommendations for the future The report can be in alternative formats …

Alternative formats: Website, blog DVD or video Performance Presentations Set of reference cards Book

What will you do with the evaluation when it is finished? Present it in a suitable format Tell the relevant people about it Ensure that recommendations are acted upon Don’t leave it on the shelf!

Best Practice in Educational and Cultural Action Conception of the project Achievement Evaluation Remediation / Improvement

Inspiring Learning for All Generic Learning Outcomes Generic Social Outcomes Ways of measuring hard-to-measure things Shows to other organisations what museums do, to benefit their visitors

Generic Learning Outcomes

Generic Social Outcomes

Working with community groups Youth groups Services for disadvantaged people Services for people with disabilities Communities of place Communities of interest Work with partner organisations, who already have a good relationship with local people

Why talk to teenagers and children? United Nations Convention on the Rights of the Child, article 12: “assure to the child who is capable of forming his or her own views the right to express those views freely in all matters affecting the child, the views of the child being given due weight in accordance with the age and maturity of the child.” Every child should be able to express their views and influence decisions affecting them. “Child” is anyone under 18 years old, so includes teenagers.

Why talk to teenagers and children? They have views in their own right - they are not just part of a family, part of a school, or on the way to being an adult It develops a relationship between the teenagers and the museum They can learn skills, e.g. decision making, interviewing, analysing data They can have a positive influence on their community It can increase their confidence and social skills

Ways of involving teenagers in the museum: Volunteer tour guides Provide training for them as volunteers in shop and café - good skills for jobs Young people’s forum Curating exhibitions Creating interpretation (e.g. film, guide) for an exhibition Projects to develop skills (e.g. reading, writing, art)

Young people as evaluators It is better if young people are involved in planning and doing the evaluation They can learn, and grow as a person Everyone (any age) brings their own knowledge and world-view Everyone makes their own meaning UNICEF guide to participatory evaluation with teenagers: participatory-evaluation.pdf

Maslow’s Hierarchy of Needs

Outcomes Star: a way to measure things like increase in self- confidence

Do… Make it fun! Explain the process and what influence they can have Provide a welcoming place Provide food which young people like Respect young people’s views; confidentiality, privacy Celebrate achievements Treat young people as individuals, not as a group Use e.g. artists to help with creative evaluation Be aware of needs of people with physical / learning disabilities Tell young people the outcomes of the project Provide help with transport to the museum

Don't… Make assumptions - about what they know (or don’t know), want or can do Promise too much Forget about child protection (especially in mixed age groups) Do adult things e.g. boring meetings Consult too much - people will get bored Do it too quickly – good consultation takes time