What difference has it made? Evaluation and impact assessment for NOF-digitise projects Susi Woodhouse, Senior Network Adviser

Why evaluation? NOF digitise is the largest programme of publicly-funded content creation so far supported by the Government and as such will deliver a ground-breaking framework of resources to support learning in its broadest sense and will provide a foundation for the future.

Proof of concept Demonstrate value and impact –a political tool –a financial lever –a service development baseline –added value and additionality Learn from our actions Share lessons and management models Plan for a sustainable future

Many stakeholders Government (national and local) Strategic agencies (Resource, Regional bodies) Funding bodies (NOF) Cross-sectoral involvement (libraries, museums, archives, education, community,voluntary...) …and USERS, users, users...

Measuring Performance InputsProcessesOutputsOutcomesImpacts £ spent Learning packages used Items digitised No. of web visits Fulfilled Citizens

What is “Impact”? An effect on an individual or group which –may be positive or negative –may be what was intended or unintentional –may result in changed attitudes behaviours products –may be short or long term –may be critical or trivial –…………….and so on

Impact and Learning Surface learning –Skim, memorise, regurgitate for tests Little long-term impact Deep learning –Involved, questioning, interactive, integration with own knowledge Significant long-term impact A habit of critical thinking

Learning and Critical Thinking Problem identification –A triggering event arouses interest Problem definition –The subject is clarified; ways of exploring it are identified; links are made to personal experience Problem exploration –New approaches & new solutions are explored; issues are understood; ideas are disentangled Problem applicability –Solutions are judged; practical knowledge is assessed Problem integration –Solutions are acted upon; ideas are applied (After Garrison, 1992)

Levels of Impact 6 Changed action 5 Changed world view 4 Changed perception 3 Improved knowledge 2 Better Informed 1 Awareness raised 0 None -1 Dismissive -2 Hostility

Measuring Impact Who is making judgements? –the user? –the institution? –the government? –society? What does each of them mean by ‘good’? What does each of them think the project is for?

What are digi projects for? What would positive impacts look like? The website –A statement which draws attention to its producers –Happy users, made so by a well-designed, easy-to-use product that engages interest

What are digi projects for? What would positive impacts look like? Sustainability and visibility –Objects still available, in good condition, in 100 years’ time –Use of digital surrogates raises awareness of originals and creates greater demand for access to them and other items –promotes role and purpose of host institutions –other institutions wish to contribute materials to the site

What are digi projects for? What would positive impacts look like? To provide access knowledge –Sought information immediately available –Unknown sources brought to user’s attention –Facilities and services universally available –materials re-purposed by others (e.g. schools, broadcasting, private sector)

What are digi projects for? What would positive impacts look like? Supporting learning –Increased literacy of users who are not exposed to other learning –learners using digi materials pass examinations –evidence of increased confidence and sense of wellbeing in users

What are digi projects for? What would positive impacts look like? Interpretation, selection and context- setting of materials –Users not left to fend for themselves. –No information overload and therefore less stressed than the average “googlebug” or “Yahoo Henrietta” CONCLUSION: Before you analyse your project, be clear what it is for!

What do we measure? Impacts –We CANNOT measure impact directly –We CAN measure surrogates of impact The real questions are about determining which surrogates are adequate for our purposes

Some surrogates How big the website is How much it cost How many digital objects it contains How much they cost How many staff you employ How much they cost How many letters the project staff have after their names

Some other surrogates What the users say about the service –warm-tummy stories What users actually do –understanding how the site is used What users actually produce –what the impact of using the site is (e.g. new skills, new job, coursework, their own contribution to the site)

Stories from the web Site providing a service which encourages development of children as fulfilled readers and creative writers, and develops their IT skills 7.8million hits over 12 months, 28% stayed longer than 19 minutes Stories Clubs in many libraries

Stories impacts Children attending the clubs have increased IT skills and book knowledge Children are using the library more Family members are using the library more skills and confidence of children and staff all increased

The warm-tummy factor “I think your web page is wonderful! I even have our site on ‘Favourites’ because it is one of my favourite sites!” (Samantha, aged 10) “This is the best web ever, I’ve started to write stories now” (Jane, aged 8)