1. UKOLN is supported by: Evidence, Impact, Metrics: Proposal For Further Work Brian Kelly UKOLN University of Bath Bath, UK This work is licensed under a Attribution- NonCommercial-ShareAlike 2.0 licence (but note caveat) Acceptable Use Policy Recording this talk, taking photos, discussing the talk using Twitter, blogs, etc. is permitted but please try to minimise distractions to others. Acceptable Use Policy Recording this talk, taking photos, discussing the talk using Twitter, blogs, etc. is permitted but please try to minimise distractions to others. Blog:Twitter: http://ukwebfocus.wordpress.com/ @briankelly

2. Timetable UKOLN’s EIM Work 2010/11  Context  Examples  Additional documents Proposals for 2011/12  Work for DI programmes  Work relevant to DI programmes  Work for sector 2 Discursive Tangible deliverables, identified owners & dates That’s interesting I didn’t know that I disagree with that I want a piece of the action That will help my programme This is already being done It’s too hard

3. Background Evidence, Impact and Metrics Work Engage with community in order to explore ways of gathering evidence of the value of online services and explore appropriate metrics to facilitate comparisons Scoped to institutional use of web services and professional/institutional use of social web services Three 1-day workshops (~100 participants); 6 additional talks and ~24 evidence-based blog posts (21,800+ views) 3 Discursive

4. Changes In Cultural Some initial scepticism and mistrust We know we’re doing well Bean-counters You’ll use evidence against us Metrics are flawed Systems can be gamed …. Fortunately we had external support in changing the culture … 4 Discursive

5. The Context The need to demonstrate value in light of Government cuts 5 NB 1D view: no comments on ROI, comparisons with print, TV, … costs Discursive

6. Daily Telegraph Daily Telegraph data journalist used FOI to gather figures on Web site redevelopment costs. UK Web Focus & SiD blog posts pointed out flaws in interpretation. Sector realised that they couldn’t ignore such threats. 6 Discursive

7. Open Data Government to ‘force’ Unis to publish data about courses, student experiences, … Builds on Labour’s Open Gov initiative An opportunity to be welcomed? Can take initiative in publishing open data about uses of online services 7 Discursive

8. Numbers Matter 8 “Sales of 25 million iPads in just 14 months. 200 million iOS devices in total. 15 billion songs downloaded since 2003. 130 million books. 14 billion apps downloaded from a store that now runs to 425,000 apps.” BBC News 7 Jun 11 JISC-funded OII Impact report: “The media and the public are influenced by numbers and metrics” Discursive

9. Bath University Cares Bath University is the “” The Sunday Times (and THE) University League Tables are based on metrics Ignoring the reality can be dangerous 9 Discursive

10. Evidence-Gathering Evidence-based surveys and discussions published on UK Web Focus blog Aims: Explore methodologies (strengths & weaknesses) Provide benchmarks for future comparisons Identify patterns of uses which can help to inform practice & policy Invite & encourage discussions on survey methodologies and interpretations and sense-making processes 10 Evidence

11. Examples The Nudge Principle (cf Michael Neilson’s talk at SOLO11) applied to RSS use across Scottish Unis Use of HTML and RDFa by Welsh Unis Evidence of JISC PoWR’s impact Cookie usage by Russell Group Unis Mobile-readiness of Russell Group Unis Effectiveness of comms channels (e.g. JISCMail, Slideshare) and how evidence can inform policy 11 Evidence

12. Proposal 1 Evidence-gathering to support Linked U call Snapshot of existing approaches to HTML across selected HEIs to act as benchmark (Nov 2011) More comprehensive snapshot in Jun 2012 Report to inform JISC Presentation at IWMW 2012 aimed at engaging sector Audiences: JISC PM, Web managers Activity: Survey, interpretation and open discussion, presentation. Times: Part 1 in Nov 2011 Part 2 in Jun/Jul 2012 12 Proposal

13. Proposal 2 Evidence-gathering on use of Cookies Snapshot of existing uses of cookies and privacy policies across selected HEIs and other sectors to act as benchmark (Dec 2011) More comprehensive snapshot in Jun 2012 Analysis to inform discussions with JISC Legal Presentation at IWMW 2012 aimed at engaging sector Report on recommendations Audiences: Institutions, Web managers, JISC PMs Activity: Survey, interpretation and open discussion, presentation. Times: Part 1 in Dec 2011 Part 2 in Jun/Jul 2012 13 Proposal

14. Proposal 3 Consultancy to JISC PMs Advice provided to JISC PMs on possible ways of gathering quantitative evidence related to their programmes Audiences: JISC PMs Activity: Discussions and, if required, suggestions Times: As appropriate 14 Proposal

15. Proposal 4 Evidence-gathering to support xxxx call Snapshot across additional JISC Call in area relating to UKOLN expertise (Libraries, mobiles?) Report to inform JISC and sector Audiences: JISC PMs, xxx stakeholders Activity: Survey, interpretation and open discussion, presentation. Times: To be negotiated 15 Proposal

16. Discussion 16