1. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 1 Institute for Learning and Research Technology Dr. Grainne Conole

2. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 2 Institute for Learning and Research Technology An integrated approach to evaluating learning technologies Grainne Conole, Ed Crewe - University of Bristol Martin Oliver - University College London Jen Harvey - Dublin Institute of Technology

3. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 3 Institute for Learning and Research Technology Toolkits as an approach to evaluating and using learning materials Grainne Conole, Ed Crewe - University of Bristol Martin Oliver - University College London Jen Harvey - Dublin Institute of Technology

4. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 4 Institute for Learning and Research Technology Background Evaluation Assistant is –an online toolkit to help practitioners carry out evaluations –a six-month project funded by the UK JISC Committee for Awareness, Liaison and Training (JCALT) –a collaboration between Bristol University, University College London the Dublin Institute of Technology

5. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 5 Institute for Learning and Research Technology Toolkits - a definition Toolkits lie between –Frameworks provide theoretical overview or point of reference less restrictive than toolkits, also less supportive –Wizard easy to use software ‘black boxes’ software tools which can make decisions draws on pre-defined templates easier to use than a toolkit, but more restrictive

6. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 6 Institute for Learning and Research Technology Underlying assumptions Designed to  be easy-to-use for practitioners and produce demonstrable benefit  provide guidance, but not be prescriptive  be adaptable and easy to customise to the local context  establish a comprehensive resource of relevant material

7. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 7 Institute for Learning and Research Technology Outline Provides a structured resource that –that can be used to plan, scope and co-ordinate an effective evaluation –provides progressively more detailed information on particular topics and links to appropriate resources –allows the user can follow up relevant issues –reduces the time taken to plan an evaluation –can be used iteratively over time

8. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 8 Institute for Learning and Research Technology Toolkit architecture The toolkit consists of three components –Evaluation Planner helps define and scope the evaluation –Evaluation Advisor guides the user through the process –Evaluation Presenter provides support for presenting the results

9. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 9 Institute for Learning and Research Technology Evaluation planner Consists of seven stages –About planner –What are you evaluating? –Reasons –Context –Who is it for –Devising the question –Summary

10. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 10 Institute for Learning and Research Technology Who’s the evaluation for? This stakeholder analysis: –defines the stakeholders for the evaluation ie anyone who may have a stake in the evaluation –looks at the different types of stakeholders and their associated concerns and issues –maps the stakeholders to the evaluation process

11. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 11 Institute for Learning and Research Technology Formulating the question(s) Aligned to stakeholders analysis Question(s) refined to: –explore stakeholder concerns –provide comparison –yield measurement –challenge or expose stakeholder’s perspective

12. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 12 Institute for Learning and Research Technology Evaluation advisor Consists of four stages –About advisor –Data capture methods –Data analysis –Summary

13. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 13 Institute for Learning and Research Technology Filtering of choices At each stage of advisor –users are asked a series of questions –these help formulate the evaluation –this is linked to a filtering process –leading to a defined choice of data capture methods and data analysis

14. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 14 Institute for Learning and Research Technology Evaluation presenter Consists of four stages –About presenter –Closing the loop –Presentation tools

15. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 15 Institute for Learning and Research Technology Presenting the findings The final step –closes the loop by relating findings back to the stakeholders Options for presenting findings include:  Data sets or spreadsheet of costings  Executive summary of activity  Narrative account of the evaluation  Oral presentation or poster of findings  Research reports

16. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 16 Institute for Learning and Research Technology Further information Web site: http://www.ltss.bris.ac.uk/JCALT/ Email: g.conole@bristol.ac.uk Project team –University of Bristol Grainne Conole, Ed Crewe, Martin Belcher –University College London Martin Oliver –Dublin Institute of Technology Jen Harvey

17. The Institute for Learning and Research Technology is a centre of excellence in the development and use of Information and Communication Technology to support learning and research 17 Institute for Learning and Research Technology Relevant references Conole, G. & Oliver, M. (1998), A pedagogical framework for embedding C and IT into the curriculum. ALT-J, 6 (2), 4-16 Oliver, M. & Conole, G. (1998), Evaluating Communication and Information Technologies: a toolkit for practitioners. Active Learning, 8, 3-8 Conole, G., Oliver, M. & Harvey, J. (2000), ‘Scoping study’, a report for the ‘Evaluation Toolkit for practitioners JISC JCALT project, University of Bristol Oliver, M. & Conole, G., (2000), Assessing and enhancing quality using toolkits. Journal of Quality Assurance in Education, 8, 1, 32-37 Conole, G., Oliver, M. & Harvey, J. (2000), ‘Toolkits for practitioners’, ALT-C 2000, Manchester Conole, G., Oliver, M. & Harvey, J. (2000), ‘Toolkits as an approach to evaluating and using learning materials’, ASCILITE 2000, Coffs Harbour, New South Wales Conole, G., Oliver, M. & Harvey, J. (2000), ‘An integrated approach to evaluating learning technologies’, IWALT 2000, Palmerston North, New Zealand