1. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Design, implementation and evaluation of a 'generic' ePortfolio: the Newcastle experience. Simon Cotterill, Tony McDonald, Paul Drummond, Geoff Hammond School of Medical Education Development University of Newcastle, UK

2. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk SST (Student Support & Tutoring) FDTL-4 project ePortfolios in Medicine Evidencing outcomes PDP 1 st Newcastle-Nottingham PARs project, 1998-2000 2nd Newcastle-Nottingham PARs project, 2000-02 Internet-PARs DfES funded PDP focus Background

3. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk An FDTL-4 project developing ePortfolios for undergraduate Medicine (2002-2005): Newcastle University (lead site) Leeds University Sheffield University Dundee University (for consultancy) Project funded by Background

4. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Aims 1. Personal Development Planning (PDP) "a structured and supported process undertaken by an individual to reflect upon their own learning, performance and/or achievement and to plan for their personal, educational and career development” Quality Assurance Agency for Higher Education, 2001.

5. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk a) Towards a life-long learning record School HE CPDAims 2. Supporting Life-Long Learning (LLL) b) Encouraging the development of LLL Skills: Evidencing Learning Outcomes Transition to Independent learning Preparation for professional requirements: - Appraisals - Assessment - Revalidation

6. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Aims (technical) 3. Design a ‘generic’ ePortfolio -transferable / useful in different contexts 4. Integration with VLEs

7. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Context-specific tools Common tools ‘Generic’ framework Common contents structure sharing cross-referencing searchable integrated action planning Customisation select tools by course/year outcomes / skills sets nomenclature graphics / layout simple tool editor CV Outcomes / skills log PDP Reflective learning diary Meetings log SWOT Integrated into the common contents structure Architecture Developed with Open Source products: Zope, MySQL

8. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk create context-specific tools via simple Web forms Student view Selecting tools by Course and Year Course Admin view ‘Generic’ framework: Customisaton

9. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Configuring learning outcomes / skills sets Course Admin view ‘Generic’ framework: Customisaton

10. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk ‘Generic’ framework: Sharing

11. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Piloting & Implementation The ePortfolio developed at Newcastle has been applied in the following contexts: 2003/4Undergraduate Medicine Contract Research Staff (CPD) Generic Portfolio (project demonstrator) 2004/5Dentistry (Undergrad + Vocational Training) Undergraduate Biosciences St Andrew’s University (Medicine)

12. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Generic ePortfolio: Public Demonstrator http:www.eportfolios.ac.uk

13. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Medicine at Newcastle: Integration with a bespoke VLE

14. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Common Tools: Learning Diary (Medicine – Newcastle)

15. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Context-specific Tools: PDP for Year 4 Student Selected Components

16. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Common Tools: Record courses (CRS)

17. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Common Tools: Learning Outcome / Skills (CRS)

18. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Common Tools: SWOT (Bioscience prototype)

19. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Common Tools: CV (St Andrew’s University)

20. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Common Tools: Tutor meetings (St Andrew’s University)

21. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Context-specific Tools: Dentistry Project involves 6 Dental schools and 2 Postgraduate Deaneries

22. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Promoting Continuity & LLL in Dentistry Context-specific Tools: Dentistry

23. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Integration #2: Blackboard (via a ‘tab’)

24. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Integration #2: Blackboard (via the tool bar)

25. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Evaluation #1: Medicine (Years 1&2) Cotterill SJ, Sarma S, McDonald AM, Bradley P Ethical approval / informed consent Focus groups + Questionnaires Key issues: Generally well received but what motivation ? not assessed not viewed by tutors (staff interface completed later) Need better clarity of purpose Wanted a more quirky / fun design ! Learning diary - useful at first, less so over time Good to browse the Learning Outcomes

26. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Ethical approval / informed consent Questionnaires (before and after using the ePortfolio) Change in awareness of intended learning outcomes ? Factors influencing use of the ePortfolio Attitudes and perceptions of using the ePortfolio Student perception of impact of using the ePortfolio Evaluation of technical features and ‘usability’ 186 students completed the portfolio (~100% compliance) 105 questionnaires completed (56% of cohort) at 30/06/2004 Evaluation #2: Medicine (Year 4) PDP for student selected components Cotterill SJ, McDonald AM, Bradley P, Robinson R, Hammond GR

27. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Evaluation - positive “ It encouraged me to really give thought to what I wanted to achieve during the option, which was especially useful as this was my first option. As a result of the portfolio I think I got much more out of the option than I would have otherwise. ” “ It made me concentrate on creating aims at the start of the option and allowed me to plan the option with my supervisor in a defined way. Overall it made my learning for the option more organised and focused. ”

28. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Evaluation - barriers & problems: “Most of the things I learned couldn't be 'measured/quantified' so I felt it was of little value to try and invent a way in which they could.” “ Access to computer as option took place in Barnard Castle, records may have been more detailed if completed straight after meetings etc. ”

29. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Building the ePortfolio was a useful learning experience 80% thought it was a useful learning experience Having clearly defined intended learning outcomes influenced the way in which I approached the option 72% felt that the LOs influenced their approach

30. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk 93% reflected on their learning after their SSC At the end of the option I spent time considering what I had learned from this option.

31. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Conclusions #1 The ‘generic’ ePortfolio Highly configurable Can be readily adapted for different contexts Existing tools have an emphasis on PDP -but the portfolio framework can serve other purposes Integration in VLEs Fully integrated into a VLE for Medicine Initial work to integrate with Blackboard has been successful

32. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Evaluation (in Medicine) ePortfolios implemented / piloted in 2003/4 Demonstrated: Feasibility & Acceptability Most students perceive these as beneficial However, unless assessed / mandatory the motivation to complete is reduced. Conclusions #2

33. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk A systematic map and synthesis review of the effectiveness of personal development planning for improving student learning. EPPI-Centre (2002) http://eppi.ioe.ac.uk Educational Objectives (including PDP) Some evidence that PDP had a positive impact on planning and approach to learning (Year 4 SSC) Encouraged reflection in some students Good preparation (and practice) for LLL & CPD However, long-term impact on learning is unknown Further research is essential Some (limited) evidence that PDP improves learning: Conclusions #3

34. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Additional Developments: ePortfolios Extension Toolkit (ePET) -A JISC funded project to develop a ‘Web services’ interface for the portfolio (by 04/2005) http://www.eportfolios.ac.uk/ePET IAMSECT ‘Shibboleth’ Single Sign-On project -JISC funded project http://iamsect.ncl.ac.uk Further information on the ePortfolio: http://www.eportfolios.ac.uk Simon Cotterill (S.J.Cotterill@ncl.ac.uk)

35. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk Implementation of ePortfolios - Humans are important too ! Buy-in of key stakeholders: Learners Clarity of purpose ? Sense of Ownership ? Tutors & Admin staff Extra work ? Senior Curriculum staff Integration with existing processes Training (all) Monitoring and evaluation

36. A 'generic' ePortfolio: the Newcastle experience Background Aims Design Implement Evaluation Conclusions www.eportfolios.ac.uk InstitutionalData Portfolio for Presentation Portfolio for Application (job / promotion) Portfolio for Assessment Portfolio for Accreditation/ Revalidation PDP (shared) PDP / Reflective (private) Portfolio for Appraisal Learner’s‘repository’ Central data: Transcript MIS/ HR data Programme data: Granular assessment data Outcomes / skills sets Selecting sub-sets of portfolio data for different purposes: