ePortfolios 2005 Electronic Portfolios Simon Cotterill School of Medical Education Development University of Newcastle upon Tyne, UK 1. What are ePortfolios ? 2. FDTL-4 & related ePortfolio projects 3. Evaluation 4. Key Issues with electronic portfolios / PDP

ePortfolios What are ePortfolios ? 2. FDTL-4 ePortfolio and related projects 3. Evaluation 4. Issues with electronic portfolios / PDP

ePortfolios 2005InstitutionalData Portfolio for Presentation Portfolio for Application (job / promotion) Portfolio for Assessment Portfolio for Accreditation/ Revalidation PPD (shared) PPD / Reflective (private) Portfolio for Appraisal Learner’s‘repository’ Central data: Transcript MIS/ HR data Programme data: Granular assessment data Outcomes / skills sets ePortfolios are learner-centric & may serve multiple purposes

ePortfolios 2005 Model presented by Jan van Tartwijk (Utrecht University) Presented at the ALT-SURF Spring Conference & Research Seminar, Edinburgh 2004 Overview (Planning / PDP) Content (Evidence) Discussion 3D Model

ePortfolios 2005 Sharable / facilitate interactionSharable / facilitate interaction Transportable - supporting continuity in LLLTransportable - supporting continuity in LLL Highly customisableHighly customisable Multi-purpose eg. formative & summative = reduced duplicationMulti-purpose eg. formative & summative = reduced duplication Multiple structures / viewsMultiple structures / views Easier cross-referencingEasier cross-referencing SearchableSearchable Reduced adminReduced admin Secure access from a range of locationsSecure access from a range of locations Not left on the bus !Not left on the bus ! 10 ‘value added’ features of an online approach to portfolios

ePortfolios What are ePortfolios ? 2. FDTL-4 ePortfolio & related projects 3. Evaluation 4. Issues with electronic portfolios / PDP

ePortfolios 2005Background 2nd Newcastle-Nottingham PARs project,

ePortfolios 2005 Managed Environments for Portfolio-based Reflective Learning. Integrated Support for Evidencing Outcomes. An FDTL-4 project: Newcastle University (lead site) Leeds University Sheffield University St Andrew’s University Dundee University (for consultancy) Developing Web based portfolios to support reflective approaches for evidencing the attainment of programme outcomes in undergraduate Medicine. Integration with managed learning environments for Medicine. Project funded by

ePortfolios 2005Newcastle

ePortfolios 2005Sheffield Reflect on Intensive Clinical Experiences (ICE)

ePortfolios 2005Leeds

ePortfolios 2005 St Andrew’s

ePortfolios What are ePortfolios ? 2. FDTL4 ePortfolio project 3. Evaluation 4. Key Issues with electronic portfolios / PDP

ePortfolios 2005 Evaluation of a Phase I Portfolio/Log Book (paper and electronic versions). (paper and electronic versions). Cotterill SJ, Sarma S, McDonald AM, Bradley P Ethical approval / informed consent Focus groups - facilitated by a 4th year student (Sean Sarma) Questionnaires - based on findings of the focus groups Key issues from the focus groups Generally well received but what motivation ? not assessed not viewed by tutors (staff interface completed later) Need better clarity of purpose Learning diary - useful at first, less so over time Good to browse the Learning Outcomes

ePortfolios 2005 Ethical approval / informed consent Questionnaires (before and after using the ePortfolio) change in awareness of intended learning outcomes ? factors influencing use of the ePortfolio attitudes and perceptions of using the ePortfolio student perception of impact of using the ePortfolio evaluation of technical features and ‘usability’ 186 students completed the portfolio (~100% compliance) 157 questionnaires completed (84% of cohort) at 30/09/2004 Evaluation of an electronic portfolio to facilitate reflective learning in stage 4 medical students during their student selected components. Cotterill SJ, McDonald AM, Bradley P, Robinson R, Hammond GR

ePortfolios 2005 Year 4 SSC Portfolio (Newcastle)

ePortfolios 2005 Building the ePortfolio was a useful learning experience 80% thought it was a useful learning experience 83% felt they had recorded good evidence 93% reflected on their learning after the option Having clearly defined intended learning outcomes influenced the way in which I approached the option 72% felt that the LOs influenced their approach n = 157

ePortfolios 2005 Evaluation - positive (SSC portfolio) “ It encouraged me to really give thought to what I wanted to achieve during the option, which was especially useful as this was my first option. As a result of the portfolio I think I got much more out of the option than I would have otherwise. ” “ It made me concentrate on creating aims at the start of the option and allowed me to plan the option with my supervisor in a defined way. Overall it made my learning for the option more organised and focused. ” Planning and approach to learning:

ePortfolios 2005 Evaluation – negative (SSC portfolio) “ Most of the things I learned couldn't be 'measured/quantified' so I felt it was of little value to try and invent a way in which they could. ” “ Access to computer as option took place in Castle, records may have been more detailed if completed straight after meetings etc. ” Hard to measure: Access to IT facilities:

ePortfolios What are ePortfolios ? 2. FDTL4 ePortfolio project 3. Evaluation 4. Key Issues with electronic portfolios / PPD

ePortfolios 2005 Humans are important too ! Need buy-in of key stakeholders: Learners Clarity of purpose ? Sense of Ownership ? Tutors & Admin staff Extra work ? Resources ? Senior Curriculum staff Employers / Professional bodies Integration with existing workflow and/or change in practice and culture ? Training and support (all) Monitoring and evaluation

ePortfolios 2005 Software needs to match pedagogy (and not visa-versa !) Use or develop flexible solutions To meet both: Institutional requirements Subject-specific requirements eg. The FDTL4 ‘ generic ’ ePortfolio is component-based: High level of customisation eg. nomenclature Facility for course administrators to create context-specific tools via simple Web forms, without detailed technical skills. Local developers can develop more sophisticated tools using Open Source products. Share and contribute back to the ePortfolio community.

ePortfolios 2005 Getting the right balance Assessed Necessary for motivation / engagement ? “Either increase its importance or bin it - sitting on the fence is utterly pointless” First year Medical student talking about a newly introduced non- assessed paper log-book & portfolio (2004) Non-assessed Necessary for open-honest reflection

ePortfolios 2005 Towards a life-long learning record School HE Continuing Development Interoperability (technical standards to support the interaction and transfer of data between IT systems) IMS specifications /standards: Learner Information Package (LIP) Enterprise / Content Packaging ePortfolio (early stage of development) CETIS UK Learner Profile (based on IMS-LIP) JISC ePET project to transfer data to/from the Newcastle ‘generic’ ePortfolio Can I get my data out ?

ePortfolios 2005 Research and Evaluation Does the ePortfolio improve learning ? How so ? Some evidence that PDP improves learning (more needed) A systematic map and synthesis review of the effectiveness of personal Development Planning for improving student learning. EPI Centre Can ePortfolios foster a reflective approach to learning ? How defined ? How measured ? How do particular groups of learners approach reflective portfolios ? Learning Styles Gender and Age Cultural Background John Mole. The geography of thinking. Clin Med 2002; 2:343-5 Does the technology change the learning process ? What are learner and staff perceptions / attitudes towards ePortfolios ?

ePortfolios 2005 Current & Future Developments  FDTL4 Transferability Project (with St Georges Hospital Medical School, London)  Involvement in 4 JISC regional ePortfolio projects (FE – HE)  ePortfolio support for 2 CETLs in Health (incl. use of mobile devices)  Dentistry undergraduate + vocational training  ePortfolios in other contexts:  Postgraduate Research Students  Contract Research Staff  Undergraduate Bioscience

ePortfolios 2005 Further Information: Further Information: Cotterill SJ., McDonald AM., Drummond P., Hammond GR. Design, implementation and evaluation of a ‘ generic ’ e-portfolio: the Newcastle experience (Proc ePortfolios 2004) Hennessy S., Howes A. Using ePortfolios to assess the Reflective Capabilities of Medical Students (Proc. ePortfolio 2004) Papers and bibliography available at:

ePortfolios 2005 Context-specific tools ‘Generic’ tools ePortfolio framework Common contents structure sharing cross-referencing searchable integrated action planning Customisation select tools by course/year outcomes / skills sets nomenclature graphics / layout simple tool editor CV Outcomes / skills log PDP Reflective learning diary Meetings log SWOT Integrated into the common contents structure Flexible component-based architecture Developed with Open Source products: Zope, MySQL

ePortfolios 2005 Ability for Learners to Share their Portfolio Content Users can control access to their portfolio

ePortfolios 2005 Integration with a VLE (via a ‘tab’ in Blackboard)