© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Principles of Electronic Portfolios Simon Cotterill, Tony McDonald School of Medical Education Development University of Newcastle upon Tyne, UK

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 The workshop draws on the experiences of developing, implementing and evaluating Web-based ePortfolios, at the University of Newcastle, UK.Objectives To discuss some of the principles of electronic portfolios To gain familiarity with the potential educational and technical ‘value-added’ features of using electronic media. To consider approaches to the implementation of ePortfolios Proposed Structure Presentation: Principles of ePortfolios Exercise: ePortfolios in your context Presentation: Case study: FDTL4 ePortfolio project Exercise: Strengths and weaknesses Plenary / discussion

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004UnstructuredFormativeSample Best work Factual / Quantitative Learner Owned StructuredSummative All work Representative Reflective / Interpretive Employer Owned Introduction – What are Portfolios ?

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Introduction – What are Portfolios? A collection of work or evidence on performance –Processes (the ‘journey’) –Learning Outcomes (the ‘destination’) –Record volume and variety of experiences  Essays, project work, logs of experiences/achievements, artwork, records of accomplished work, audits etc. Portfolios are defined by their PURPOSE(s) –Employment: Builder gets commissions on the basis of his record of satisfied customers –Portfolio for application: The researcher includes a personal profile in a grant application to show experience –Professional requirements: Med/Law/Education - evidence of essential experience and competency that can be audited

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 A computer application which stores the user’s information on a local drive.A computer application which stores the user’s information on a local drive. A static Web page eg. CV on the InternetA static Web page eg. CV on the Internet A portfolio composed using ‘simple tools’ (e.g. Word, Dreamwaever, Powerpoint etc.)A portfolio composed using ‘simple tools’ (e.g. Word, Dreamwaever, Powerpoint etc.) A sophisticated database-driven system, typically accessed over a network.A sophisticated database-driven system, typically accessed over a network. What might an Electronic Portfolio be?

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Model presented by Jan van Tartwijk (Utrecht University) Presented at the ALT-SURF Spring Conference & Research Seminar, Edinburgh 2004 Overview (Planning / PDP) Content (Evidence) Discussion 3D Model

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, Should bring significant advantages over paper- based alternatives 2. Should be considered in the context of wider (human) processes 3. Clarity of purpose(s) 4. One size does not fit all 5. Should be learner-centric 6. Should be an integral part of the learning experience 7. Should support life-long learning 8. Research / evaluation is essential ….not an exhaustive list ! Principles of Electronic Portfolios

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Principle 1. ePortfolios should bring significant advantages over paper-based alternatives

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Highly customisableHighly customisable Multi-purpose eg. formative & summative = reduced duplicationMulti-purpose eg. formative & summative = reduced duplication Multiple structures / viewsMultiple structures / views Easier cross-referencingEasier cross-referencing Sharable / facilitate interactionSharable / facilitate interaction TransportableTransportable SearchableSearchable Reduced adminReduced admin Secure access from a range of locationsSecure access from a range of locations Not left on the bus !Not left on the bus ! 10 ‘value added’ features of an online approach to portfolios

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Potential Weaknesses of ePortfolios There may be barriers to effective use of electronic portfolios e.g. Limited access to computers / InternetLimited access to computers / Internet Varied IT experience & Technophobia !Varied IT experience & Technophobia ! Reliability of IT systemsReliability of IT systems Possible duplication / overlap with paper systemsPossible duplication / overlap with paper systems Research / Evaluation is essential (Principle 8)Research / Evaluation is essential (Principle 8) Optimisation will take a number of yearsOptimisation will take a number of years

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Principle 2. ePortfolios should be considered in the context of wider (human) processes

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Humans are important too ! Need buy-in of key stakeholders: Learners Clarity of purpose ? Sense of Ownership ? Tutors & Admin staff Extra work ? Senior Curriculum staff Employers / Professional bodies Integration with existing workflow and/or change in practice and culture ? Training and support (all) Monitoring and evaluation

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Principle 3. Clarity of purpose(s) Choose / design your portfolio according to purpose Learners and assessors also need to be clear about purpose and requirements (motivation / fairness) ePortfolios can readily support multiple purposes Portfolio assessment

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Summative and Formative Purposes Portfolios are often used to evidence the achievement of learning outcomes for summative assessment.Portfolios are often used to evidence the achievement of learning outcomes for summative assessment. There may be potential problems if the portfolio serves both formative and summative processes.There may be potential problems if the portfolio serves both formative and summative processes. Reflection is less likely to be open and honest if the learner knows that the work will be assessed (loss of authenticity).Reflection is less likely to be open and honest if the learner knows that the work will be assessed (loss of authenticity). The production of a portfolio can itself be a formative learning process ie. it is as much a ‘journey’ as an end-point for assessment.The production of a portfolio can itself be a formative learning process ie. it is as much a ‘journey’ as an end-point for assessment.

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 The use of electronic portfolios may potentially reduce the tension between formative and summative processes by supporting both assessed and private / non-assessed content.The use of electronic portfolios may potentially reduce the tension between formative and summative processes by supporting both assessed and private / non-assessed content. Students have the choice to select which content is private and which is made available to assessors, appraisers, tutors, peers, and others involved in their education.Students have the choice to select which content is private and which is made available to assessors, appraisers, tutors, peers, and others involved in their education. Summative and Formative Purposes (2)

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004InstitutionalData Portfolio for Presentation Portfolio for Application (job / promotion) Portfolio for Assessment Portfolio for Accreditation/ Revalidation PDP (shared) PDP / Reflective (private) Portfolio for Appraisal Learner’s‘repository’ Central data: Transcript MIS/ HR data Programme data: Granular assessment data Outcomes / skills sets Supporting Multiple Purposes

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Portfolio Assessment Key issues: Validity Validity Reliability Reliability Attitudes & behaviours Attitudes & behaviours Time demands Time demands Baume D. A Briefing on the Assessment of Portfolios. LTSN Generic Centre. Assessment Series No.6 (2001) Friedman M, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical Teacher. 2001; 23: begin Where to begin ? Knows Knows how Does Shows how Miller GE. Acad Med 1990:S63-7 Assessing the development of competencies over time

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 ePortfolios and Assessment -Potential benefits of ePortfolios: Formative input during portfolio building (not just at end) Formative input during portfolio building (not just at end) Random sampling & ‘drilling down’ Random sampling & ‘drilling down’ Peer Assessment (formative and summative) Peer Assessment (formative and summative) Automatic inclusion of assessment results + feedback Automatic inclusion of assessment results + feedback Automated summary information (quantitative) Automated summary information (quantitative) Reduced monitoring / admin demands Reduced monitoring / admin demands -Piloting essential before use in ‘high-stakes’ assessment -Need for further research in this field ! ePortfolio of assessment (results / essays etc) for assessment

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Principle 4. One size does not fit all Institutional Subject / context requirements vs. specific (standardisation) requirements ePortfolios should be highly flexible / customisable support diversity (accessibility, learning styles etc)

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 The need for flexibility & customisation install from a set of ‘generic’ tools create context-specific tools via simple Web forms Course Admin view Student view e.g. selecting tools by course / year groups

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 e.g. configuring learning outcomes / skills sets Course Admin view Student view The need for flexibility & customisation (2)

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Personalisation / customisation Sense of ownership / privacy / sharing Constructivist theories. is eLearning contributing to ‘Paradigm shift’ ? StudentIndependent Learner TeacherFacilitator DidacticInteractive + Explorative 1 locationDistance learning 1 providerMultiple providers Principle 5. ePortfolios should be learner-centric

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Students control which parts of their portfolio can be viewed for specific purposes or by specific people Mixed-Ownership Model Corporate (HR) data Personal Data Tutor’s Data Emphasis on Learner Ownership / Control

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Ability for Learners to Share their Portfolio Content Users can control access to their portfolio

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Flexibility for the Learner to Add, Link & Cross-Reference ePortfolio Artefacts Adding ‘child’ objects to an artefact Cross-reference An artefact with A skill

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Principle 6. ePortfolios should be an integral part of the learning experience Educational (integration with the curriculum) Technical (integration within the MLE)

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 ePortfolio integrated with the VLE Tool properties include ‘module code’ = dynamic links to Study Guides + resources Learning outcomes linked to curriculum databases: ‘modules’ and units Links to other systems eg. SSC selection Educational integration Should not be perceived as a ‘bolt-on’ Support a holistic view of curriculum / outcomes (as well as specific components) Technical integration (example from Medicine at Newcastle)

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Principle 7. ePortfolios should support life-long learning support life-long learning Help develop life-long learning skills Continuity in LLL (transferability)

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Towards a life-long learning record School HE Continuing Development Interoperability (technical standards to support the interaction and transfer of data between IT systems) IMS specifications /standards: Learner Information Package (LIP) Enterprise / Content Packaging ePortfolio (early stage of development) CETIS UK Learner Profile (based on IMS-LIP) ePortfolio interaction with VLEs / MLEs Continuity in LLL (transferability)

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Preparation for professional requirements: 3.Independent learning skills 4.Personal development planning 5.Time-efficient evidencing of performance / development for: - Appraisals - Assessment - Revalidation Aiming to promote a reflective / deep-learning approach ….research The ePortfolio process should promote the develop life-long learning skills

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Principle 8. Research / Evaluation is Essential Are the purpose(s) being fulfilled ?

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 ‘A good fit happens over time with lots of use’ Spandel, 1997 cited by Jan van Tartwijk (ALT-SURF Conference, 2004) Action Research approaches Incremental development and refinement Qualitative research methodologies (+some quantitative - not mutually exclusive!) Ethical and practical considerations Evaluation and Refinement

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Research and Evaluation Does the ePortfolio improve learning ? How so ? Some evidence that PDP improves learning (more needed) A systematic map and synthesis review of the effectiveness of personal Development Planning for improving student learning. EPI Centre Can ePortfolios foster a reflective approach to learning ? How defined ? How measured ? How defined ? How measured ? How do particular groups of learners approach reflective portfolios ? Learning Styles Learning Styles Gender and Age Gender and Age Cultural Background Cultural Background John Mole. The geography of thinking. Clin Med 2002; 2:343-5 Does the technology change the learning process ? What are learner and staff perceptions / attitudes towards ePortfolios ?

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, Should bring significant advantages over paper- based alternatives 2. Should be considered in the context of wider (human) processes 3. Clarity of purpose(s) 4. One size does not fit all 5. Should be learner-centric 6. Should be an integral part of the learning experience 7. Should support life-long learning 8. Research / evaluation is essential ….the challenge is putting the principles into practice ! Principles of Electronic Portfolios

© 2004, School of Medical Education Development, University of Newcastle upon Tyne ePortfolios Ottawa Conference, 2004 Further information: Further information: