Principles of Electronic Portfolios Simon Cotterill, Tony McDonald School of Medical Education Development University of Newcastle upon Tyne, UK

Objectives Proposed Structure The workshop draws on the experiences of developing, implementing and evaluating Web-based ePortfolios, at the University of Newcastle, UK. Objectives To discuss some of the principles of electronic portfolios To gain familiarity with the potential educational and technical ‘value-added’ features of using electronic media. To consider approaches to the implementation of ePortfolios Proposed Structure Presentation: Principles of ePortfolios Exercise: ePortfolios in your context Presentation: Case study: FDTL4 ePortfolio project Exercise: Strengths and weaknesses Plenary / discussion

Introduction – What are Portfolios ? Unstructured Formative Sample Best work Factual / Quantitative Learner Owned Structured Summative All work Representative Reflective / Interpretive Employer Owned Portfolios have been around for years, mean all things to all people.

Introduction – What are Portfolios? A collection of work or evidence on performance Processes (the ‘journey’) Learning Outcomes (the ‘destination’) Record volume and variety of experiences Essays, project work, logs of experiences/achievements, artwork, records of accomplished work, audits etc. Portfolios are defined by their PURPOSE(s) Employment: Builder gets commissions on the basis of his record of satisfied customers Portfolio for application: The researcher includes a personal profile in a grant application to show experience Professional requirements: Med/Law/Education -evidence of essential experience and competency that can be audited Aims of the session What are ePortfolios ? Portfolios in Medicine: context and rationale

What might an Electronic Portfolio be? A computer application which stores the user’s information on a local drive. A static Web page eg. CV on the Internet A portfolio composed using ‘simple tools’ (e.g. Word, Dreamwaever, Powerpoint etc.) A sophisticated database-driven system, typically accessed over a network. The term “electronic portfolio” is potentially confusing as there are many definitions…

3D Model Overview (Planning / PDP) Content Discussion (Evidence) Model presented by Jan van Tartwijk (Utrecht University) Presented at the ALT-SURF Spring Conference & Research Seminar, Edinburgh 2004

Principles of Electronic Portfolios 1. Should bring significant advantages over paper- based alternatives 2. Should be considered in the context of wider (human) processes 3. Clarity of purpose(s) 4. One size does not fit all 5. Should be learner-centric 6. Should be an integral part of the learning experience 7. Should support life-long learning 8. Research / evaluation is essential ….not an exhaustive list !

Principle 1. ePortfolios should bring significant advantages over paper-based alternatives

10 ‘value added’ features of an online approach to portfolios Highly customisable Multi-purpose eg. formative & summative = reduced duplication Multiple structures / views Easier cross-referencing Sharable / facilitate interaction Transportable Searchable Reduced admin Secure access from a range of locations Not left on the bus ! My Portfolio Easier cross-referencing eg. linking a record for course with one ore more learning outcomes this has a pedagogic impact as it encourages a more ‘longitudinal outlook’ rather than just simply completing a linier sequence of forms. Shareable potential for formative input and peer-learning Transportable can be output in a variety of formats Reduced admin monitor form completion and automate reminders

Potential Weaknesses of ePortfolios There may be barriers to effective use of electronic portfolios e.g. Limited access to computers / Internet Varied IT experience & Technophobia ! Reliability of IT systems Possible duplication / overlap with paper systems Research / Evaluation is essential (Principle 8) Optimisation will take a number of years There are some potential drawbacks to electronic portfolios.

Principle 2. ePortfolios should be considered in the context of wider (human) processes

Humans are important too ! Need buy-in of key stakeholders: Learners Clarity of purpose ? Sense of Ownership ? Tutors & Admin staff Extra work ? Senior Curriculum staff Employers / Professional bodies Integration with existing workflow and/or change in practice and culture ? Training and support (all) Monitoring and evaluation

Principle 3. Clarity of purpose(s) Choose / design your portfolio according to purpose Learners and assessors also need to be clear about purpose and requirements (motivation / fairness) ePortfolios can readily support multiple purposes Portfolio assessment

Summative and Formative Purposes Portfolios are often used to evidence the achievement of learning outcomes for summative assessment. There may be potential problems if the portfolio serves both formative and summative processes. Reflection is less likely to be open and honest if the learner knows that the work will be assessed (loss of authenticity). The production of a portfolio can itself be a formative learning process ie. it is as much a ‘journey’ as an end-point for assessment.

Summative and Formative Purposes (2) The use of electronic portfolios may potentially reduce the tension between formative and summative processes by supporting both assessed and private / non-assessed content. Students have the choice to select which content is private and which is made available to assessors, appraisers, tutors, peers, and others involved in their education.

Supporting Multiple Purposes Portfolio for Presentation Portfolio for Assessment Portfolio for Application (job / promotion) Portfolio for Accreditation/ Revalidation Learner’s ‘repository’ Institutional Data Portfolio for Appraisal Central data: Transcript MIS/ HR data Programme data: Granular assessment data Outcomes / skills sets PDP (shared) PDP / Reflective (private)

Portfolio Assessment Where to begin ? Key issues: Validity Baume D. A Briefing on the Assessment of Portfolios. LTSN Generic Centre. Assessment Series No.6 (2001) Friedman M, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical Teacher. 2001; 23:535-51 Knows Knows how Does Shows how Miller GE. Acad Med 1990:S63-7 Assessing the development of competencies over time Key issues: Validity Reliability Attitudes & behaviours Time demands

ePortfolios and Assessment of assessment (results / essays etc) for assessment -Potential benefits of ePortfolios: Formative input during portfolio building (not just at end) Random sampling & ‘drilling down’ Peer Assessment (formative and summative) Automatic inclusion of assessment results + feedback Automated summary information (quantitative) Reduced monitoring / admin demands -Piloting essential before use in ‘high-stakes’ assessment -Need for further research in this field !

Principle 4. One size does not fit all Institutional Subject / context requirements vs. specific (standardisation) requirements ePortfolios should be highly flexible / customisable support diversity (accessibility, learning styles etc)

The need for flexibility & customisation e.g. selecting tools by course / year groups Course Admin view Student view install from a set of ‘generic’ tools create context-specific tools via simple Web forms

The need for flexibility & customisation (2) e.g. configuring learning outcomes / skills sets Course Admin view Student view

Principle 5. ePortfolios should be learner-centric Personalisation / customisation Sense of ownership / privacy / sharing Constructivist theories. is eLearning contributing to ‘Paradigm shift’ ? Student Independent Learner Teacher Facilitator Didactic Interactive + Explorative 1 location Distance learning 1 provider Multiple providers

Emphasis on Learner Ownership / Control Students control which parts of their portfolio can be viewed for specific purposes or by specific people Mixed-Ownership Model Personal Data Corporate (HR) data This may ease the tension between using portfolios for formative and summative assessment Tutor’s Data

Ability for Learners to Share their Portfolio Content Users can control access to their portfolio

Adding ‘child’ objects to an artefact Flexibility for the Learner to Add, Link & Cross-Reference ePortfolio Artefacts Adding ‘child’ objects to an artefact Cross-reference An artefact with A skill

Principle 6. ePortfolios should be an integral part of the learning experience Educational (integration with the curriculum) Technical (integration within the MLE)

Educational integration Technical integration Should not be perceived as a ‘bolt-on’ Support a holistic view of curriculum / outcomes (as well as specific components) Technical integration (example from Medicine at Newcastle) ePortfolio integrated with the VLE Tool properties include ‘module code’ = dynamic links to Study Guides + resources Learning outcomes linked to curriculum databases: ‘modules’ and units Links to other systems eg. SSC selection 2 main areas of development in the first year of the project. Firstly a generic portfolio This is a stand-alone application We have developed a common portfolio framework -this allows sharing, sorting, cross-referencing It has been developed to be highly customisable: -components can be customised by course and year group -learning outcomes or skills sets can be customised by course -terminology can be customised eg. ‘tutor’ vs. ‘educational supervisor’ Secondly we have applied this technology to our undergraduate MBBS programme This version has been integrated into the virtual learning environment Developments are in place for piloting in 2003/4 across the curriculum: -phase 1 students have a reflective diary, CV and other tools -trialled in the stage 3 essential junior rotation in Child Health -all our stage students have to complete the portfolio for 1 of their 3 SSCs

Principle 7. ePortfolios should support life-long learning Help develop life-long learning skills Continuity in LLL (transferability)

Continuity in LLL (transferability) Towards a life-long learning record School HE Continuing Development Interoperability (technical standards to support the interaction and transfer of data between IT systems) IMS specifications /standards: Learner Information Package (LIP) Enterprise / Content Packaging ePortfolio (early stage of development) CETIS UK Learner Profile (based on IMS-LIP) ePortfolio interaction with VLEs / MLEs

The ePortfolio process should promote the develop life-long learning skills Preparation for professional requirements: Independent learning skills Personal development planning Time-efficient evidencing of performance / development for: - Appraisals - Assessment - Revalidation Aiming to promote a reflective / deep-learning approach ….research

Principle 8. Research / Evaluation is Essential Are the purpose(s) being fulfilled ?

Evaluation and Refinement ‘A good fit happens over time with lots of use’ Spandel, 1997 cited by Jan van Tartwijk (ALT-SURF Conference, 2004) Action Research approaches Incremental development and refinement Qualitative research methodologies (+some quantitative - not mutually exclusive!) Ethical and practical considerations

Research and Evaluation Does the ePortfolio improve learning ? How so ? Some evidence that PDP improves learning (more needed) A systematic map and synthesis review of the effectiveness of personal Development Planning for improving student learning. EPI Centre 2003 http://eppi.ioe.ac.uk Can ePortfolios foster a reflective approach to learning ? How defined ? How measured ? How do particular groups of learners approach reflective portfolios ? Learning Styles Gender and Age Cultural Background John Mole. The geography of thinking. Clin Med 2002; 2:343-5 Does the technology change the learning process ? What are learner and staff perceptions / attitudes towards ePortfolios ? There are potential drawbacks to electronic portfolios. Demographic factors People in different cultures are taught to think differently. How we gather information, process, rationalise, justify and communicate our ideas is culturally determined……”

Principles of Electronic Portfolios 1. Should bring significant advantages over paper- based alternatives 2. Should be considered in the context of wider (human) processes 3. Clarity of purpose(s) 4. One size does not fit all 5. Should be learner-centric 6. Should be an integral part of the learning experience 7. Should support life-long learning 8. Research / evaluation is essential ….the challenge is putting the principles into practice !

Further information: S.J.Cotterill@ncl.ac.uk Tony.McDonald@ncl.ac.uk http://www.eportfolios.ac.uk