1. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Can electronic portfolios facilitate personal development planning and meet the requirements for HE Progress Files and CPD ? On behalf of the consortium “Managed Environments for Portfolio-based Reflective Learning” (http://www.eportfolios.ac.uk). * University of Newcastle upon Tyne ^ University of Leeds & University of Sheffield + University of Dundee Simon Cotterill*, Tony McDonald*, Trudie Roberts ^, Chris Roberts &, Margery Davis +, Geoff Hammond* Newcastle University

2. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Overview 1.Background 2.Electronic portfolios 3.FDTL4 Project 4.Electronic Portfolio Approaches for HE Staff 5.Questions / Discussion !

3. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Background: PDP Increasing demands on HEIs to support personal development planning (PDP)Increasing demands on HEIs to support personal development planning (PDP) Recommendation 20 of the National Committee of Inquiry into Higher Education 1997 (Dearing Report)Recommendation 20 of the National Committee of Inquiry into Higher Education 1997 (Dearing Report) “… institutions of higher education, over the medium term, develop a Progress File. The File should consist of two elements: –a transcript recording student achievement which should follow a common format devised by institutions collectively through their representative bodies; –a means by which students can monitor, build and reflect upon their personal development.”

4. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 QAA recommendations on: QAA recommendations on: - transcripts - progress files PDP is primarily a formative process by which students can monitor, build and reflect upon their personal and academic development. PDP is primarily a formative process by which students can monitor, build and reflect upon their personal and academic development. Built on the model of the adult / independent learner Built on the model of the adult / independent learner Background: PDP

5. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Background: Internet Approaches to PDP Internet Personal and Academic Records (PARs), Newcastle/Nottingham - Two sequential projects funded by the DFES LUSID (Liverpool), RAPID (Loughborough), PESCA (Exeter) ProFile (Bangor). Oxford Brookes / Thames Valley Joint Progress File SHELL (Plymouth+)

6. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 1 st Internet-PARs project: DfEE Recording Achievement Theme: Internet-PARs Project (1998 - 2000) Partnership between Universities of Newcastle and Nottingham to:- … design, evaluate and disseminate a process, facilitated by C&IT, which would enable HEIs to construct PARs accessible on the Internet … to form the basis of a continuing record of lifelong reflective learning and personal development … demo via: http://www.internet-pars.ac.uk

7. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Academic Academic Support Support Progress Progress Review Review Development DevelopmentPlanning Tutor’s view:

8. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Newcastle-Nottingham Internet-PARs Project Phase 2: Making New Links (2000 - 2002) DfEE / DfES funded, HEFCE administeredDfEE / DfES funded, HEFCE administered … web-based systems to support continuing professional development (CPD) …… web-based systems to support continuing professional development (CPD) … –in Medicine (pre-registration house officers) –in Education (newly-qualified teachers) … dissemination of experience of implementing both paper- based and web-based PARs at the institutional level …… dissemination of experience of implementing both paper- based and web-based PARs at the institutional level … –a range of types of consultancy available to HEIs to facilitate planning and PDP implementation The 2 nd Internet-PARs project

9. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 MyPIMD Web portal Admin data (HR Database) Personal datacommunications role based communication Overview of the web portal in Medicineemployment records of past, present & future posts education record courses, leave, & budget development plan, evidence, & reflect on experiences

10. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 MyPIMD http://mypimd.ncl.ac.uk MyPIMD http://mypimd.ncl.ac.uk Access controlled by login (establishes role and level of access). Access controlled by login (establishes role and level of access). Contents customised (current post, grade, specialty and location). Contents customised (current post, grade, specialty and location).

11. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003education

12. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003developmentEncouraging recording & reflecting practice Facilitating mentoral processes Assessing learning needs and action planning

13. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Using admin (HR) and personal data to provide a chain of evidence for appraisals, assessments, revalidation… encouraging and facilitating on-going documentation and reflection on professional development

14. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003development Use of exemplification to encourage reflection

15. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 PORTFOLIOS What are portfolios ?What are portfolios ? FDTL4 project developing electronic portfoliosFDTL4 project developing electronic portfolios

16. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 What are Portfolios ? UnstructuredFormativeSample Best work Factual / Quantitative Student Owned StructuredSummative All work Representative Reflective / Interprative HEI Owned

17. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 What is an Electronic Portfolio ? 1.A computer application which stores the user’s information on a local drive. 2.A static Web page eg. CV on the Internet 3.A dynamic database-driven systems that is accessed over a network.

18. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Benefits of Electronic Portfolios Multi-purposeMulti-purpose OrganisedOrganised Multiple structures / viewsMultiple structures / views SearchableSearchable TransportableTransportable SharableSharable Not left on the bus !Not left on the bus !

19. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Potential Weaknesses of Electronic Portfolios There may be barriers to effective use of electronic portfolios: It may also be important to use portfolios as part of a wider strategy to develop the reflective analytical skills necessary for PDP. Poor access to computing / Internet facilitiesPoor access to computing / Internet facilities Technophobia !Technophobia ! Reliability of IT systemsReliability of IT systems Possible duplication / overlap with paper systemsPossible duplication / overlap with paper systems

20. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Potential Weaknesses of Reflective Portfolios It is unclear if reflective portfolios favour particular groups: Learning Style - “natural reflectors”? Gender and Age Cultural Background John Mole. The geography of thinking. Clin Med 2002; 2:343-5

21. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Formative and Summative Assessment Portfolios come in many forms and are often used to evidence the achievement of learning outcomes for summative assessment. There may be potential problems if the portfolio serves both formative and summative processes. For example reflection and PDP is less likely to be open and honest if the student knows that the work will be assessed. More fundamentally the production of a portfolio can itself be a learning process ie. it is as much a journey as an end-point for assessment.

22. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 The use of electronic portfolios may potentially reduce the tension between formative and summative processes by supporting both assessed and non-assessed areas. Students may also have the choice to make certain material available to assessors, appraisers, tutors and others involved in their education. Formative and Summative Assessment

23. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Students control which parts of their portfolio can be viewed for specific purposes or by specific people Data Ownership Model Corporate (HR) data Personal Data Tutor’s Data

24. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Managed Environments for Portfolio-based Reflective Learning Integrated Support for Evidencing Outcomes. The project consortium: Newcastle University (lead site) Leeds University Sheffield University Dundee University (for consultancy) 3 years funding through The HEFCE Fund for the Development of Teaching and Learning (FDTL). Developing Web based portfolios to support reflective approaches for evidencing the attainment of programme outcomes in undergraduate Medicine. Closely integrated with on-line curricula and study guides to become an integral component of managed learning environments for Medicine.

25. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 What is a managed learning environment ? TLTP3-86: Facilitated Network Learning in Medicine & Health Sciences (Networked Learning Environments) Newcastle University – Medicine Nottingham University – Medicine Durham University – Biomedical Sciences Northumbria University – subjects allied to medicine Sheffield University – Medicine Study GuidesLearning Resources Module Outline FormsSelf-assessment

26. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Aims of the FDTL-4 project To develop an on-line portfolio system to support a reflective approach to evidencing the attainment of programme outcomes.To develop an on-line portfolio system to support a reflective approach to evidencing the attainment of programme outcomes. To promote the development of the reflective capabilities of medical students, giving greater responsibility for managing their own learning and preparing for aspects of work-based and lifelong learning.To promote the development of the reflective capabilities of medical students, giving greater responsibility for managing their own learning and preparing for aspects of work-based and lifelong learning. To develop strategies to facilitate assessment of curriculum outcomes that are not amenable to traditional instruments of assessment.To develop strategies to facilitate assessment of curriculum outcomes that are not amenable to traditional instruments of assessment.

27. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Newcastle NLE curriculum map - outcomes quality process delivery - campus and at distance student record system registers off-line learning learning resources other collegesother agenciesbusiness systems self-assessment tutor/mentor support communication tracking Managed Learning Environment Managed Learning Environment (after JISC) Undergraduate portfolio and transcript for reflective learning - triggered by learning episodes related to outcomes

28. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 communicate studyexperientiallearningcycle Benchmarks  Outcomes DP Specs  Study Guides  Learning Resources learningenvironment experience / practice review / reflect experiment / apply think / conceptualise

29. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 communicate study evidenceexperientiallearningcycle Benchmarks  Outcomes DP Specs  Study Guides  Learning Resources personal & academicrecord learninglog evidencelearning

30. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 communicate study assess evidence studenttranscript experientiallearningcycle learninglog Benchmarks  Outcomes DP Specs  Study Guides  Learning Resources personal & academicrecord assessassess

31. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 communicate study reflect assess evidence studenttranscript learninglog personal & academicrecord experientiallearningcycle Benchmarks  Outcomes DP Specs  Study Guides  Learning Resources assess assess

32. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 The doctor as a professional The Right Person Doing It How the doctor approaches his/her practice Doing the Thing Right What the doctor is able to do Doing the Right Thing LearningOutcomes Who? How? What? When? Where?

33. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Recording against terminal learning outcomes:

34. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Defining intended learning outcomes for options:

35. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Life Long Learning / CPD Electronic portfolios can support life-long learning by drawing on personal and academic information prior to HE, such as the UCAS profiles.Electronic portfolios can support life-long learning by drawing on personal and academic information prior to HE, such as the UCAS profiles. The HE portfolios could later form the foundation for vocational or post-graduate portfolios, especially where portfolios are structured around learning outcomes which are geared towards key- skills and/or professional requirements.The HE portfolios could later form the foundation for vocational or post-graduate portfolios, especially where portfolios are structured around learning outcomes which are geared towards key- skills and/or professional requirements.

36. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Life-long Learning Approach in Medicine: portfolios reflection / PDP -integrated into an outcomes based curriculum continuing need to demonstrate CPD: annual RITA assessments annual RITA assessments annual consultants appraisal annual consultants appraisal 5 yearly revalidation 5 yearly revalidation Undergrad.Postgrad. 5yrs 10yrs Pre-HE The use of electronic portfolios aims to foster the skills of independent learning, reflection and PDP

37. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Portfolios for HE Staff Baume D, Yorke M. The reliability of assessment by portfolio on a course to develop and acredit teachers in higher education. Studies in higher education. 2002; 27: 7-25 HE Staff may also benefit from electronic portfolio approaches including personal profiles and research plans which tie in to management information systems and networked learning environments.

38. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 The Personal Plan at Newcastle Teaching and Learning (UG and PG teaching and teaching development)Teaching and Learning (UG and PG teaching and teaching development) Research (Research programmes, awards, staff and students)Research (Research programmes, awards, staff and students) Third Strand (spinouts, outreach work and industrial collaborations)Third Strand (spinouts, outreach work and industrial collaborations) Roles (administrative duties, training and support roles)Roles (administrative duties, training and support roles) Integrates centrally held data sets: RIS awards data under "Research Grants held"RIS awards data under "Research Grants held" Graduate School records of PG student supervision under "Research Students".Graduate School records of PG student supervision under "Research Students".

39. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Evaluation and Research Issues Portfolio assessment – validity and reliability issuesPortfolio assessment – validity and reliability issues Enhancing reliability in portfolio assessment: discussions between assessors. Pitts J, Coles C, Thomas P, Smith F. Med Teach 2002; 24(2):197-201 The use of portfolios for assessment of the competence and performance of doctors in practice. Wilkinson TJ, Challis M, Hobma SO, Newble DI, Parboosingh JT, Sibbald RG, Wakeford R. Med Educ 2002: 36(10):918-24 Portfolio-based assessments in medical education: are they valid and reliable for summative purposes? Roberts C, Newble DI, O'Rourke AJ. Med Educ. 2002: 36(10):899-900 [Commentary] AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment Friedman M, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ Medical Teacher. 2001; 23:535-51 Portfolio assessment in medical students' final examinations. Davis MH, Friedman M, Harden RM, Howie P, Ker J, McGhee C, Pippard MJ, Snadden D. Med Teach 2001; 23(4):357-366 An evaluation of a standards based portfolio. [Nursing / Outcomes Based] Gallagher P Nurse Educ Today 2001;21(5):409-16

40. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Reflective ability / practice Reflective ability / practice - agreeing a context sensitive definition of “reflection” - qualitative analysis of anonymised portfolio data based on this definition of reflection based on this definition of reflection - quantitative analysis of anonymised portfolio data - what factors influence reflective ability / practice ? - gender, age-group, ethnic groups etc. - any relationship with outcomes (grades) ? Evaluation and Research Issues

41. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Learning styles / learning approachesLearning styles / learning approaches - Honey & Momford used in study skills course - ASSIST questionnaire User evaluationUser evaluation - questionnaires + focus groups Student attitudesStudent attitudes - questionnaires + focus groups (longitudinal) - effect of implementation techniques ? Evaluation and Research Issues

42. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Impact of portfolios on the Education EnvironmentImpact of portfolios on the Education Environment Roff S. Development and validation of the Dundee Ready Education Environment Measure (DREEM). Medical Teacher, 1997; 19: 295-299 -Does the introduction of electronic portfolios change the educational environment ? Other issuesOther issues -Can computer mediated environments promote open reflection, sharing and sence of ownership ? -Does the use of exemplification benefit or hinder reflection ? Evaluation and Research Issues

43. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Conclusions Electronic portfolios have the potential to facilitate PDP and meet the requirements for HE Progress Files.Electronic portfolios have the potential to facilitate PDP and meet the requirements for HE Progress Files. –Offer flexibility for both formative and summative purposes. Electronic portfolios in HE may provide the foundation for portfolios in CPD.Electronic portfolios in HE may provide the foundation for portfolios in CPD. –Transfer of data 17/18 HE CPD –Common outcome themes –Particularly valid where the curriculum is driven by professional requirements

44. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 Conclusions (2) There remain many unanswered questions about the educational impact of electronic portfolios.There remain many unanswered questions about the educational impact of electronic portfolios. –Importance of evaluation and research Electronic portfolio approaches may also be applied to staff in HE for a range of purposes.Electronic portfolio approaches may also be applied to staff in HE for a range of purposes.

45. © 2003, School of Medical Education Development, University of Newcastle upon Tyne Electronic Portfolios Joint SEDA/SRHE Conference 2003 further information http://www.eportfolios.ac.uk http://www.internet-pars.ac.uk http://nle.ncl.ac.uk http://www.ltsn.ac.uk/genericcentre http://www.recordingachievement.org