1. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Evaluating Portfolios: Mirrors Maps and Sonnets Simon Cotterill Sue Gill Jamie Thompson EPICS ePortfolios project & PDP Forum http://www.epics.ac.uk

2. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Simon meets Jamie ePortfolio meets PDP Debate widens EPICS and the NE Regional Forum Philosophy, Practice and Evaluation Symposium The story so far

3. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets EPICS - North East regional collaboration around e-portfolio progression pathways with illustrative studies Part of the JISC Distributed e-Learning Programme. Partners include: 5 HEIs FE colleges (OWL network) JISC Regional Centre Case studies -demonstrate feasibility / scalability -supporting life-long learning http://www.epics.ac.uk

4. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets to improve (iterative) What do we mean by “Evaluation” & “Research”? Evaluating which ePortfolio product to use -checklists / criteria eg. Richardson & Ward, 2005 Formative Evaluation (independent or practitioner) Action Research (practitioner) Summative Evaluation – making a judgment; did the intervention achieve its stated goal(s) Research (= rigorous scholarly evaluation?) –tends to be more grounded in theory –narrow, well defined research question –prospective research plan –aims to be reproducible (difficult in complex educational environments) } Before we begin…

5. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Clarity of Purpose Back to basics: Evaluation should address the main aims / purpose(s) of your portfolio / PDP –What is the purpose? –How will you know if it has been achieved? i.e. measure it? (quantitatively and/or qualitatively) Use this to inform your evaluation strategy Are there existing (validated) research instruments that will address your questions? e.g. ELLI (life-long learning) e.g. ASSIST (approaches to learning) e.g. Bloom’s taxonomy (reflective learning)

6. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Diez, M. (1994) The Portfolio: Sonnet, Mirror and Map in Burke, K. (1996) Professional Portfolios. Arlington Heights: IRI Skylight

7. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets See also Story, Journey, Laboratory (P and L Paulson) Positivist v Constructivist (Wilkerson and Lang) My digital clone, work companion, butler, dashboard, planner, IPR Management assistant (Serge Ravet) Toothbrush, Caterpillar, Kaleidoscope, Window

8. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Paulsons (1994) Positivist Portfolios “The purpose of the portfolio is to assess learning outcomes and those outcomes are, generally, defined externally. Positivism assumes that meaning is constant across users, contexts, and purposes… The portfolio is a receptacle for examples of student work used to infer what and how much learning has occurred.” (p.36) Constructivist Portfolios “The portfolio is a learning environment in which the learner constructs meaning. It assumes that meaning varies across individuals, over time, and with purpose. The portfolio presents process, a record of the processes associated with learning itself; a summation of individual portfolios would be too complex for normative description.” (p.36) Tension between two approaches “The two paradigms produce portfolio activities that are entirely different... “The positivist approach puts a premium on the selection of items that reflect outside standards and interests.... “The constructivist approach puts a premium on the selection of items that reflect learning from the student’s perspective.” (p.36)

9. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Iterative PDP / ePortfolio Map Mirror Sonnet –Where am I going and how will I get there? –Reflecting … beyond surface learning … making meaning. –Presenting my truth. Who I am and how I know.

10. EmployerLearnerTeacherHE? Mirror Map Sonnet Stakeholder Mode

11. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets EmployerLearnerTeacher Mirror Map Example 1. Does planning and evidencing learning outcomes and improve learning in SSMs for 4 th year medical students? Sonnet

12. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Cotterill SJ, Aiton J, Bradley PM, Hammond GR, McDonald AM, Struthers J, Whiten S. A flexible component-based ePortfolio: adapting and embedding in the curriculum. In: In Jafari A, Kaufman C, ed. Handbook of Research on ePortfolios. Pennsylvania: Idea Group Inc, 2006 Evaluating a ‘MAP’ type portfolio Example: ePortfolio for year 4 Medicine, Newcastle University Mapping/Planning Recording/Evidencing Reflection

13. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Q. Building the ePortfolio was a useful learning experience 80% thought it was a useful learning experience 83% felt they had recorded good evidence 93% reflected on their learning after the [placement] Q. Having clearly defined intended learning outcomes influenced the way in which I approached the [placement] 72% felt that the LOs influenced their approach n = 157 Fixed-response questions (eg Likert scale) n=157 (84% response rate)

14. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets “ It encouraged me to really give thought to what I wanted to achieve during the [placement], which was especially useful as this was my first [placement]. As a result of the portfolio I think I got much more out of the [placement] than I would have otherwise. ” “ It made me concentrate on creating aims at the start of the [placement] and allowed me to plan the [placement] with my supervisor in a defined way. Overall it made my learning for the [placement] more organised and focused. ” …but simple open-ended questions can be very illuminating: “Most of the things I learned couldn't be 'measured/quantified' so I felt it was of little value to try and invent a way in which they could.” eg. what did you most like [dis]like about using the portfolio?

15. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Measuring Development ELLI Achievement v Personal development Characteristics of learning power Measurable change ELLI and HE 2006 – 2008 Situated research Range of contexts Interested in participating?

16. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Employ er LearnerTeacherHE? Mirror Map Sonnet Stakeholder Mode

17. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Group Discussion Critique of the model In terms of your own experience and use of PDP/ePortfolio; what do you think of the MMS metaphor? Which other perspectives are important? Developing the model Identify a type of portfolio and a perspective relevant to your practice or experience; share and discuss relevant research or evaluation questions and methodology

18. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets The problem of evaluation Evaluation of achievement not appropriate (positivist assumptions) Causative connections between PDP/ePortfolio difficult to establish Notion of situational evaluation

19. Evaluating PDP & ePortfolios: Mirrors Maps & Sonnets Thank you for your participation! Contacts: Jamie.Thompson@northumbria.ac.uk S.J.Cotterill@newcastle.ac.uk EPICS ePortfolios project & PDP Forum http://www.epics.ac.uk