1. Victoria Crisp & Christine Ward The PePCAA Project: Formative Scenario-based CAA in Psychology for Teachers

2. Crisp & Ward: The PePCAA project Topics The PePCAA project and its aims Scenarios: development, types and CAA features Examples of scenarios Results of trialling Problems in sharing scenarios transnationally Achievements.

3. Crisp & Ward: The PePCAA project PePCAA – Pedagogical Psychology Computer Assisted Assessment FIM-NeuesLernen, University of Erlangen LabSET, University of Liege Universitat Oberta de Cataluña (OU of Catalonia) PGU, Göteborg University UCLES - University of Cambridge Local Examinations Syndicate

4. Crisp & Ward: The PePCAA project PePCAA Scenario-based computer-delivered formative assessments in pedagogical psychology for teachers and trainee teachers. Main aims: to improve the quality of teaching, by linking psychological theory to practical instances of its use in the classroom to explore more innovative techniques of computer assisted assessment

5. Crisp & Ward: The PePCAA project Expected benefits of the approach (1): Intrinsic interest of scenarios Showing relevance of psychological theory Increasing trainee awareness of psychological theory and research Developing awareness of why some strategies are more effective Stimulating discussion.

6. Crisp & Ward: The PePCAA project Expected benefits of the approach (2): Allowing trainees to assess own knowledge Motivational benefits of CAA Opportunity to experiment with CAA.

7. Crisp & Ward: The PePCAA project Scenario development Early discussion and drafting of examples Drafting by one partner Comment by at least one other partner Revision and trialling Development of PLAC (PePCAA Learning Assessment Circle).

8. Crisp & Ward: The PePCAA project Scenario types: Problem Solving: diagnose causes of a classroom difficulty, suggest possible solutions and their theoretical basis, predict effects of proposed intervention. Best Practice: plan the best strategy to achieve a goal, justify the strategy by reference to theory, predict the outcomes.

9. Crisp & Ward: The PePCAA project

10. Metacognition and confidence ratings Promote awareness of one’s own knowledge Confidence rating for each automatically scored answer (in some assessments) Calculation of Confidence, Prudence and Nuance Required a work-around in Perception.

11. Crisp & Ward: The PePCAA project Some examples of scenario assessments http://www.pepcaa.odl.org/

12. Crisp & Ward: The PePCAA project

17. Scenario / software features (1): a scenario (or ‘case study’) description option to read a short explanation of relevant psychological ideas first ‘essay’ question, followed by feedback stating the main points expected ‘essay’ question followed by a multiple- response question asking which points included, with feedback on choices made

18. Crisp & Ward: The PePCAA project Scenario / software features (2): range of objective question types division of some assessments into blocks ‘branching’ dependent upon the candidate’s response to a question confidence measurement explanatory feedback, with weblinks

19. Crisp & Ward: The PePCAA project Trialling in the UK Small groups of trainee teachers Scenarios from both UK and non-UK partners Feedback collected through both open and closed questions Some follow up in interviews.

20. Crisp & Ward: The PePCAA project Trialling – format of feedback Statement (eg ‘The assessment is easy to understand’) to be rated as follows: +2 strongly agree +1 agree 0 neither agree or disagree -1 disagree -2 strongly disagree

21. Crisp & Ward: The PePCAA project Scenarios trialled in the UK - primary 2: PE lesson with 10 year olds (6 students) 3: Number facts (6) 4: Assessing Science (6) 6: Problem Solving (7) 7: Attachment Theory and Parenting (6) 8: Friendships (6) 9: Play (6) PGU 7 (Sweden): Teaching conceptual change.(6)

22. Crisp & Ward: The PePCAA project Trialling of primary scenarios in UK – Average ratings (scale – 2.0 to + 2.0) Knowledge of pedagogical psychology is very important to me 0.64 My current knowledge of pedagogical psychology is very good - 0.39 Assessment is easy to understand 0.88 Assessment is about a real world situation 1.22

23. Crisp & Ward: The PePCAA project Trialling of primary scenarios in UK – Average ratings (scale – 2.0 to + 2.0) Assessment is relevant for teachers 1.36 Assessment is well structured 0.88 Assessment is too long - 0.33 Assessment is stimulating 0.87 Working on assessment was helpful 0.95 Feedback given was helpful 0.97

24. Crisp & Ward: The PePCAA project Trialling of primary scenarios in UK – Key findings Generally favourable responses to all scenarios Ratings for Swedish scenario similar to those for UK One scenario (UCLES 7) shown to be too long Amendments made in the light of responses

25. Crisp & Ward: The PePCAA project Scenarios trialled in the UK - secondary 1: History Lesson (7 students) 5: Dance lesson with 11 year olds (7) Labset 2 (Belgium): Why success isn’t explained like failure (7) FIM 8 (Germany): Self-managed learning (7)

26. Crisp & Ward: The PePCAA project Trialling of secondary scenarios in UK – Average ratings (scale – 2.0 to + 2.0) Knowledge of pedagogical psychology is very important to me 0.46 My current knowledge of pedagogical psychology is very good - 0.46 Assessment is easy to understand 0.04 Assessment is about a real world situation 0.54

27. Crisp & Ward: The PePCAA project Trialling of secondary scenarios in UK – Average ratings (scale – 2.0 to + 2.0) Assessment is relevant for teachers 0.54 Assessment is well structured 0.14 Assessment is too long - 0.43 Assessment is stimulating - 0.04 Working on assessment was helpful 0.04 Feedback given was helpful - 0.21

28. Crisp & Ward: The PePCAA project Trialling of secondary scenarios in UK: Key findings Average ratings much lower than for primary scenarios Ratings for Belgian and German scenarios much lower than those for UK Amendments made to UK scenarios in the light of responses.

29. Crisp & Ward: The PePCAA project Trialling of scenarios in UK: Some general findings Mixed response to open-response questions not automatically marked Mixed response to multiple-response questions Improvements required to candidate information and end of test feedback Some students found the psychological language difficult.

30. Crisp & Ward: The PePCAA project Problems in transnational sharing of scenarios Mismatch in the coverage of psychology Depth of treatment and detail differed, even where topics were common More practical approach in the UK & Sweden Assumptions about the background to the scenario might not fit another country Careful editing of language needed.

31. Crisp & Ward: The PePCAA project Achievements of the PePCAA project A framework for scenario development, plus actual examples PePCAA Learning Assessment Circle (PLAC) Showed benefits of scenarios in motivating students and prompting reflection Demonstrated taking CAA beyond simple multiple-choice. Potential applications outside pedagogical psychology

32. Crisp & Ward: The PePCAA project Try the assessments for yourself on http://www.pepcaa.odl.org/