1. Teaching Excellence Seminar School of Science University of Science and Technology The Design and Implementation of Outcome-Based Assessment in Science Teaching Excellence Seminar School of Science University of Science and Technology 22 March 2013 Isabella Wai-Yin Poon wypoon@cuhk.edu.hk Department of Statistics Associate Dean of Science Director of Centre for Learning Enhancement & Research The Chinese University of Hong Kong

2. 2 Content 1. Introduction 2. An OBA Assessment Model 3. Assessment as Learning Activities 4. Evaluation of Outcome at Programme Level 5. Recent Initiatives at Science of CUHK

3. 3 Introduction

4. 4 Outcome-based Approach (OBA) OBA  Actively promoted by UGC since 2005  Weave into the (2012) 4-year curriculum Many professors have no knowledge on this UGC  Not straightjacket  Institutions should take ownership  Inter-institutional Task Force established

5. 5 OBA: An International Trend? http://www.cwsei.ubc.ca http://www.cwsei.ubc.ca

6. 6 OBA Development @ CUHK General consensus at CUHK  Clearly articulated learning outcomes  Curriculum components align with learning outcomes Content Learning activities Assessment  Assessing outcomes and collecting feedback

7. 7 How We Proceeded? OBA has long been used  implicit rather than explicit manner These concepts are part of our culture  Although we are not eloquent in using the language 1 st step: articulate programme-level learning outcomes  Outcomes developed in 2006 A series of TDG projects (faculty- or department-levels) to implement initiatives that address issues of our concern  Front-line initiatives carry the spirit of OBA  Formulate in an OBA framework

8. 8 Science Teaching: Areas for improvement Interaction in class Variety of learning activities Cultivate the development of learning outcomes  Generic skills  Value/attitude Assessment  Align with OBA?  Summative vs Formative assessment Summative assessment: Provides evidence of student attainment Formative assessment: Assessment for learning  Criterion- vs norm-referencing Criterion: Pegged to pre-determined or expected learning otucomes Norm: Defined by relative performance Various TDG or other projects  Faculty, programme or individual level

9. 9 Faculty-level TDG Project 1 “Engaging Science Students in the Design and Engagement of Assessment”  October 2005 – November 2008  Website: http://www.cuhk.edu.hk/sci/TDG/http://www.cuhk.edu.hk/sci/TDG/

10. 10 Faculty-level TDG Project 2 “Promoting the Use of the Outcome-Based Approach to Science Programmes: Development of Learning Outcome Statements and Measurement Instruments”  October 2006 – November 2008  Website: http://www.cuhk.edu.hk/sci/OBA/http://www.cuhk.edu.hk/sci/OBA/

11. 11 Faculty-level TDG Project 3 “Using Clickers to Enhance Teaching and Learning: Pilot Case Studies in Science Programmes”  December 2008 – October 2009  Website: http://www.cuhk.edu.hk/sci/clickerhttp://www.cuhk.edu.hk/sci/clicker

12. 12 Faculty-level TDG Project 4 The Development of an OBA Assessment Model for Science Subjects”  August 2009 to Dec 2012  Website: will soon launch

13. 13 Faculty-level TDG Project 5 “Inter-faculty Academic-related Advising Service”  November 2009 –  Website: Restricted access (CUHK students only)  http://serverfarm.elearningpro.com/cuhk/softskillmentor http://serverfarm.elearningpro.com/cuhk/softskillmentor

14. 14 Faculty-level TDG Project 6 “Learning by Engaging and Teaching”  January 2013 – Website: http://www.cuhk.edu.hk/cpse/LET/http://www.cuhk.edu.hk/cpse/LET/

15. 15 An OBA Assessment Model

16. 16 Norm- vs criterion-referencing Heated debate  Science: Norm-referenced approach CUHK 2008 QAC Audit Report (p.23) “There is therefore an argument that criterion-referencing as opposed to norm referencing is more consistent with OBA. The Panel encourages CUHK to engage with the pressure placed by OBA on norm-based grading and consider the place of criterion based assessment and the measures which may be introduced, other than norm referencing, to track and correct grade inflation” Similar statements found in the QAC Audit reports for other UGC-funded institutions  HKBU, 2009, p.18;  HKU, 2009, p.22;  CITYU, 2010, p.21

17. 17 Two Components The Assessment Framework  How does a course, and in particular its assessment, support the development of programme LOs ? The Assessment Chart  Develop the grade descriptors  Use both norm referencing and criterion referencing

18. 18 Assessment Framework

19. 19 Assessment Chart

20. … Assessment Chart 20

21. 21 Characteristics of the Model User-friendly Align assessment with outcomes Scaffold the planning and design of assessment (components) Communicate clear expectation to students A probabilistic but not a deterministic model

22. 22 A probabilistic but not a deterministic model Age Individual Expected Height

23. 23 A Project: Assessment as a Learning Activity

24. 24 Objective Use assessment as a learning activity  More formative assessment Promote a variety of assessment activities  Don’t rely too much on examination Increase students’ interest in the learning activities Enhance fairness in assessment

25. 25 Two Strategies Strategy 1: Students participate in the design of assessment criteria Strategy 2: Students conduct self- and peer- assessments based on the set of criteria  Method introduced to ensure students to seriously and critically conduct self- and peer-assessments

26. 26 Strategy 1: Assessment Criteria Ask students to develop the set of criteria through a rigorous process  All criteria are proposed by students Too time consuming Provide students with a set of criteria for students’ deliberation and decision  Easier and more efficient

27. 27 … Strategy 1: Assessment Criteria Ownership of the assessment criteria  Students critically think about what constitute quality work  Students know in advance the set of criteria  Pay attention to various aspects Some aspects may otherwise be ignored  Writing skills in addition to content  Presentation skills o Use of IT, tables, graphs o Time control Enhance transparency and fairness on assessment activities

28. 28 Strategy 2: Self- and peer-assessment Self-assessment is graded based on whether students can assess themselves properly

29. 29 … Strategy 2: Self- and peer-assessments Improve fairness in peer- & self-assessments Students must pay attention to peer’s work in order to assess themselves properly  More critical  More serious  Learn from peers Issues to note  Benchmark the self- and peer-assessment  % of self-assessment contributes to total score  More discussion in a paper* *Poon, W.Y., McNaught, C., Lam P., & Kwan H.S. (2009). Improving assessment methods in university science education with negotiated self- and peer-assessment. Journal of Assessment in Education: Principles, Policy & Practice, 16:3, 331-346.

30. 30 Experiences in Science Used by many science teachers in  Group presentation  Individual presentation  Group project written report  Individual written assignment  Laboratory experiment report  Poster presentation  In-class activities in student-oriented courses Debate Role-play Portfolio of activities designed to explore a topic

31. 31 Evaluation Examples of students’ positive comments  “Quite interesting and innovative.”  “Innovative, can allow students to identify and think about what really matters.”  “This can force students to pay attention to the presentations of the others instead of daydreaming or chatting to each other.”  “It allows students to be more aware of their performance towards their peers, such that the presentation is more interesting and interactive.”  “It helps us to focus on the aspects other than the content itself, like the presentation skills and how the questions from audience are answered. Through this process, we could learn much more from each other.”

32. 32 …Evaluation Examples of students’ suggestions  “As an undergraduate student, it is a bit difficult for us to assess other’s work.”  “Students may not have enough knowledge to comment other’s group topic.”  “A weighting can be assigned to each marking criterion to better reflect the importance of each criterion.”  “There may be strong disagreement among group members in deciding the scores … I think the assessment will be much fairer if it is carried out individually.”

33. 33 Teachers’ Views In general well-received by teachers  Also Students  Easy to use  Challenges in classes of large size Applicable to a variety of activities Students paying more attention to peer’s works Can address the issue of free-riders in group project  Promote the use of group project as an effective assessment method as learning activity

34. 34 …Teachers’ Views Most effective for assessing generic skills  Some students still prefer teacher’s assessment Content knowledge or professional skills  Ask students to peer- and self-assess generic skills but not content or professional skills Can nurture the development of a wide range of (skill- and value-domain) outcomes  Self-managed learning, life-long learning, team-work, active learning, critical thinking, honesty, integrity…

35. Evaluation of Outcome at Programme Level Student Experience Questionnaire (SEQ) 35

36. Student Experience Questionnaire Administer to all undergraduate programmes @ CUHK  3-year cohort: 1 st & final year  4-year cohort: 2 nd & final year Two parts, 16 constructs, 33 questions Part 1: Students’ Capability Development  7 constructs, 14 questions Part 2: Teaching & Learning Environment  9 constructs, 19 questions All questions are on 5-point scale  from (1) strongly disagree to (5) strongly agree 36

37. … Student Experience Questionnaire The 7 constructs for “Capability Development”  Critical thinking  Creative thinking  Self-managed learning  Adaptability  Problem solving  Communication skills  Interpersonal skills and groupworks 37

38. … Student Experience Questionnaire The 9 constructs for “Teaching and Learning Environment”  Active learning  Teaching for understanding  Feedback to assist learning  Assessment  Relationship between teachers and students  Workload  Relationship with other students  Cooperative learning  Coherence of curriculum 38

39. Areas for Attention Capability  Communication skills  Interpersonal skills and groupwork Teaching & Learning Environment  Active learning My teaching staff use a variety of teaching methods We are given the chance to participate in class  Workload TDG projects developed to improve in these areas 39

40. … Science SEQ Results (2003 – 2012) Improvement observed  Communication skills  Active learning Many factors contribute to the improvement Many front-line teaching and learning initiatives 40

41. 41 Recent Initiatives at the Science Faculty at CUHK

42. 42 Project Information: Book & Website Published “Recent Initiatives in Teaching and Learning: Faculty of Science 2010”  Overview a number of T&L projects

43. 43 Website: http://www.cuhk.edu.hk/sci/ritlhttp://www.cuhk.edu.hk/sci/ritl

44. 44

45. 45 More details for each project

46. 46 Q & A Thank you very much.