1. seminar01102003/using CRA to promote student learning01102003.ppt Tackling the Challenges to Professional Education Project: Reforming Professional Education for a Knowledge-based Society Using Criterion-Referenced Assessment to promote student learning Dr. Charles C. Chan, APSS, HKPU

2. seminar01102003/using CRA to promote student learning01102003.ppt Assessing student learning outcomes Who: teachers, instructors, peers and even students themselves can administer the assessment of learning outcomes. For whom: students (useful feedback for further improvement: formative evaluation), instructors (formative and summative evaluation), and school or university administration (summative and quantitative evaluation). What: subject and content specific, and/or generic transferable skills; performance at practicum. How: subjective grading, GPA, CRA: taxonomy of educational objectives (cognitive) and behavioral competency.

3. seminar01102003/using CRA to promote student learning01102003.ppt Criterion-referenced assessment Judges performance against a set of pre-determined criteria; Is independent of any other student’s results; Increase students’ intrinsic motivation and their sense of being responsible for their learning outcomes; Encourages the cooperation between students; Improves the effectiveness of teachers’ feedback to students.

4. seminar01102003/using CRA to promote student learning01102003.ppt Comparison between NRA and CRA DimensionsNRACRA Assessment methodCompare with peersMark with pre-set criteria FeedbackSummativeFormative DistributionNormalSkewed BenefitsEasy to administer; Save time. Clear feedback; Provide guidelines for further improvements; Increase students’ intrinsic motivation. LimitationsDiscourage cooperation among students; Limits the effectiveness of feedback; Increase students’ anxiety. Time-consuming; Not widely used.

5. seminar01102003/using CRA to promote student learning01102003.ppt CRA: The SOLO taxonomy (Biggs & Collis, 1982) SOLO levelCharacteristics of responses Prestructural Irrelevant information is included or the task is not tackled incorrectly Unistructural One relevant information is included Multistructural More than one independent information are presented Relational Various relevant elements are integrated that an organized picture is presented Extended Abstract The original coherent picture is generalized to a new and more abstract domain, reflecting a higher mode of processing

6. seminar01102003/using CRA to promote student learning01102003.ppt CRA: Behavioral Competence Taxonomy ProfessionalismKnowledgePractice Competence Acquisition Show awareness towards major societal problems Recognize social welfare policies Observe and listen skillfully Assimilation Self-management; personal growth Understand social, cultural factors to specific client system Specify goals of interventions Adaptation Manage workload in an orderly manner Constructive use of available community resources Monitor and co- ordinate different work of the action plans Performance Accept advice and criticismApply theories learnt with flexibility Handle emergencies effectively Aspiration Incorporate new approach to practice Initiate to expand one’s knowledge base Recommend future improvements

7. seminar01102003/using CRA to promote student learning01102003.ppt Project I Subject: Social Psychology (level 3) [01-02] Participants: 35 undergraduate students (yr1-3) CRA used: the SOLO taxonomy Details: Students are required to write three short papers with subject-specific scenarios and questions provided (one baseline, one mid-term and one final). The SOLO taxonomy is used to analyze students’ assignments and students are informed of their SOLO levels as feedback of their learning outcomes after each assessment.

8. seminar01102003/using CRA to promote student learning01102003.ppt Project I – Assignment sample Short Paper I (500 words). You and one of your secondary school friends were successful to get into a degree program in the HK PolyU and the CityU / HKU respectively after one failure attempt. Right after both of you received the rejection letters, you met each other and found that the two of you had very different emotions. Q.1Right after that failure attempt, recall what you and your friend came up with separately to account for that rejection. Q.2Compare and contrast those reasons that were from you and from your friend and whether these reasons explained the different emotional states you were in. Q.3By using theories-in-use, demonstrate to your friend that you understand why s/he thought the way s/he did. Q.4Critically analyze why your friend agreed / disagreed with your perception of why s/he felt the way s/he did.

9. seminar01102003/using CRA to promote student learning01102003.ppt Project I – Record sheet of SOLO level Name of student Paper 1Paper 2Paper 3

10. seminar01102003/using CRA to promote student learning01102003.ppt Our Project I - Results Student learning outcome:  2.54 (SOLO 1); 3.00 (SOLO 2); 3.25 (SOLO 3)  Correlates with instructors’ judgment/subjective ratings.  Undergraduates generally produce “multistructural” level papers.  Students gradually improve during the semester.

11. seminar01102003/using CRA to promote student learning01102003.ppt Our Project I Instructors’ comments:

12. seminar01102003/using CRA to promote student learning01102003.ppt Our Project II Subject: Social Work Fieldwork I, II Participants: 103 Social work undergraduates (including both bachelor and diploma students); 40 fieldwork supervisors; CRA used: Behavioral Competence Taxonomy (self- developed, based on Hauenstein model and APSS fieldwork manual); Details: fieldwork supervisors rated students’ performance in practicum with BCT (baseline, mid- term and final).

13. seminar01102003/using CRA to promote student learning01102003.ppt Our Project II – Record Sheet Student: _________________________ Supervisor: _______________________ Period under supervision: ___________________________________ Name of the agency: _______________________________________ Nature of the setting: _______________________________________ Nature of the major pre-selected piece of professional practice*: ______________________________________________ _______________________ Please check the following box to identify the nature of the major practice that requires the student to perform during the time period under review (Choose either one):  1. Require execution of relatively tangible, routine and well-defined practice procedures  2. Require re-negotiation, re-definition and problem-solving skills in the execution of practice procedures Remark: __________________________ * may include planning and execution of a service program for adolescents, supportive counseling for pregnant teens etc. ProfKnowledgePC Level Illustration Please identify the level of behavioral competence: 1= acquisition; 2= assimilation; 3= adaptation; 4= performance; 5= aspiration

14. seminar01102003/using CRA to promote student learning01102003.ppt Our Project II - Results Students improve over the semester BaselineMid-termFinal Professionalism 1.792.793.42 Knowledge 1.392.553.16 Practice Competence 1.432.843.50

15. seminar01102003/using CRA to promote student learning01102003.ppt Our Project II - Results Baseline: “Student is trying to acquire the knowledge of service and the organization.” Mid-term : “Shows potentiality in integrating her experience and practice. Asks more sensible questions and more responsive in supervision.” Final : “Student tried to apply social work values and principles into practice e.g. confidentiality of cases. Student was getting more confident in working with different client group.”

16. seminar01102003/using CRA to promote student learning01102003.ppt Our Project II - Supervisors ’ comments “It takes quite some time to be familiar with the instruments.” “BCT offers more objective assessment than the traditional impression grading system.” “It’s good to see students improve gradually over the semester.” “Sometimes it is difficult to decide which level students attained.” “Some placement setting (the nature) may limit the level of attainment a student can achieve.”

17. seminar01102003/using CRA to promote student learning01102003.ppt Summary and Conclusion CRA benefits both students and teachers by providing valuable and objective feedback for continuous improvement in learning outcomes.

18. seminar01102003/using CRA to promote student learning01102003.ppt References Biggs, J. B. & Collis, K. F. (1982). Evaluating the quality of learning: the SOLO taxonomy (structure of the observed learning outcome). New York: Academic. Chalmers, D. & Fuller, R. (1996). Teaching for Learning at University: Theory and Practice. London, UK: Kogan Page Ltd. Chan, C.C., Tsui, M.S., Chan, M., & Hong, J. (2002). Applying the SOLO Taxonomy on studetns’ learning outcomesL an empirical study. Assessment & Evaluation in Higher Education, 27,6,511-528 Chan, C. C. & Cheng, G. H.-L. (2002). Using the SOLO Taxonomy to assess students’ written work. Submitted to Teaching of Psychology. Chan, C. C., Cheng, G. H.-L. and Chan, M. Y.-C. (2002, June). Assessing APSS Students’ Learning Outcomes and Learning Process. Teachers’ Manual version 2.1. (Available from the Network for Health and Welfare Studies, Department of Applied Social Sciences, the Hong Kong Polytechnic University)