1. Assessing Learning Outcomes:
Thinking Critically about Critical Thinking and Written Communication Skills
Vera Beletzan, Paula Gouveia, and
Kim Robinson Humber College
Welcome and introductions - Vera

2. Agenda Overview of Research Study
Experience the Critical Thinking Assessment Tool
Discussion and Questions
VERA

3. Humber: A Canadian Polytechnic
Vera
Who we are:
Location = Toronto
Size = Size = 30, 000 FT / 56,000 PT Ss
Institution = One of 12 Polytechnics in Canada – define
Credentials = from apprenticeship to postgrad
Scaling up in this context…

4. VERA

5. Phase 1 (2013-14): Humber & HEQCO-LOAC Objectives:
Develop LO assessment tool
Pilot tool to measure and assess students’ critical thinking and written communication (CTWC) skills in their written course work
Assess tool for usability
Revise tool in light of faculty feedback
Vera

6. Phase 2 (2014-16): Cross-sectional and Longitudinal Analyses
What is the validity of the assessment tool?
What is its reliability when used repeatedly?
Where do students experience greater levels of skill achievement in CTWC?
4) Are students making gains in CTWC learning outcomes over time?
5) What is the usability and scalability of the revised tool?
Vera

7. Cross-Sectional Analysis
Three CTWC learning experiences targeted for comparison of skills achievement:
- EXPLICIT = cross-college reading & writing courses (WRIT)
- IMPLICIT = vocational - Police Foundations Program (PFP)
- CONTENT-BASED = General Arts and Sciences (GAS) critical thinking course (GCRT)   
Paula

8. Participants: Cross Sectional-Analysis
650 students from 3 of Humber’s schools:
Business (BUS) (19%)
Liberal Arts and Sciences (LAS) (21%)
Social and Community Services (SCS) (57%)
Paula – 650 students represented appr 3% of Humber’s f/t postsecondary student population in fall 2014.

9. 1) What is the validity of the assessment tool?
Content and Face Validity: Substantial
Concurrent & Convergent Validity: CTWC scores correlated with a variety of assessments (TER for concurrent / final exam and course grades for convergent)
= Significant, positive and large / moderate correlation
Paula

10. 2) What is the reliability of the tool?
Inter-rater Reliability (IRR):
Consistency of faculty ratings of total CTWC skills scores = excellent
However, IRR for individual CTWC categories ranged from excellent/good to fair/poor
Internal Consistency: Excellent to good
Paula
Fair / poor on comprehension categories

11. 3) Where do students experience greater levels of skill achievement in CTWC?
Paula
WRIT 100 = SEM 1 / WRIT 200 = SEM 2 / PFP = SEM 3
Students in 1st and 2nd semester WRIT courses, where CT skills explicitly taught, demonstrated the highest achievement
Students in 3rd semester PFP courses, where CT skills more implicitly embedded, achieved moderate to high success, but lower than in the 2nd semester WRIT course
Students in 2nd semester GCRIT course, where CT taught as content knowledge, had lowest achievement of all

12. Longitudinal Analysis:
4) Are students making gains in learning the CTWC skills over time?
CTWC skills assessed in mandatory, sequenced cross-college WRIT courses over two semesters to determine students’ gains over time
Participants: 32 students from BUS (34%) and SCS (63%)
Paula

13. Paula Significant gains between WRIT 100 and WRIT 200 not shown
However, limitations to the longitudinal analysis include:
small sample size (32 students)
2 points in time (i.e. only 2 pieces of written work were assessed over the 2 courses/semesters - YEAR 1 - and not over the entire span of students’ programs)

14. 5) What is the usability and scalability of the tool?
46 faculty members piloted the tool between 2014 and 2016: LAS (76%), BUS (14%) and SCS (10%)
74% of faculty who used the tool agreed it was easy to use
58% of faculty agreed it provides meaningful information to students on their CTWC skill level and how to improve it over time
Paula

15. Recommendations I Teach CT explicitly and as a skill
Teach CT consistently and build over time
Position courses where CT taught explicitly (e.g. WRIT) strategically throughout a program for maximum exposure and ample time to practice
Vera

16. Recommendations II
Cross-institution scale-up is possible, but the tool is most effective in courses where CTWC skills are taught explicitly (e.g. WRIT)
Significant faculty training is needed to establish a common understanding of essential skills, their importance and their assessment
Training will improve overall reliability and hence accuracy and validity
Vera

17. Recommendations III
Finally, to ensure the tool is meaningful, students must understand which skills are being assessed and why
For significant gains, students’ skills must be evaluated throughout their programs, with ample opportunity for practice
Vera

18. Experience the Assessment Tool
STEPS:
Read the assignment instructions.
Read the student response (paper).
Assess the response using the CTWC Assessment Scorecard.
Discuss your experience in groups.
Kim

19. Any Final Questions / Comments?
Kim, Paula, and Vera

20. Thank you. vera.beletzan@humber.ca paula.gouveia@humber.ca
Sign Off Slide