A New Approach to Performing Course Evaluations: Using Q Methodology to Better Understand Student Attitudes Joe Jurczyk Susan Ramlo University of Akron Paper Presented at 20 th Annual Q Conference

Student Course Evaluations Conducted to measure the effectiveness and quality of: teaching course design other learning resources that are implemented in the classroom.

Student Course Evaluations By 1993 nearly 90% of all college institutions required student course evaluations (Seldin, 1993). “When student evaluations of faculty are used summatively to determine retention, promotion, and merit pay, there is the potential for serious consequences in the classroom.” (Adams, 1997)

Traditional Surveys Likert scale Open-ended questions Analyzed with summative data (e.g. average satisfaction with textbook: 4.4)

Purpose of Study Use Q to gain insight into student point of view. Understand groupings that exist in classroom Exploratory research to determine method feasibility, lessons learned, obstacles Research conducted at a two-year college in large, midwestern university

Methodology: Q-sort materials 40 individual pieces of paper with statements regarding: –Overall course structure –Lecture quality –Lab quality –Lecture instructor –Lab instructor quality Packet with instructions, questionnaire, & fill-in grid Grid for placing / arranging statements during the sorting process Voluntary & anonymous

Sample Statements 1. I had adequate time to complete lab exercises. 2. My instructor used teaching methods well suited to the course. 3. My instructor organized this course well. 4. My lab instructor was available during office hours. 5. Course assignments were interesting and stimulating. 6. Course assignments helped in learning the subject matter. 7. My lab instructor provided sufficient help in the lab. 8. My instructor was well prepared for class meetings. 9. The objectives for the lab activities were well defined. 10. I kept up with the studying and work for this course. 11. Lab facilities were adequate. 12. I actively participated in class activities and discussions. 13. My lab instructor was prepared for lab lectures and discussions. 14. I was interested in the content of this course before taking it. 15. My instructor adapted to student abilities, needs, and interests. Statements obtained from Indiana University Evaluation Manual:

The courses evaluated 4 courses evaluated using Q methodology during lecture during the last week(s) of the course. Spring (1) & 8 week-summer session (3) 2004 All science courses within the same department within a Community & Technical college –Each consisted of lecture & laboratory –2 Basic Chemistry (summer) –1 Technical Physics: Mechanics II (spring) –1 Technical Physics: Heat & Light (summer) 4 different lecture instructors

Basic Chemistry – Summer 04 Service course for various associate degree programs (criminal justice technology, fire protection, allied health) & pre-nursing students Summer sections were primarily female (gender not requested on questionnaire) Focuses on concepts of inorganic chemistry (minimum math) 2 different instructors –Tenure-track teaching M & W, day time, 8 week session; taught both lab & lecture. –Part-time instructor, Saturdays, 8 week session; taught both lab & lecture

Technical Physics 4 half-semester offerings (8 week sessions) Required, non-calculus physics courses for Engineering Technology programs Only 1 female student in each of the 2 courses evaluated (gender not requested on questionnaire) Technical Physics: Mechanics II –Spring 2004, weeks 9 through 16 –Tenured professor taught lecture –Part-time instructor taught laboratory Technical Physics: Heat & Light –Summer 2004, 8 week session –Same part-time instructor taught both lab & lecture

Observations of the Q sort process Spring-semester 2004 – Mechanics II Per student comments - Students were frustrated because they had completed the required college, Likert- scale evaluation during the prior course meeting (redundancy). Per observations - Students were frustrated because of the lack of space available during the sort (tablet-desks). Students made negative comments regarding the amount of time required to perform the sort. Frustration led some students to stop the Q sort process &, therefore, negatively affected participation.

Observations of the Q sort process Summer 2004 – 3 evaluations Students did not evaluate summer courses with the college, Likert- scale instrument – may have lessened frustration. Students sat at or had available long tables – sufficient space to distribute the Q sort materials – seemed to lessen frustration. Nearly all students completed the survey in less than 20 minutes (less time than the spring Q sort). Some students had trouble distributing the 40 statements into 3 preliminary groups (e.g. liked instructor/course so all “agree”). Problems with preliminary sort seemed to prolong the Q sort process (last to hand in their sorts &, possibly, may have been among those who did not complete the sorting process).

Unsolicited student comments Complaints that the Q sort took longer to complete than the college’s Likert- instrument Statements that concerning difficulty discerning a difference between certain statements… –1 student gave the following example: 34. My instructor makes difficult material easily understandable 28. My instructor explains the material clearly.

Technical Physics - Evaluation 1 Factor Loadings Evaluations X X X X X X X % expl.Var

Technical Physics - Evaluation 1 No. Statement No. Z-SCORES Factor 1: Self-confident 17 I feel that I performed up to my potential in this course The total amount of material covered in the course was reaso I kept up with the studying and work for this course Course assignments helped in learning the subject matter Course assignments were interesting and stimulating Overall, I would rate the textbook/readings as excellent Factor 2: Negative self 21 I knew what was expected of me in this course My instructor adapted to student abilities, needs, and inter I had adequate time to complete lab exercises I kept up with the studying and work for this course I feel that I performed up to my potential in this course Lab facilities were adequate Factor 3: Mixed feelings about lab & content 30 My lab instructor clearly explained the procedures to be use My lab instructor promptly returned reports and assignments I learned a lot in this course I was interested in the content of this course before taking The labs were important to learning in this course My lab instructor was prepared for lab lectures and discussi

Basic Chemistry(weekdays)– Evaluation 2 Factor Loadings Evaluations X X X X X X X X % expl.Var

Basic Chemistry(weekdays)–Evaluation 2 No. Statement No. Z-SCORES Factor 1: Positive view of lecture instructor 34 My instructor made difficult material easily understandable My instructor was well prepared for class meetings My instructor answered questions carefully and completely Course assignments were interesting and stimulating I was interested in the content of this course before taking Lab assignments were interesting and stimulating Factor 2: Positive view of lab instructor 7 My lab instructor provided sufficient help in the lab I had adequate time to complete lab exercises My lab instructor was prepared for lab lectures and discussi The total amount of material covered in the course was reaso I was interested in the content of this course before taking The course improved my understanding of concepts in this fie Factor 3: Well prepared for the course but it was not rigorous enough 14 I was interested in the content of this course before taking I kept up with the studying and work for this course The labs were important to learning in this course I feel that I performed up to my potential in this course I learned a lot in this course The total amount of material covered in the course was reaso

Basic Chemistry(Saturdays)–Evaluation 3 Factor Loadings Evaluations X X X X X X X X X X X % expl.Var

Basic Chemistry(Saturdays) – Evaluation 3 No. Statement No. Z-SCORES Factor 1: Positive self / negative instructor 17 I feel that I performde up to my potential in this course I kept up with the studying and work for this course I actively participated in class activities and discussions This course increased my interest in the subject matter My instructor explained the material clearly My instructor answered questions carefully and completely Factor 2: Hands-on/ negative instructor 11 Lab facilities were adequate I actively participated in class activities and discussions I had adequate time to complete lab exercises My instructor answered questions carefully and completely My instructor made difficult material easily understandable My instructor explained the material clearly Factor 3: Positive instructor / negative course structure 2 My instructor used teaching methods well suited to the cours My instructor showed genuine interest in students My instructor was well prepared for class meetings Lab assignments were interesting and stimulating Overall, I would rate the textbook/readings as excellent Course assignments were interesting and stimulating

Summer Heat and Light – Evaluation 4 Factor Loadings Evaluations X X X X X X X X X X X % expl.Var

Summer Heat and Light – Evaluation 4 No. Statement No. Z-SCORES Factor 1 Positive instructor / negative materials 29 My lab instructor promptly returned reports and assignments My instructor made difficult material easily understandable My instructor used teaching methods well suited to the cours Overall, I would rate the textbook/readings as excellent Lab assignments were interesting and stimulating Lab facilities were adequate Factor 2 – Positive self – negative lab & materials 31 I learned a lot in this course I feel that I performed up to my potential in this course This course increased my interest in the subject matter Lab sessions were well organized Overall, I would rate the textbook/readings as excellent My lab instructor related lab exercises to lectures and read Factor 3 - Good overall instruction / negative self 8 My instructor was well prepared for class meetings I knew what was expected of me in this course My lab instructor related lab exercises to lectures and read I had adequate time to complete lab exercises I feel that I performed up to my potential in this course I kept up with the studying and work for this course

Recommendations Single facilitator – provides consistent instructions to students and ability to compare observations of multiple groups (classes) Findings improve with larger classes – more stable factors Future research: multiple evaluations - study changes over time: between individuals and between groups

Questions ? Joe Jurczyk – Sue Ramlo – Paper and Presentation available at: