1. UW-ACE Survey Results: Students’ Experiences of Using Technology in Learning Vivian Schoner and Katrina Strampel Centre for Learning and Teaching through Technology (LT3) University of Waterloo

2. The Setting: Teaching, Technology, and UW-ACE  Student Encounters with the Blended Learning Experience  Technically Viable? Yes  Pedagogically Sound? Yes, No, Maybe, Depends…  Survey Research Results

3. The UW-ACE Survey  We asked students to tell us about their experiences with Online Learning Activities Online Learning Activities Online Collaborative Learning (small group) Online Collaborative Learning (small group) Technology Function Technology Function Student Satisfaction Student Satisfaction Online Instructor Feedback Online Instructor Feedback

4. Students Completing Survey Faculty Quest Enrolment % Survey AHS15256 Arts30416 ENG 7395 ENV14147 MATH31704 SCIENCE94744

5. Comparisons of Scales

6. Technology Function: Access; Ease of Use; Navigation 26. logging in to be… 27. accessing my courses to be… 28. navigating around the UW-ACE course pages to be… 29. accessing the course tools/announcements to be… 30. posting, reading, responding to messages in the online discussion forum to be… 31. accessing feedback on submissions to be… 32. accessing my grades to be… 33. submitting assignments to be… 34. accessing quizzes to be… 35. accessing team or small group resources to be…

7. Scale Results Technology (alpha.953) Faculty% Very Difficult % Difficult % Undecided % Easy % Very Easy Cum. Positive% SCIENCE248325385 FES1412353883 ARTS359325183 AHS1512265682 MATH6614344874 ENG61116323567 Av for % Combined Neg 9 UnD 12 Pos 79

8. Technology: Accessibility and Ease of Use  “quick access to grades and assignment feedback, access to past lecture notes, announcements, additional resources”  “UW-ACE helped a lot in centralizing my courses online”  “the fact that you can connect anytime/anywhere”  “a safe online environment”  “easy to use, helpful, good interactions with profs and peers”  “I could sign in and have everything at my fingertips”

9. Appreciation for Technical Potential  “it enhances the learning process by offering supplemental resources for course related topics”  “it is certainly appropriate to post lecture notes and other resources, but the absolute most important thing to remember is not to invent uses just because the UW-ACE software is there. Legitimate uses are great, but online activities just for the sake of doing them are frustrating”  “it is desirable because it gives students an alternative way to find information about the course, also allows them to communicate with others who need help and who can offer help”  “if you have a question, but hate raising your hand and asking in front of all your peers, it can be asked and answered online”  “I believe it’s an integral part of the university to have students using technology which will be useful once we are out there in the workforce. Developing this skill is essential for us to be able to learn to be technologically diverse”

10. Overall Student Satisfaction: Course Organization; Instructions; Workload; Technical Support 36. the organization of the course content and activities? 37. the instructions for completing learning activities? 38. the number of course learning activities through the term? 39. the distribution of course learning activities through the term? 40. the technical support? 41. the professional look of the course web page(s)

11. Scale Results Overall Student Satisfaction (alpha.916) FacultyVery Unsatisfied UnsatisfiedUndecidedSatisfiedVery Satisfied Cum Positive % AHS2417383977 SCIENCE3416423577 ARTS5714443074 FES361743074 ENG3822353267 MATH59244271562 Av for % Combined Neg 10 UnD 18 Pos 72

12. Online Instructor Feedback: Helpful; Informative; Appropriate; Timely 42. instructor feedback helped me learn and understand difficult concepts 43. the frequency of instructor feedback met my information needs 44. the quality of instructor feedback was helpful and informative 45. the timeliness of instructor feedback was appropriate for my needs 46. the amount of instructor feedback was appropriate for my needs

13. Scale Results Online Instructor Feedback (alpha.972) FacultyStrongly Disagree DisagreeUndecidedAgreeStrongly Agree Cum. Positive % ARTS5617393372 AHS4722363167 SCIENCE51021263864 FES13824312455 ENG21826411354 MATH23925261743 Av for % Combined Neg 18 UnD 23Pos 59

14. Online Instructor Feedback  “I got assignments back sooner and easier with the site”  “the professor always kept us informed of news by posting announcements frequently on the UW-ACE website”  “the immediate feedback from quizzes written as well as the correct answers is a definite asset”  “took too long for marks to be posted”  “not being able to see answers on actual quizzes”  “grades are not properly recorded”  “there was an announcement that marks for an important section would be posted which they never were”

15. Online Learning Activities: Learning Support Features, Concepts, Skills, Applications; Integrated Parts; Study Time; Performance Level 1. Discussion forum activities 2. Self-Assessment (un-graded) 3. Self-assessment (graded) 4. Collaborative/Peer Review submissions 5. Individual Submissions 6. Lecture notes 7. Links to external resources 8. Information literacy/library modules 9. Lab modules/activities 10. helped me learn the new skills and concepts for this course 11. helped in that it reduced the amount of time spent studying text or notes in order to grasp the material to be learned 12. helped me to better analyze and apply key points in the material to be learned 13. encouraged me to take responsibility for my own learning 14. I found that working online improved my performance on graded course assessments 15. I found that the online component(s) were well integrated with the other course components not offered online

16. Scale Results Online Learning Activities (alpha.965/.935/.919) FacultyStrongly Disagree DisagreeUndecidedAgreeStrongly Agree Cum. % Positive ENG6527362662 FES61121352762 AHS71322302858 SCIENCE11 24332157 ARTS111220332457 MATH121425341549 Av for % Combined Neg 20 UnD 23 Pos 57

17. Online Learning Activities  “I’m sure there are situations where it would be desirable to “use technology to support teaching and learning”, but that in itself is too broad a goal. You should be using technology to solve a particular problem, or to compensate for a recognized weakness in existing methods. In the case of my course, I can’t see how UW-ACE has added anything to the course. I found the written quizzes far more helpful than the online ones, and I prefer to read printed course notes rather than getting course material from the web (you can’t highlight/underline/scribble on a webpage)”  “it helped me develop the habit of studying for the course regularly. It allowed me to learn course material through practice questions. A chance to do the same quizzes again and again which helped me learn and understand course concepts better. Unlimited time to do the quiz”  “a good way to practice concepts learned in class. Particularly I found the ‘fill in the blank’ type exercises to be very helpful”  “the information was very complete and organized. I feel that the textbook for the course would not have been sufficient on its own to learn all the course material. The resources offered on UW-ACE enhanced my ability to learn the material”

18. Online Collaborative Learning: Learning Activities; Peer Interaction; Reflective Learning; Community 16. Team or small group graded assignments 17. Whole class discussion activities 18. Peer review submissions 19. Team or small group un-graded assignments 20. Giving/receiving peer feedback 21. I was able to develop a better understanding of the material because of my interaction with my peers 22. I put more thought into my comments when interacting with others online 23. I felt part of a learning community when working online 24. I found that working online encouraged me to take responsibility for my group contribution 25. I am more comfortable asking questions to members of my group in an online setting

19. Scale Results Online Collaborative Learning (alpha.961/.964/.895) FacultyStrongly Disagree DisagreeUndecidedAgreeStrongly Agree Cum. % Positive ARTS101422332154 AHS101720341953 SCIENCE12728292453 FES81627431749 ENG191318351449 MATH141235 439 Av for % Combined Neg 25 UnD 25 Pos 50

20. Online Collaborative Learning: Benefits  “the group discussion page made it a lot easier to communicate with my group members since we are all in different classes and communication for projects is an issue”  “I don’t mind the discussions online, I’m more comfortable since it gives the sense of anonymity (privacy thing)”  “It depends on what type of activities: I think group tutorials are a good way to learn, whereas group projects are hard to realize”  “they allow me to communicate with others, evaluate their thoughts and compare them to my own”  “the addition would benefit learning because it would provide a place for quick review and knowledge testing as well as the style of the lecturers questions”

21. Construct Validity: Pearson Correlation Matrix

22. Identifying Patterns Science: Technology (85) and Student Satisfaction (77) 81% Instructor Feedback (64), Learning Activity (57), Collaborative Learning (53) 57% Applied Health Sciences: Technology (82) and Student Satisfaction (77) 80% Instructor Feedback (67), Learning Activity (58), Collaborative Learning (53) 59% Environmental Studies: Technology (83) and Student Satisfaction (74) 79% Learning Activity (62), Instructor Feedback (55), Collaborative Learning (49) 55% Arts: Instructor Feedback (72), Technology (83), Student Satisfaction (74) 76% Learning Activity (57) and Collaborative Learning (54) 56% Math: Technology (74) and Student Satisfaction (62) 68% Learning Activity (49), Instructor Feedback (43), Collaborative Learning (39) 44% Engineering: Technology (67), Student Satisfaction (67), Learning Activity (62) 65% Instructor Feedback (54) and Collaborative Learning (49) 52%

23. All Faculties, All Scales

24. General Patterns  Technology function and student satisfaction scales represent the more straightforward scales group addressing technical support and student workload in all Faculties  Online Learning Activity, Online Collaborative Learning (Small Group) and Instructor Feedback scales represent the teaching and learning pedagogy group in four of the six Faculties  Exceptions Instructor Feedback, a pedagogy scale, grouped with technical support and student workload Instructor Feedback, a pedagogy scale, grouped with technical support and student workload Learning Activity, a pedagogy scale, grouped with technical support and student workload Learning Activity, a pedagogy scale, grouped with technical support and student workload

25. Faculty Scales

28. Survey Results: 865 Students; 113 Courses; 6 Faculties

29. Percent Student Response Ratings by Faculty FacultyStrongly Disagree UndecidedAgreeStrongly Agree Total # Responses Total # Course Total # Students Taking Survey AHS5918333519241290 ARTS79163632401342185 ENG711223624904637 ES6929362918361696 MATH121025351826289120 SCI77203234629928337 Av. %7.339.1620.1734.6728.6717,604113865 % Rounded 79203529

30. Overview of Student Commentary  Students are aware of the benefits of technology in the classroom and commented frequently on instructors under use of this resource.  They responded positively to technology use perceived to add real value to learning outcomes as evidenced in improved performance on quizzes, tests, assignments or projects.  Students provided suggestions for improvements and/or described new features they would like to see developed for UW-ACE.  They did not see value-added in working in small groups or teams online when they could readily interact with their peers in person.

31. Recommendations for Online Pedagogy  We need to further explore ways to improve instructional design and teaching techniques for online learning and collaborative learning activities. (LT3 Centre interface)  Some of the needed changes suggested by the students should be considered for implementation either in the technology itself, using the technology more fully to support learning, and reducing the ambiguity (undecided) noted in all the Scales except Technology Function.

32. Looking Ahead  We need to think how to reduce the “undecided” rates, Rethink providing this as a response choice Rethink providing this as a response choice Forced choice is one option Forced choice is one option Use “intermediate” instead of “undecided” Use “intermediate” instead of “undecided” Provide instructors with key information on the pedagogy of online teaching and learning Provide instructors with key information on the pedagogy of online teaching and learning  Encourage instructors to direct their students to the online survey available in students’ course pages  Analysis of the features of individual course profiles should be undertaken (Liaisons)

33. THE Capstone Statement Learning technologies should be used to support, enhance, supplement and/or expand learning opportunities; Technology should NEVER be used to replace teaching and learning in the classroom setting. Learning technologies should be used to support, enhance, supplement and/or expand learning opportunities; Technology should NEVER be used to replace teaching and learning in the classroom setting.