1. Ted J. Branoff, Ph.D. North Carolina State University College of Education Department of Mathematics, Science and Technology Education Large Course Redesign of GC120 Moving the Foundations of Graphics Course from a Face-to-face to a Blended Learning Format

2. GC120 - Content

3. Previous Structure of GC120 Daytime sections taught by full-time faculty –Met 2 times per week for 110 minutes –2-3 sections rotated into a computer lab during these meetings Night sections taught by part-time faculty –Met once per week for 3.5 hours –Typically had their own dedicated computer lab

4. Issues with GC120 Consistency between sections –SolidWorks instruction –Evaluating SolidWorks assignments –Proportion of sketching and SolidWorks assignments Access to SolidWorks help outside of class Large number of students on wait list (about 130 each semeter)

5. Large Course Redesign Grant Summer 2008 - Submitted proposal for DELTA Large Course Redesign Grant. Proposed moving all sections of GC120 to a hybrid or blended format. Proposed more laptop sections. Proposed help sessions in Elluminate. Proposed an automated SolidWorks grader.

6. Moving Sections to Blended Format During the Fall of 2008 content was moved from WebCT to Moodle. Currently all day-time sections of GC120 are blended. Continue to train night faculty.

7. New Course Structure Daytime sections meet in class once per week –Answer students questions –Talk about key concepts for the week –Check homework Online content completed outside of class –Voiced-Over PowerPoint Presentations –Voiced-Over SolidWorks Demonstrations –Videos of an instructor sketching –Study Guides –Low-stake Online Assessments in Moodle – 10-20 questions – 2 Attempts

8. More Laptop Sections Gradually moving to more laptop sections. Have renovated Poe 106 to handle 60 students. Still have many issues to overcome with student owned computers.

9. Online Help Sessions Offered some help Elluminate help sessions during the Fall 2008 semester for exam reviews and project help. Students still seem to prefer face-to-face help sessions. We have started offering face-to-face help sessions each semester.

10. Automated SolidWorks Grader Began programming in Fall 2008. Made some improvements during Fall 2009 with departmental funding. Still have issues with ease of use for faculty.

11. Fall 2008 Survey Prior to moving to Moodle, compared 3 pilot sections of blended structure to F2F sections. Materials were organized in open web pages. Conducted in the 13 th week of semester. Questions about: –Previous experience with online or hybrid courses –Instructional preference –General order when navigating through online materials

12. Fall 2008 Results Some previous experience with online and blended courses. Prefer blended over face-to-face. Prefer face-to-face over online. Students navigated through material in many different ways.

13. Fall 2008 Results No differences between face-to-face and blended sections on the midterm exam. Significant difference between the groups for the final exam.

14. Moving to Moodle We can now look at how students are actually navigating through materials. What are the best strategies for success?

15. Fall 2009 Study Research Questions In what order did students progress through the materials related to the textbook? What was the typical number of attempts at each assessment? Did students who attempted all of the online assessments perform better on the midterm and final exams than students who only attempted a few assessments?

16. Participants GC120 Section Frequency% 0035534% 0045634% 0055332% Total164100%

17. Participants – Academic Year YearFrequency% Freshmen42% Sophomores11570% Juniors2918% Seniors1610% Total164100%

18. Participants – Academic Major Academic MajorFrequency% Aerospace Engineering1710% Civil Engineering/Construction Man.4527% Mechanical Engineering4427% Other Engineering Majors2918% Education74% First Year College85% Other Majors149% Total164100%

19. Moodle LMS

20. Moodle Lesson Format

21. Voiced-Over Presentations

22. Sketching Videos

23. Results In what order did students progress through the materials related to the textbook?

24. Results – Order of Navigation

25. Results What was the typical number of attempts at each assessment?

26. Results – Number of Assessment Attempts

27. Results Did students who attempted all of the online assessments perform better on the midterm and final exams than students who only attempted a few assessments?

28. Results – Midterm Means by Assessment Attempts

29. Results – Exam Means by Assessment Attempts

30. Analysis – Correlation between Midterm Exam and Online Assessment Attempts Spearmans Rho Assessment Attempts Before Midterm Midterm Exam Assessment Attempts Before the Midterm Exam Correlation Coefficient 1.000 Sig. (2-tailed). N 164 Midterm Exam Correlation Coefficient.233**1.000 Sig. (2-tailed).004. N 164 ** Correlation is significant at the 0.01 level (2-tailed)

31. Analysis – Correlation between Final Exam and Online Assessment Attempts Spearmans Rho Assessment Attempts Before Final Exam Final Exam Assessment Attempts Before the Final Exam Correlation Coefficient 1.000 Sig. (2-tailed). N 164 Final Exam Correlation Coefficient.283**1.000 Sig. (2-tailed).000. N 164 ** Correlation is significant at the 0.01 level (2-tailed)

32. Discussion / Conclusions While a number of students followed the explicitly recommended order of material use (i.e., view the streaming video before attempting the quiz), many students took alternate approaches to usage order. In addition, this usage order also changed over the course of the semester.

33. Discussion / Conclusions The weekly online quiz assessments were only worth a total of 10% of students final grade and the midterm and final worth collectively 40% of the grade. It is our assumption that students primarily saw the value in the weekly assessments as preparing them for the larger summative assessments of the midterm and final.

34. Discussion / Conclusions The data collected seem to support the conclusion that those students who attempted more weekly assessments (and/or made use of the streaming videos) did better on the midterm and final exams.

35. Discussion / Conclusions Probably the most important finding of this study is that the logging tools provided in Moodle provides a powerful tool for instructors to gather and analyze data on how students make use of the resources provided online. It is now easier to chart student trends and performance in a more accurate way than in past incarnations of this course.

36. Discussion / Conclusions This provides instructors with better information to use in the redesign of course materials for the future. Since the production of the multimedia learning resources is a labor-intensive practice, this formative data provides valuable evidence as to whether such material is being used by students and whether it provides real educational value.

37. Future Research Develop methods for more fine-grained analysis of log data. –More data points over the semester to better understand trends –Analysis of first versus second tries on quizzes –Use of SCORM-compliant learning resources in conjunction with Moodle that allow richer data collection usage (e.g., how long did they view a video and how many times did they stop and start it)

38. Questions?