1. 10/30/01EDUCAUSE 20011 Copyright Statement Copyright [Mzumara, Gavrin, & Chisholm] [2001]. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

2. 10/30/01EDUCAUSE 20012 WebScience: Assessment of Just-in-Time Teaching (JiTT) at IUPUI Howard R. Mzumara Andrew D. Gavrin & Robin Chisholm Indiana University Purdue University Indianapolis Tel: 317-278-2214 Fax: 317-274-3400 E-mail: hmzumara@iupui.eduhmzumara@iupui.edu http://webphysics.iupui.edu/webscience/assessment.html

3. 10/30/01EDUCAUSE 20013 Presentation Outline Brief background information about JiTT What is JiTT? Components of JiTT Strategy JiTT Diversity Main Instructional Objectives for JiTT Class Assessment of JiTT: Evaluation Questions Methods of Data Collection Methods of Data Analysis Preliminary Results To Date Summary of Findings and Conclusions

4. 10/30/01EDUCAUSE 20014 The Problem Many students come to college underprepared to take college-level courses Introductory science and math classes have a mix of traditional and nontraditional students with a wide range of backgrounds and skills Student retention is a major problem Goal: Meet students at their current ability level and help them learn and gain from college courses they enroll in, and help to improve retention.

5. 10/30/01EDUCAUSE 20015 What is JiTT? JiTT involves Web assignments and an interactive classroom that fundamentally rely upon one another JiTT is a strategy for improving student learning, retention, and attitudes http://webphysics.iupui.edu/jitt/jitt.html

6. 10/30/01EDUCAUSE 20016 JiTT Requires Feedback Feedback Loop between the Web and Classroom Activities

7. 10/30/01EDUCAUSE 20017 Web-based Components of JiTT “Warm-Up”- short quiz or conceptual exercise with an electronic due date just hours before class “Puzzle” - weekly assignment “What is (Physics) Good For?” – weekly essay on a related topic (“extra credit”) Information and communication pages –Chapter Outlines, News, etc.

8. 10/30/01EDUCAUSE 20018 JiTT Diversity Several Science & Math Courses use JiTT: – Physics 152 (Mechanics and Sound) & Physics 251 (Heat, Electricity and Optics) – Biology N100 (Contemporary Biology) – Chemistry 105 (Principles of Chemistry) – Mathematics M118 (Finite Mathematics) & Mathematics 164 (Integrated Calculus & Analytic Geometry)

9. 10/30/01EDUCAUSE 20019 JiTT Course Homepages at IUPUI: http://webphysics.iupui.edu/webscience/webscience.html Physics 152 & 251: http://webphysics.iupui.edu/introphysics.html http://webphysics.iupui.edu/introphysics.html Biology N100: http://www.biology.iupui.edu/biocourses/N100/index.html Chemistry 105: http://webphysics.iupui.edu/webscience/courses/chem105/ chem105home.html Mathematics M118: http://webphysics.iupui.edu/webscience/courses/mathm118 /mathm118home.html

10. 10/30/01EDUCAUSE 200110 Main Instructional Objectives: Encourage frequent, short study sessions (facilitate development of good study habits and time management skills) Connect textbook to the real world Encourage the development of students’ critical thinking skills, estimation skills, and ability to deal with ill-defined problems Develop cooperative work habits and communication skills Emphasize teamwork, collaborative learning, and effective communication

11. 10/30/01EDUCAUSE 200111 Using the Warm-Ups: “The Interactive Lecture” Before class: –½ hour to review responses –15 minutes to select quotes –Lecture emphasis may be adjusted During Class: –Lectures are built around student answers to Warm-Ups –Quotes from scafford –Lecture content is unaffected –Students gain ownership of “lecture” session

12. 10/30/01EDUCAUSE 200112 The Recitation Section Format of collaborative recitation/discussion session: –15 minute “sermon” from instructor (short review of homework) –Students form groups (teams of 2-4 students) –Groups work on white-boards (and practice problem solving, teamwork & communication) –Faculty, peer mentors circulate in classroom

13. 10/30/01EDUCAUSE 200113 High Tech vs. Low Tech Approach High Tech: –“Warm-Up Questions” –Electronic communication –Other online activities (e.g., Web search) Low Tech: –Workplace skills –Course community –Mentoring

14. 10/30/01EDUCAUSE 200114 For more about Just-in-Time Teaching Book by Novak, et al. (1999) Just-in-Time Teaching: Blending Active Learning with Web Technology (published by Prentice Hall) JiTT Website: http://webphysics.iupui.edu/jitt.htmlhttp://webphysics.iupui.edu/jitt.html Physics Text: http://www.prenhall.com/giancoli/http://www.prenhall.com/giancoli/

15. 10/30/01EDUCAUSE 200115 Assessment of JiTT: Evaluation Questions To what extent is the JiTT strategy helping students? In what specific areas is the introduction of JiTT making an impact in facilitating teaching and learning in science and mathematics courses? In what ways can the JiTT courses be significantly improved?

16. 10/30/01EDUCAUSE 200116 Data Collection Methods Mixed-method evaluation approach included: Document Review Student Attitude Surveys Classroom Observation Focus Groups Achievement Tests or Common Final Exams

17. 10/30/01EDUCAUSE 200117 JiTT Assessment: Evaluation Instruments Student Attitude Surveys –Pre- and Post-Course Survey consisting of six (6) sub-scales or dimensions: Confidence Expectations Rapport Study Skills Group Work Computer

18. 10/30/01EDUCAUSE 200118 JiTT Assessment: Evaluation Instruments – cont’d Classroom Observation –Rating Scale and Narrative Form Purpose: To obtain a view of the climate, rapport, interaction, and functioning of JiTT classes (particularly with respect to checking the adequacy and effectiveness of the JiTT strategy in facilitating interactivity, teamwork, or collaborative learning among students and faculty)

19. 10/30/01EDUCAUSE 200119 JiTT Assessment: Evaluation Instruments – cont’d Student Focus Groups –Purpose: To obtain students’ perspectives and experiences regarding use of Web resources to enhance learning, particularly with respect to making classroom activities more interactive, efficient, and more appropriate for meeting needs. –Focus groups were designed to evaluate the impact and effectiveness of the JiTT strategy (particularly use of Web resources) in support of teaching and learning in science and math.

20. 10/30/01EDUCAUSE 200120 URL for JiTT Assessment Copies of assessment instruments are available online at: http://webphysics.iupui.edu/webscience/ assessment.html

21. 10/30/01EDUCAUSE 200121 Methods of Data Analysis Qualitative Data Analysis: –Content Analysis –Formulation of Coding Categories or Theme Quantitative Data Analysis –Descriptive and Inferential Statistics –Use of Frequency Counts and/or Percentages

22. 10/30/01EDUCAUSE 200122 Reporting of Evaluation Findings Three Broad Areas of Assessment: Course Attrition Rates (retention/persistence) Student Cognitive Gains (cognitive factors) Student Attitudes (affective area)

23. 10/30/01EDUCAUSE 200123 Assessment: Attrition Rates (PHYSICS) MeasureNo JITTUsing JiTT DFW rate (Mechanics) 48%29% DFW rate (E&M) 33%17% Cognitive Gain on FCINot Available0.31

24. 10/30/01EDUCAUSE 200124 Assessment: Attrition Rates (PHYS 152)

25. 10/30/01EDUCAUSE 200125 Assessment: Attrition Rates (PHYS 251)

26. 10/30/01EDUCAUSE 200126 Cognitive Area: GPA Comparisons Preliminary GPA Comparison in Physics (Note: USAFA classes only): GROUPIncoming GPAOutgoing GPA No JiTT2.6932.719 Consistent JiTT2.7962.967

27. 10/30/01EDUCAUSE 200127 Cognitive Area: Performance on FCI Force Concept Inventory (FCI): –Nationally accepted achievement test –Conceptual understanding – = 0.23 (traditional courses) – = 0.31 (IUPUI Phys 152) – = 0.48 (good interactive courses)

28. 10/30/01EDUCAUSE 200128 Affective Area: Student Attitudes Student responses to end-of-semester voluntary anonymous survey on the use of JiTT strategy were overwhelmingly positive 94% of students indicated they thought Warm-Ups were a good idea Students commented repeatedly that Warm-Ups helped them to learn rather that memorize, be prepared for class, and keep up (pace) with material throughout the semester

29. 10/30/01EDUCAUSE 200129 Affective Area - continued Post-course satisfaction survey: –Students were asked “If you could pick your instructor and you had a choice between two equal instructors, except that one instructor used Warm-Ups and the other didn’t, which instructor would you pick?” 91% preferred Warm-Ups.

30. 10/30/01EDUCAUSE 200130 Affective Area – continued Post-course satisfaction survey: –Overall I would rate the Warm-Up system as Excellent (10) – Very good (7.5) – Satisfactory (5) – poor (2.5) – Very poor (0). Result (overall rating): 8.21

31. 10/30/01EDUCAUSE 200131 Affective Area – continued Are Warm-Ups a good idea? Why? –“The Warm-Ups are OK. They seem to help us think before lecture into the main points that will be discussed, so that we are not that lost when we walk into class.” –“I think the Warm-Ups are a good idea because without them most students would not even look at the material before class.” –“I think that Warm-Ups are a good idea in that it makes us get in the book and search things out.”

32. 10/30/01EDUCAUSE 200132 Affective Area – continued Are Warm-Ups a good idea? Why? –“Yes! They force me to look at the new stuff a little before coming to class, and help me get a general idea on if I need to pay special attention in class.” –“Yeah! I think the Warm-Ups are a very good idea. It helps me to get a head start on understanding the new materials better.”

33. 10/30/01EDUCAUSE 200133 Affective Area - continued Improvement in Students’ Study Habits: –Discourage cramming –Encourage daily work Practice Skills: –Analytic thinking –Estimation skills –Ill-defined problems –Communication and teamwork

34. 10/30/01EDUCAUSE 200134 JiTT Assessment: References and Resources Gavrin & Novak (1997) Novak, Patterson, Gavrin, & Enger (1998) Novak, Patterson, Gavrin, Christian, (1999) Department of Biology (June 2001) Annual Report Department of Physics (July 1999, June 2001) Annual Reports Hake, R. R. (October 2001) Evaluation of Progress on the WebScience (JiTT) Project at IUPUI

35. 10/30/01EDUCAUSE 200135 Conclusion There is no single way to “do JiTT” JiTT works with a wide variety of courses, students, schedules, etc. Affective assessment: Excellent results so far Cognitive assessment: Good –Additional research and/or analysis of relevant data is needed to establish statistical and/or practical significance of findings.

36. 10/30/01EDUCAUSE 200136 Question and Answer Session