10/30/01EDUCAUSE Copyright Statement Copyright [Mzumara, Gavrin, & Chisholm] [2001]. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

10/30/01EDUCAUSE WebScience: Assessment of Just-in-Time Teaching (JiTT) at IUPUI Howard R. Mzumara Andrew D. Gavrin & Robin Chisholm Indiana University Purdue University Indianapolis Tel: Fax:

10/30/01EDUCAUSE Presentation Outline Brief background information about JiTT What is JiTT? Components of JiTT Strategy JiTT Diversity Main Instructional Objectives for JiTT Class Assessment of JiTT: Evaluation Questions Methods of Data Collection Methods of Data Analysis Preliminary Results To Date Summary of Findings and Conclusions

10/30/01EDUCAUSE The Problem Many students come to college underprepared to take college-level courses Introductory science and math classes have a mix of traditional and nontraditional students with a wide range of backgrounds and skills Student retention is a major problem Goal: Meet students at their current ability level and help them learn and gain from college courses they enroll in, and help to improve retention.

10/30/01EDUCAUSE What is JiTT? JiTT involves Web assignments and an interactive classroom that fundamentally rely upon one another JiTT is a strategy for improving student learning, retention, and attitudes

10/30/01EDUCAUSE JiTT Requires Feedback Feedback Loop between the Web and Classroom Activities

10/30/01EDUCAUSE Web-based Components of JiTT “Warm-Up”- short quiz or conceptual exercise with an electronic due date just hours before class “Puzzle” - weekly assignment “What is (Physics) Good For?” – weekly essay on a related topic (“extra credit”) Information and communication pages –Chapter Outlines, News, etc.

10/30/01EDUCAUSE JiTT Diversity Several Science & Math Courses use JiTT: – Physics 152 (Mechanics and Sound) & Physics 251 (Heat, Electricity and Optics) – Biology N100 (Contemporary Biology) – Chemistry 105 (Principles of Chemistry) – Mathematics M118 (Finite Mathematics) & Mathematics 164 (Integrated Calculus & Analytic Geometry)

10/30/01EDUCAUSE JiTT Course Homepages at IUPUI: Physics 152 & 251: Biology N100: Chemistry 105: chem105home.html Mathematics M118: /mathm118home.html

10/30/01EDUCAUSE Main Instructional Objectives: Encourage frequent, short study sessions (facilitate development of good study habits and time management skills) Connect textbook to the real world Encourage the development of students’ critical thinking skills, estimation skills, and ability to deal with ill-defined problems Develop cooperative work habits and communication skills Emphasize teamwork, collaborative learning, and effective communication

10/30/01EDUCAUSE Using the Warm-Ups: “The Interactive Lecture” Before class: –½ hour to review responses –15 minutes to select quotes –Lecture emphasis may be adjusted During Class: –Lectures are built around student answers to Warm-Ups –Quotes from scafford –Lecture content is unaffected –Students gain ownership of “lecture” session

10/30/01EDUCAUSE The Recitation Section Format of collaborative recitation/discussion session: –15 minute “sermon” from instructor (short review of homework) –Students form groups (teams of 2-4 students) –Groups work on white-boards (and practice problem solving, teamwork & communication) –Faculty, peer mentors circulate in classroom

10/30/01EDUCAUSE High Tech vs. Low Tech Approach High Tech: –“Warm-Up Questions” –Electronic communication –Other online activities (e.g., Web search) Low Tech: –Workplace skills –Course community –Mentoring

10/30/01EDUCAUSE For more about Just-in-Time Teaching Book by Novak, et al. (1999) Just-in-Time Teaching: Blending Active Learning with Web Technology (published by Prentice Hall) JiTT Website: Physics Text:

10/30/01EDUCAUSE Assessment of JiTT: Evaluation Questions To what extent is the JiTT strategy helping students? In what specific areas is the introduction of JiTT making an impact in facilitating teaching and learning in science and mathematics courses? In what ways can the JiTT courses be significantly improved?

10/30/01EDUCAUSE Data Collection Methods Mixed-method evaluation approach included: Document Review Student Attitude Surveys Classroom Observation Focus Groups Achievement Tests or Common Final Exams

10/30/01EDUCAUSE JiTT Assessment: Evaluation Instruments Student Attitude Surveys –Pre- and Post-Course Survey consisting of six (6) sub-scales or dimensions: Confidence Expectations Rapport Study Skills Group Work Computer

10/30/01EDUCAUSE JiTT Assessment: Evaluation Instruments – cont’d Classroom Observation –Rating Scale and Narrative Form Purpose: To obtain a view of the climate, rapport, interaction, and functioning of JiTT classes (particularly with respect to checking the adequacy and effectiveness of the JiTT strategy in facilitating interactivity, teamwork, or collaborative learning among students and faculty)

10/30/01EDUCAUSE JiTT Assessment: Evaluation Instruments – cont’d Student Focus Groups –Purpose: To obtain students’ perspectives and experiences regarding use of Web resources to enhance learning, particularly with respect to making classroom activities more interactive, efficient, and more appropriate for meeting needs. –Focus groups were designed to evaluate the impact and effectiveness of the JiTT strategy (particularly use of Web resources) in support of teaching and learning in science and math.

10/30/01EDUCAUSE URL for JiTT Assessment Copies of assessment instruments are available online at: assessment.html

10/30/01EDUCAUSE Methods of Data Analysis Qualitative Data Analysis: –Content Analysis –Formulation of Coding Categories or Theme Quantitative Data Analysis –Descriptive and Inferential Statistics –Use of Frequency Counts and/or Percentages

10/30/01EDUCAUSE Reporting of Evaluation Findings Three Broad Areas of Assessment: Course Attrition Rates (retention/persistence) Student Cognitive Gains (cognitive factors) Student Attitudes (affective area)

10/30/01EDUCAUSE Assessment: Attrition Rates (PHYSICS) MeasureNo JITTUsing JiTT DFW rate (Mechanics) 48%29% DFW rate (E&M) 33%17% Cognitive Gain on FCINot Available0.31

10/30/01EDUCAUSE Assessment: Attrition Rates (PHYS 152)

10/30/01EDUCAUSE Assessment: Attrition Rates (PHYS 251)

10/30/01EDUCAUSE Cognitive Area: GPA Comparisons Preliminary GPA Comparison in Physics (Note: USAFA classes only): GROUPIncoming GPAOutgoing GPA No JiTT Consistent JiTT

10/30/01EDUCAUSE Cognitive Area: Performance on FCI Force Concept Inventory (FCI): –Nationally accepted achievement test –Conceptual understanding – = 0.23 (traditional courses) – = 0.31 (IUPUI Phys 152) – = 0.48 (good interactive courses)

10/30/01EDUCAUSE Affective Area: Student Attitudes Student responses to end-of-semester voluntary anonymous survey on the use of JiTT strategy were overwhelmingly positive 94% of students indicated they thought Warm-Ups were a good idea Students commented repeatedly that Warm-Ups helped them to learn rather that memorize, be prepared for class, and keep up (pace) with material throughout the semester

10/30/01EDUCAUSE Affective Area - continued Post-course satisfaction survey: –Students were asked “If you could pick your instructor and you had a choice between two equal instructors, except that one instructor used Warm-Ups and the other didn’t, which instructor would you pick?” 91% preferred Warm-Ups.

10/30/01EDUCAUSE Affective Area – continued Post-course satisfaction survey: –Overall I would rate the Warm-Up system as Excellent (10) – Very good (7.5) – Satisfactory (5) – poor (2.5) – Very poor (0). Result (overall rating): 8.21

10/30/01EDUCAUSE Affective Area – continued Are Warm-Ups a good idea? Why? –“The Warm-Ups are OK. They seem to help us think before lecture into the main points that will be discussed, so that we are not that lost when we walk into class.” –“I think the Warm-Ups are a good idea because without them most students would not even look at the material before class.” –“I think that Warm-Ups are a good idea in that it makes us get in the book and search things out.”

10/30/01EDUCAUSE Affective Area – continued Are Warm-Ups a good idea? Why? –“Yes! They force me to look at the new stuff a little before coming to class, and help me get a general idea on if I need to pay special attention in class.” –“Yeah! I think the Warm-Ups are a very good idea. It helps me to get a head start on understanding the new materials better.”

10/30/01EDUCAUSE Affective Area - continued Improvement in Students’ Study Habits: –Discourage cramming –Encourage daily work Practice Skills: –Analytic thinking –Estimation skills –Ill-defined problems –Communication and teamwork

10/30/01EDUCAUSE JiTT Assessment: References and Resources Gavrin & Novak (1997) Novak, Patterson, Gavrin, & Enger (1998) Novak, Patterson, Gavrin, Christian, (1999) Department of Biology (June 2001) Annual Report Department of Physics (July 1999, June 2001) Annual Reports Hake, R. R. (October 2001) Evaluation of Progress on the WebScience (JiTT) Project at IUPUI

10/30/01EDUCAUSE Conclusion There is no single way to “do JiTT” JiTT works with a wide variety of courses, students, schedules, etc. Affective assessment: Excellent results so far Cognitive assessment: Good –Additional research and/or analysis of relevant data is needed to establish statistical and/or practical significance of findings.

10/30/01EDUCAUSE Question and Answer Session