Option 3: Project Planning Time Go To http://educause.adobeconnect.com/discussion1/ Option 1: The SEI Consultant Is In Option 2: SEI Case Brief Stay Here Option 3: Project Planning Time Return here in 30 minutes

Assessing the “effectiveness” of podcasts deployed to support learning in inquiry-based chemistry laboratories Cynthia B. Powell, Ph.D. Department of Chemistry and Biochemistry Abilene Christian University April 13, 2011

THE PLAN: LABORATORY RESOURCE FILES Written laboratory materials MSDS sheets Syllabi, grading rubrics, and safety contract Podcasts in two categories: Chemistry calculations & concepts General laboratory techniques MSDS in window…..you really can see what you need to!

RESEARCH DESIGN Resource files and podcasts that can be accessed on the iPhones as needed during labs taught with an inquiry-based curriculum Quasi-experimental design with two treatment groups: podcast treatment group and a pre-lab lecture treatment group Gather quantitative and qualitative data to compare the effect of the treatments Description of preparation and deployment of podcasts. Hope is to allow students to gather information “knowledge” about lab procedures so that they can work more independently.

SUMMARY OF RESEARCH DESIGN Group Treatment Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Week 11 Week 12 “Podcast” treatment Podcast Access X Pre-laboratory Lecture “Lecture” treatment **Interrupted time series design

GOAL IN DATA COLLECTION In any type of education research you should be concerned about gathering RELEVANT information with maximum VALIDITY and RELIABILITY. Began with surveys completed by TAs, instructors and students during the first semester of deployment to get a qualitative picture of podcast usage and effectiveness to help determine relevant research questions.

RESEARCH QUESTIONS Focus of Research Question 1: How frequently will student research teams access podcasts? Focus of Research Question 2: Are there differences in the number, types, and topics of interactions between students who have podcast access and those taught with a pre-lab lecture? Focus of Research Question 3: Are there performance differences between students who have podcast access and those taught with a pre-lab lecture?

ASSURING VALIDITY OF PODCASTS Polling of instructors who had used the curriculum for multiple semesters Careful preparation of podcasts Evaluation of podcasts by chemistry professors Evaluation of podcasts by focus group members: (a) strong students (b) weak students (c) teaching assistants

ASSURING RELIABILITY OF DEPLOYMENT Determining how to deploy the podcasts in an easily accessible format Finding a secure and dependable way to release the podcasts that allowed reliable tracking of usage and testing deployment during a pilot study Finding a secure way to release the podcasts that would limit cross-contamination

SELECTING THE SAMPLES : VALIDITY Used samples of convenience as opposed to random sampling out of necessity. (Same instructor, same curriculum, student selected schedule) Compared the samples to determine whether there were confounding issues that might skew the data. Use t-tests and chi-square analysis to determine if there were statistically significant differences. (1) ACT/SAT scores (2) gender distribution (3) motivation level distribution (4) classification distribution (5) logic test scores (GALT) * 4 podcast treatment sections n=81, 2 lecture treatment sections n=51 RELIABILITY: it’s a required class!!! Format of sample in the research study. Teamwork is essential and fosters independence. Centerpiece of the interaction between instructors and students in a class taught with inquiry-based format is what we’ll call a “scaffolding interaction”…students are encouraged and led when struggling, but they are allowed to struggle with concepts. Podcasts provide some technical support that helps eliminate struggle based on simple knowledge

DEMOGRAPHIC DESCRIPTION OF TREATMENT GROUPS Podcast treatment n = 81 Lecture treatment n = 51 Observed Motivation 40.7% High 40.7% Medium 18.5% Low 39.2% High 39.2% Medium 21.6% Low Classifications 75.6% Freshmen 15.8% Sophomores 7.3% Juniors 1.2% Seniors 72.5% Freshmen 15.7% Sophomores 7.8% Juniors 3.9% Seniors Gender 43.2% Male 56.8% Female 39.2% Male 60.8% Female Mean ACT 25.96 + 3.84 25.20 + 3.27 Mean GALT 9.32 + 2.00 9.00 + 2.16 Shows the demographic data on the two treatment groups….I’ve included this slide just to give a feel for the typical student in our laboratories.

File usage log generated during each class period. FOCUS OF RESEARCH QUESTION 1: HOW FREQUENTLY DID STUDENT RESEARCH TEAMS ACCESS PODCASTS? File usage log generated during each class period. Determine how to deal with multiple access events so that data was recorded consistently. Tally the count data! Look for patterns in usage data.

RESEARCH QUESTION 1: PODCAST USAGE DURING THE LAB PERIOD Accessed podcast Total access events Mean access events per team Simple Statistics 75 3.13 Filtering 73 3.04 Planning an Experiment 59 2.46 Comparing Reactivity of Metals 53 2.21 Using Pipets 46 1.92 Titration Techniques Using Acids Safely 43 1.79 Vernier Gas Pressure Equipment 32 1.33 Collecting a Gas Sample Mass Determination 11 0.46

FOCUS OF RESEARCH QUESTION 2: ARE THERE DIFFERENCES IN THE NUMBER, TYPES, AND TOPICS OF INTERACTIONS BETWEEN STUDENTS WHO HAD PODCAST ACCESS AND THOSE TAUGHT WITH A PRE-LAB LECTURE? Needed a valid and reliable tool for collecting interaction data. Began with thorough literature search, no suitable tool was found, though many relevant studies were found. Development of needed tool: During the pilot project TAs recorded number and topics and types of interactions. Through a process of repeated focus group conversations and using the method of constant comparison, I constructed and tested initial tools. The tool was revised 3 times. This process included over 300 hours of sample data collection. The completed tool was used for data collection during the study.

Tier 1: Type of Interaction: clarifying or follow-up Scaffolding Interaction Categorization Scheme (SICS) Tier 1: Type of interaction Clarifying or Follow-up Tier 2: Topic and Subtopic of interaction Q1 – numerical issue Q2 – ideological issue E3 – tools/equipment E4 – investigative procedure E5 – data (quantitative or qualitative) E6 – safety P7 – claims and evidences P8 – prior knowledge or experiences Tier 2: Topic of Interaction: Q, E, or P Tier 3: Subtopic of Interaction: 1-8

RESEARCH QUESTION 2: VALIDITY OF THE SICS TOOL Validity of types and topics of interactions in SICS established by: (1) thorough process of tool development (2) comparison with published studies on interactions between TAs and students in science classrooms

Teaching assistants by treatment group RESEARCH QUESTION 2: INTER-RATER RELIABILITY FOR TEACHING ASSISTANTS USING THE INTERACTION CATEGORIZATION SCHEME  all aligned data points ( all aligned data points +  data points not aligned) Teaching assistants by treatment group Number Type Topic (Q, E or P) Subtopic (1-8) Podcast treatment TAs mid-semester end-of-semester 1.00 .96 .98 .82 .89 Lecture treatment TAs .90 .95 .50 .60

RESEARCH QUESTION 2: CLARIFYING INTERACTIONS BY EXPERIMENT WEEK During weeks 6,7, and 12 all sections received lectures and there were no podcasts. Further explanation that I don’t feel like typing here.

Mean clarifying interactions by treatment block RESEARCH QUESTION 2: CLARIFYING INTERACTIONS PER TEAM BY TREATMENT BLOCK Treatment group Mean clarifying interactions by treatment block Contrasting treatment Equivalent treatment Week 4 M SD Podcast treatment teams (n = 24) 2.942* .662 1.942 .485 3.950 Lecture treatment teams (n = 14) 4.478* .866 1.977 .605 4.210 * Welch’s t-test indicates these values are statistically significantly different at α = .05 level Cohen’s d = ( Mt – Mc ) / Spooled for contrasting treatment block = 2.18

FOCUS OF RESEARCH QUESTION 3: ARE THERE PERFORMANCE DIFFERENCES BETWEEN STUDENTS WHO HAVE PODCAST ACCESS AND THOSE TAUGHT WITH A PRE-LAB LECTURE? Locally prepared and graded quizzes, tests, lab reports and reflections, therefore extra caution should be taken Validity established by team of chemistry instructors All quizzes and tests were deployed during the pilot study as a “run through”, some adjustments were made All grading rubrics were deployed during the pilot study

RESEARCH QUESTION 3: INTER-RATER RELIABILITY FOR TEACHING ASSISTANTS USING KEYS AND RUBRICS FOR GRADED ASSIGNMENTS

RESEARCH QUESTION 3: MEAN VALUES OF OUTCOME MEASURES Podcast treatment n = 81 Lecture treatment n = 51 Lab Reports 91.6 ± 6.74 90.61 ± 5.83 Quizzes 78.79 ± 11.49 75.84 ± 12.33 Lab Final Exam 72.38 ± 13.56 73.21 ± 11.43 Lab Course Grade 87.09 ± 7.91 85.92 ± 6.66 Though no category shows a statistically significant difference in outcome measures, when we look at this data by factors ie gender, classification, and motivation we do see one statistically significant difference. The students who are highly motivated and where members of the podcast treatment group performed significantly better than those in the lecture treatment group….discussion. No category shows a statistically significant difference at the α = .05 level

RESEARCH QUESTION 3: DEMOGRAPHICS AND INTERACTION EFFECTS? “Highly motivated” Podcast treatment n = 33 Lecture treatment n = 20 Lab Reports 95.99 ± 2.74 91.80 ± 4.45 Quizzes 86.95 ± 6.56 79.44 ± 11.00 Lab Final Exam 83.24 ± 6.91 79.45 ± 10.28 Lab Course Grade 93.64* ± 3.13 88.72* ± 5.93 Though no category shows a statistically significant difference in outcome measures, when we look at this data by factors ie gender, classification, and motivation we do see one statistically significant difference. The students who are highly motivated and where members of the podcast treatment group performed significantly better than those in the lecture treatment group….discussion. * Welch’s t-test indicates these values are statistically significantly different at α = .05 level

STRENGTHS Relatively large sample size, no identified confounding variables Broad data collection, 4 hours x 6 lab sections x 10 weeks = 240 hours of podcast usage data and interaction data, over 3000 graded assignments Pilot project and study Multiple teaching assistants with consistent results across sections with same treatment LIMITATION Study conducted in a General Chemistry labs using locally written curriculum. Limited generalizability

NEXT STEPS…. Study should be repeated in a different settings to establish generalizability of results to other student populations Investigate the role of motivation in use of resources Used mobile devices in other ways during these courses….more to learn!

ACKNOWLEDGEMENTS ACU Mobile Learning Initiative Grants Colleagues in Department of Chemistry and Biochemistry Technology support staff at ACU Dr. Diana Mason, University of North Texas