1. Performance on In-class vs
Performance on In-class vs. Online Administration of Low-Stakes Assessments
Xochith Herrera1, Manher Jariwala2, Jayson Nissen1, Eleanor Close3, Ben Van Dusen1
1California State University, Chico, 2Boston University, 3Texas State University,
Introduction
Data Analysis with Hierarchical Linear Models
Research-based assessments (RBA) are powerful tools for transforming physics courses.
But using them in the classroom can be a real pain.
Level 2
(N=25)
Course Type:
Algebra-Based Mech.
Calculus-Based Mech.
Calculus-Based E&M
Missing Data
Course
Section
Student
Test
Score
Imputed with Hierarchical Multiple Imputations.
Missing data imputed m times to create m complete data sets.
Each complete data set is analyzed independently.
Results from the m analyses are combined.
Preferable to only using matched data as it maximizes statistical power and has the same basic assumptions1.
Level 1 (N=1,310)
Testing Condition: Completed Concept Inventory as a
PPT – paper and pencil test
CBT – computer-based test
Designed to address the needs of
educators and researchers, the
LASSO platform hosts, administers, scores, and aggregates a variety of RBAs for researchers and educators. Thereby, LASSO seeks to transform
We built two-level Hierarchical Linear Models for both the pretest and posttest scores with the HLM 7.01 software.
Outcome variable of test score
Level 2 variables for the course types
Level 1 variable for testing condition
RBAs from a research tool used at elite institutions into an easy to use tool that drives course transformations across STEM disciplines at a wide variety of institutions.
1. Y. Dong and C.-Y. J. Peng, “Principled missing data methods for researchers,” Springerplus 2, 0 (2013).
HLM Results
Conclusion
Collecting data with LASSO provides equivalent data to that collected with paper and pencil tests.
This can free up in class time and reduce the barriers to instructors using RBAs.
One limitation is that we do not know the extent to which missing data may skew results. Our use of HMI may have mitigated this potential effect.
Research Question
To what extent were scores on concept inventories collected online outside of class with the LASSO platform similar to those collected in class using paper and pencil tests?
Instructors and researchers use our recommended practices to maximize student participation.
Design and Methods
Student Participation
An experimental design that divided 1,310 students from 25 sections in 3 different courses into 2 groups using random stratified sampling.
For more information see Poster XXXXXXXX
In conjunction with this research we investigated how instructor practices influenced student participation rates using Hierarchical Generalized Linear Models.
We found that when instructors use all four recommended practices participation rates with LASSO were similar to those with paper and pencil tests
Differences in scores between conditions for both pre and post test
Were small and inconsistent
Were unreliable
The addition of testing condition to the models increased the variance, indicating it was a very poor predictor of test scores2.
Posttests
Paper CLASS
Online FCI or CSEM
Paper FCI or CSEM
Online CLASS
Pretests
Group 1
Group 2
Class split in two
The LASSO Platform
Predicted scores with 95% CIs.
Create account at:
Creating course within LASSO
Assigning course assessments
Uploading class roster and launching the pretests
Launching posttests at the end of the course
Download summary report
The predicted test scores for PPT and CBT conditions fell well within the 95% confidence interval for the other testing condition in the same course.
Recommended Practices
The assessments used in the study were:
FCI - Force Concept Inventory
CSEM - Conceptual Survey on Electricity and Magnetism
CLASS - Colorado Learning Attitudes about Science Survey
2. S.W. Raudenbush and A.S. Bryk, Hierarchical Linear Models: Applications and Data Analysis Methods (SAGE Publications, 2002).
reminders
Pretest participation credit
In class reminders
Posttest participation credit
This work funded in part by the NSF under grant nos. DUE ,
DUE  DUE & DUE
Back ground photo and image of rake and legs is sand art by Tony Plant, images courtesy of the Daily Mail © London Media. Cartoon of man stepping on rake: artist unknown.