Assessing the Educational Value of OER The Open Education Group T.J. Bliss Lane Fischer John Hilton III T. Jared Robinson David Wiley

Recent Studies Scottsdale Community College Houston Community College Virginia State University Kaleidoscope Open Course Initiative Mercy College Nebo School District

Houston Community College (paper available)paper available Houston Community College (HCC) is a large community college with more than 70,000 students. 33% of its students are Hispanic, 33% are African American, 17% are white, 14% Asian, and 3% are classified as ‘other.’ Fifty- nine per cent of the students are female, and 41% are male.

Textbook Adoption In 2011 HCC’s Psychology department chose to adopt Flat World Knowledge’s (FWK) textbook Introduction to Psychology. During a pilot study 690 students used the FWK text. So what happened?

HCC – Student Feedback 84% of students surveyed agreed with the statement that “Having a free online book helps me go to college.” Of the 108 students who responded to a question regarding the difficulty of using an online text, 45 (42%) said that it was easy, 28 (26%) said that it was moderately easy, 26 (24%) said that it was neither hard nor easy, and nine (8%) said that it was moderately difficult.

HCC – Results Comparing Students from two instructors who taught in Spring 2011 and Fall 2011

Virginia State University (paper available)paper available Virginia State University (VSU) has 5,300 students. They offer 55 baccalaureate and masters degree programs, two doctoral degree programs and three certificates. VSU’s Business Department adopted nine FWK texts and paid $30.00 per student for an unlimited seat license for all FWK content.

VSU – Student Feedback from 148 students

VSU – Results 95% of students accessed at least some of FWK resources (in contrast with 47% of students purchasing textbooks one year previously). Students downloaded a variety of resources

VSU – Results In total there were seven courses that used FWK textbooks (COBU101, COBU200, COBU201, COBU210, COBU300, COBU301, COBU302, COBU310, MISY350), and ten courses that did not use FWK textbooks (COBU110, COBU111, COBU155, COBU170, COBU202, COBU260, COBU304, COBU342, COBU343, COBU400). While these were clearly different courses, they were roughly equivalent in terms of course difficulty. ABC_ in Core Course s using FWK DFW_ in core courses using FWK TotalDFW % with FWK ABC_ in core courses w/o FWK DFW_ in core courses w/o FWK TotalDFW % not using FWK Fall % % Spring % %

Kaleidoscope Open Course Initiative (paper available)paper available Kaleidoscope Open Course Initiative ( kaleidoscope.org/) is an open education initiative initially comprised of eight community colleges and open access four-year institutions. These colleges work together to create courses that replace traditional, textbooks with OER. kaleidoscope.org/

Kaleidoscope Open Course Initiative During the academic year, Kaleidoscope Open Course Initiative impacted about 4,000 students across 80 teachers in eight institutions.

Kaleidoscope Open Course Initiative 58 teachers from 8 PK institutions completed at least some items on a questionnaire. 490 students completed at least some items on a questionnaire.

Kaleidoscope Open Course Initiative: Cost Instructor reports of textbook costs during Kaleidoscope Instructor reports of textbook costs pre- Kaleidoscope

Quantifying Textbook Costs

Average book cost per class section: $92.90 Total potentially spent by non- kaleidoscope students: $997, Total potentially saved by kaleidoscope students: $360, Total potential savings: $1,357,733.50

Kaleidoscope Open Course Initiative – Teacher Perceptions

Teachers: How do the OER textbooks compare with other textbooks you have used? Percent of teachers Teacher perceptions of quality

Students: How do the OER textbooks compare with other textbooks you have used? Student perceptions of text quality Percent of students

Kaleidoscope Open Course Initiative – Student Perceptions 160 students provided a description of what they thought made the OER texts better. Their responses clustered in six major categories: technical advantages (8%), learning aides (9%), customization (10%), cost (20%), access (26%), and quality/presentation (27%).

Kaleidoscope Open Course Initiative – Student Perceptions Conversely, 20 students provided a description of what they thought made the OER texts worse than other texts they had used in the past. Their responses clustered in 2 major categories; six students described technology issues and 15 students took issue with text quality.

Number of student comments

Mercy College Math About MercyAbout Mercy: Mercy College is dedicated to making a high-quality education attainable offering more than 90 undergraduate and graduate programs within its five schools: Business, Education, Health and Natural Sciences, Liberal Arts, and Social and Behavioral Sciences. The New York metropolitan area college of nearly 10,000 students has four campus locations including: Dobbs Ferry, Bronx, Manhattan, and Yorktown Heights and offers more than 25 degree programs online.

No OERAll OER Math116Fall 2011Fall 2012 TOTAL STUDENTS Pass (A-C) Not Pass (D/F/FW) % Pass63.60%68.90% % Not Pass36.40%31.10% Mercy College Math

Nebo School District The Nebo School District is a public school district located about one hour south of Salt Lake City. It serves 40 schools and 30,500 students.

Nebo School District Does the adoption of open science textbooks significantly affect science learning outcomes for secondary students?

4,137 students took the state Criterion Referenced Tests (CRTs) in earth systems, biology, or chemistry. (Excludes AP sections, special education sections, and bio-agriculture sections) 2,350 students used traditional textbooks, while 1,787 used OER textbooks.

Method Quasi-experimental design with – Treatment and control groups – Groups matched by propensity score Dependent variable: 2012 science CRT score Independent variable: Textbook condition Covariates: age, gender, special education, English language proficiency, 2011 test data, 2011 GPA, race, and free and reduced lunch

Matching Propensity Score Matching Based on the conditional probability of assignment to treatment condition, given the covariates Increased group balance by 98% Approximates random assignment

Results

Summary Houston Community College – Exploring a variety of other OER. Virginia State University – Continues to use FWK texts. Kaleidoscope Open Course Initiative – 28 schools Mercy Math – Expanding work with OER. Nebo School District – Utah has approved the textbooks for statewide use; preliminary surveys suggest up to 75,000 students will use the open textbooks.

Questions? John Hilton III