Student Efficacy: Are They Learning? Rapid Fire Panel, CNX 2014 1 April 2014 Dr. Beck Pitt The Open University

Background

OER Research Hub oerresearchhub.org #oerrhub @OER_Hub 2-year research project based at The Open University (UK) Funded by William & Flora Hewlett Foundation Aiming to build the most comprehensive picture of OER impact Eleven research hypotheses Collaboration model across different educational sectors Fellowship Scheme Global reach but with a US focus Practicing openness: CC-BY licensed research instruments / SOO Course / Impact Map oerresearchhub.org #oerrhub @OER_Hub

Keyword Hypothesis Performance OER improve student performance/satisfaction Openness People use OER differently from other online materials Access OER widen participation in education Retention OER can help at-risk learners to finish their studies Reflection OER use leads educators to reflect on their practice Finance OER adoption brings financial benefits for students/institutions Indicators Informal learners use a variety of indicators when selecting OER Support Informal learners develop their own forms of study support Transition OER support informal learners in moving to formal study Policy OER use encourages institutions to change their policies Assessment Informal assessments motivate learners using OER

Collaboration Model Collaboration Model

Collaboration Model Collaboration Model

OER Impact Map

OER Impact Map http://chaos.open.ac.uk

OER Impact Map http://chaos.open.ac.uk

Methodology

Educator and Student Surveys

Overview OER improve student performance/satisfaction People use OER differently from other online materials OER use leads educators to reflect on their practice OER adoption brings financial benefits for students/institutions Spring-Summer 2013 Developing Questionnaires with OpenStax College IRB Process Autumn 2013-Christmas 2013 Surveys deployed Ongoing Interviews with educators and students Comparative impact data / Stories of Impact Adding evidence to the Impact Map Jan-Apr 2014 Data analysis and Dissemination of findings Interviews with Educators

In total 63 usable survey responses were recorded: Student Survey In total 63 usable survey responses were recorded: OpenStax Newsletter Student User Survey (50) Incentivised College One (3) College Two (10) 44 out of the 63 respondents have used, or currently use, OpenStax College textbooks: OpenStax Newsletter Student User Survey (39) College One (2) College Two (3)

In total 97 usable survey responses were recorded: Educator Survey In total 97 usable survey responses were recorded: OpenStax Adopter Email list (52) OpenStax Newsletter Educator Survey (41)* Incentivised Individual Educators from direct introductions (4) 82 out of the 97 respondents have used, or currently use, OpenStax College textbooks: OpenStax Adopter Email list (47) OpenStax Newsletter Educator Survey (31) Individual Educators from direct introductions (4)

Student Sample 44 Respondents in total (have used/use OSC textbooks) 56.8% Male (n=25) and 43.2% Female (n=19) 86.0% of respondents live in the United States (n=37) Other respondents live in Germany, Canada, Hong Kong, UK & India (14.0%, n=6) 65.9% of respondents have English as their first spoken language (n=29) Just under 5% of respondents consider themselves to have a disability (4.5%, n=2) Just under 70% of respondents were between 15-18 years old (68.3%, n=28) Nearly 60% of respondents told us they are studying at School (59.1%, n=26). Over 60% of respondents reported having either a High School Diploma or No Formal Qualifications (n=10, 28.6% and 34.3%, n=12 respectively) Just under 10% of respondents who use OSC textbooks receive any form of financial aid (9.5%, n=4)

Educator Sample 82 Respondents in total (have used/use OSC textbooks) 75.3% Male (n=61) and 24.7% Female (n=20) Over 85% of respondents live in the United States (86.4%, n=70) Other respondents live in Canada, Singapore, Malaysia & Australia (13.6%, n=11) 90.0% of respondents have English as their first spoken language (n=72) Over 90% of respondents have a Masters degree and/or PhD (38.3%, n=31, 54.3%, n=44 respectively) Over 50% of respondents work in a HE/University context (57.3%, n=47) 78.0% of respondents do F/T face-to-face teaching (n=64) Nearly 70% of respondents have been teaching for more than 10 years (67.5%, n=54)

OER Behaviours

Student Overview Over a quarter of respondents studied their subject via OER before joining their course (26.8%, n=11). 17.5% of respondents told us that their use of OER influenced their decision to register for their current course of study (n=7) Just under a third of respondents told us that they have adapted OER to fit their needs (31.0%, n=13) The biggest challenge for respondents when using OER was knowing where to find resources (61.0%, n=25)

Student Overview 75.0% of respondents first became aware of OpenStax College Textbooks via their instructor/course (n=30). 54.5% of respondents currently use the Biology textbook for their studies (n=24). 55.0% of students told us OpenStax College textbooks were required reading and they did not purchase any other textbooks for their course (n=22)

Educator Overview 92.6% of respondents have adapted OER to fit their needs Just under 50% of respondents have created OER for study or teaching (48.1%, n=39) The biggest challenge for respondents when using OER was finding resources of a sufficiently high quality (67.5%, n=54)* Top three types of OER used for teaching/training by respondents: open textbooks (98.9%, n=81), videos (78.0%, n=64) and images (72.0%, n=59) Almost 60% of respondents think their institution benefits financially by using OER (59.3%, n=48)

Educator Overview 34.6% of respondents first became aware of OpenStax College Textbooks via an internet search (n=28). Over a quarter of respondents became aware of OSC via a colleague or other recommendation (25.9%, n=21) 61.0% of respondents currently use the College Physics textbook with their students (n=50). Over 80% of respondents teach Science (82.7%, n=67) with 77.8% using OER in this area (n=63) and over a third creating OER in this subject (34.6%, n=28)

Student Perspectives

In which of these ways (if any) has your use of Open Educational Resources made an impact on your formal studies? (Check all that apply) "Using Open Educational Resources (e.g. OpenStax College textbooks) has led to my...” Increased interest in the subjects taught (69.0%, n=29) Increased satisfaction with the learning experience (66.7%, n=28) Grades improving (47.6%, n=20) Increased participation in classroom discussions (45.2%, n=19) Having increased independence and self-reliance (45.2%, n=19) Increased engagement with lesson content (45.2%, n=19) Increased experimentation with new ways of learning (45.2%, n=19) Gaining confidence (38.1%, n=16) Increased enthusiasm for future study (38.1%, n=16) Increased collaboration with my peers (35.7%, n=15) Being more likely to complete my course of study (33.3%, n=14) Becoming interested in a wider range of subjects than before I used these resources (31.0%, n=13)

In what ways, if any, has using OpenStax College textbooks impacted on your studies? Range of responses, including comments on: 40.6% Better Study Experience (n=13) 31.3% Accessibility of Resources (n=10) 12.5% Cost savings (n=4) 9.4% Improved Grades (n=3) 9.4% Less Weight to Carry (n=3)

“Increased my Ap Biology grade dramatically.” In what ways, if any, has using OpenStax College textbooks impacted on your studies? “Made learning easier and more accessible, helped and broadened my understanding and learning, made me WANT to study, helped me become interested in Biology because of the easy layout and presentation.” “It has allowed for me to develop knowledge easily in areas that I thought would be difficult to learn in due to the inability to buy an in-depth textbook.” “Open Stax textbooks offer me peace of mind and permit me to study in a more relaxed way. I am grateful not to have to come up with hundreds of dollars per class for books. “Increased my Ap Biology grade dramatically.” (Earlier this respondent reported: “I went from being horrible in AP Biology to actually reading these and went from a D 66% up to a A 90% So far.”) “Saved my back, I already have so many other books in my backpack and using my computer for more info saved additional stress that would have been on my back.”

Educator Perspectives

Based on your experiences as a teacher, to what extent do you agree with the following statements? % of educators who strongly agree or agree with the following statements: "Use of Open Educational Resources (e.g. OpenStax College textbooks) in the classroom...” Allows me to better accommodate diverse learners’ needs (67.9%, n=53) Increases learners’ satisfaction with the learning experience (65.8%, n=52) Increases learners’ experimentation with new ways of learning (64.1%, n=50) Increases learners’ engagement with lesson content (60.3%, n=47) Increases learners’ interest in the subjects taught (55.7%, n=44) Develops learners’ increased independence and self-reliance (55.7%, n=44) Increases learners’ participation in class discussions (49.4%, n=39) Leads to improved students’ grades (47.4%, n=37) Increases learners’ enthusiasm for future study (45.6%, n=36) Builds learners’ confidence (45.5%, n=35) Increases collaboration and/or peer-support among learners (44.2%, n=34) Leads to learners becoming interested in a wider range of subjects than before they used OER (41.0%, n=32)

In what ways, if any, has using OpenStax College textbooks impacted on your own teaching practice? “I have become more intentional about tailoring my course to my students, having been at least somewhat released from the constraints imposed by the commercial publishing industry” “I am able to think a little more creatively of what new experiences I can design for my students using resources in the text or other online resources. I think the text is a great foundation, but I don’t feel constrained to shy away from employing other teaching tools for fear they might ask: “Well, if you were going to have us watch this video or do this online tutorial – why did we have to pay $250 for the textbook?” “I have always believed in using a lot of additional resources. OpenStax works nicely in this regard. The supplementals are not fancy but we have made additional testbanks and other free resources available to all our faculty in a collective approach to compile very nice additional resources.” “I create more collections of images to build presentation, rather than going straight from the book.”

“…the students are sharing notes and annotations online.” In what ways, if any, has using OpenStax College textbooks impacted on your students? “Increased reading of the textbook, and a portable textbook that can go with them for review. Unlike other e-textbooks, this one can stay with them.” “They are able to access the textbook and start doing homework immediately rather than being delayed until weeks after the start of the course due to lack of finances.” “Ease of use for students and parents. The 'note-taking" type of students can print pages, and when I find students have a specific difficulty I can even print a page to use in class. Faster students can read on devices at any time - they enjoy viewing the book from smartphones etc.” “They have the book as a resource after leaving class. With another text they would most likely sell it back. Also sharing content with others outside of class.” “…the students are sharing notes and annotations online.” “The resource gave them a sound understanding of the topic and they were able to do self-directed study from it. The result is that seniors have excelled in their exams.”

In what ways, if any, has using OpenStax College textbooks impacted on your students? “It saves all 800 students per year about $120. One student told me that she had to work 15 hours for each book that she has to buy. That really put it in perspective.” “Students seem somewhat more engaged.” “Free and available online (while they work, or are at home) make OpenStax texts a great excuse-defying tool. In a positive light- students have access to the text wherever and whenever they need/want it.” “more student engagement in lab activities and lecture presentation” “…Students love having multiple options for textbook format (epub, iBook, pdf, or hardcopy). Enrollment in my class has increased by 25% - can't say for sure it is due to the textbook, but it certainly has helped. …” “I think that it is highly beneficial to have a brand new text to use, I would have been forced through budgetary constraints to purchase other texts which are 5-10 years old..”

As a result of using OpenStax College textbooks… 96.3% of educators who use OSC textbooks are more likely to recommend OpenStax College textbooks to fellow educators/teachers (n=77) 81.0% of respondents who use OSC textbooks are more likely to discuss using OpenStax College textbooks with their institution’s administrators (n=64) Over three quarters of respondents who use OSC textbooks are more likely to use OER for teaching (77.9%, n=60)

Financial Savings

Student Financial Savings By using OpenStax College textbooks each student surveyed saved on average ca. $155 (n=19)

Summary

Summary of Results Efficacy: Are they Learning? Over 60% of both students and educators report increased satisfaction with their own or their students’ learning experience as a result of using OER/OSC textbooks Almost 70% of students surveyed told us that they thought use of OER/OSC textbooks had led to an increased interest in the subjects taught Over 45% of students thought that their grades had improved due to using OER/OSC textbooks Emerging evidence for different ways in which “open” is making a specific difference: students can both access materials earlier (e.g. pre-course/pre- payment of financial aid) but can also continue to access/use resources post- course (don’t have to sell textbook/limited access). Positive impact on learning?

Next Steps

Next Steps Further analyses of data: e.g. comparison of adopters with newsletter respondents (educators), non-users/users of OSC textbooks, students studying in different contexts, different OSC textbooks etc. Comparison of data with other open textbook surveys Disseminating results (Impact Map inc. anonymised data) Future/current work focused on building case studies and comparative impact data We need you! Looking for YOUR best examples of impact... Help us build the most comprehensive picture of the impact of OER by contributing your evidence to the Impact Map

Thanks for listening. oerresearchhub. org chaos. open. ac Thanks for listening! oerresearchhub.org chaos.open.ac.uk Twitter: @OER_Hub @BeckPitt