Jay Summet CS 1 with Robots IPRE Evaluation – Data Collection Overview

2 IPRE Evaluation Data Collection Procedure - Outline Purpose of Data Collection Data that will be collected Demographic Information Attitudes & Beliefs CS1 Knowledge Success Rate Data collection procedure Participant Protection

3 Purpose of Data Collection CS 1 with Robots is an innovative and new context for CS 1 education. We are evaluating how it compares to more traditional methods of teaching CS 1. By collecting data about both students in CS 1 with Robots and traditional CS 1 classes we can answer key questions about this new curriculum.

4 Key Questions to Answer Are students who take CS1 with Robots more likely to continue on to CS 2? Major in Computer Science? Do students enjoy the class more because of the use of Personal Robots? Are they more motivated to do well because of the Robots? Does CS1 with Robots prepare students for the next class (CS2) in the Computer Science curriculum? Are students more successful in this class than in a traditional CS 1 class? Do students learn the same concepts as in a traditional CS 1 class?

5 Data Collected In class: First Week – Survey: Demographic Information Attitudes and Beliefs (about CS/Programming)‏ Last Week – 1 survey and 1 practice test: Attitudes and Beliefs (about CS/Programming)‏ CS 1 Knowledge (as a practice test)‏ Class Success rate (Pass vs. Drop/Withdraw/Failure) After class (over the next 2 years)‏ How many students continue on to attend CS 2 Success rate in CS 2 (if they continued)‏ Choice of Major (Selection/Retention)‏

6 Data Collected – Demographic Information Demographic Information: Gender, Ethnicity, Year in school Math / Science background CS background / Programming experience Major Selected (if any)‏ Used to ensure that students in CS1 with robots classes match control groups in traditional CS1 classes. Used to determine effects that are based upon student backgrounds, and not on the CS1 with Robots curriculum.

7 Data Collected – Attitudes and Beliefs Attitudes & Beliefs survey: Expected grade in the CS1 class Computer Confidence Problem Solving Ability Comfort with Technology Computer Science == Programming? Beginning of class survey compared to end of class survey to identify changes.

8 Data Collected – CS1 Knowledge & Success Rate Optional in class “review” test on the last week of class, does not affect student grade. Tests “generic” (non-robot) CS 1 knowledge Standard programming concepts General problem solving “Passing” students (D or higher grade) vs. “Unsuccessful” (failed, dropped, or withdrew from the course)‏ Results compared to control classes to measure effectiveness of CS1-Robots curriculum / method.

9 Data Collected – Long Term Advancement Rate How many students enroll in the subsequent course (CS2) within 2 years. Passing Rate in CS2 class (if taken)‏ Recruitment/Retention in choice of Major Permission to collect this long term data is granted by the students on the consent forms they sign while in CS1. The data is collected from standard administrative records, and requires no further interaction with the students.

10 Data Collection Procedure – In Class - First Week After the instructor leaves the room, a researcher: Reads a script that: Asks for student participation. Explains why we are collecting the data. Explains that participation is optional. Answers any student questions. Distributes Consent Forms and Surveys to the students who wish to participate. Collects Surveys and Consent forms when students are finished. Dismisses the class (or brings the instructor back into the room).

11 Data Collection Procedure – In Class - Last Week Exactly like the first week procedure, except the researcher distributes: Attitudes and Beliefs Survey Practice Test Together these take approximately 45 minutes to complete. Although the researcher will give all students a new consent form, students who have already completed one in the first week do not have to complete a second.

12 Data Collection Procedure – Long Term Long term data is collected for students who have given permission on their consent forms. A list of students who gave permission to have their long term progress monitored will be cross checked each semester against: CS2 enrollment Grade in CS2 class Choice of major Data will only be recorded and reported in the aggregate, and will not be linked to student names.

13 Participant Protection Students are Human Subjects, and they must be protected. All data collection & surveys are optional, and require informed student consent (as documented by their signature on the consent form). Instructors and TA's who are in charge of issuing grades or giving lectures to the students are NOT allowed to administer surveys or see the data until after the course is complete. Individuals (researchers) who administer surveys and collect long term data MUST take a (web-based) course to obtain certification of training on the protection of human subjects.

14 Avoiding Coercion: What Instructors/TA's must avoid An instructor and/or TA who is in charge of issuing grades for a class is in a “position of power” and may not conduct surveys or view collected data (until after the class is over and grades are issued)‏ A researcher must not be in a “position of power” with respect to the students they are asking to participate in the study. (An ideal researcher is an instructor or teaching assistant from a different class that the students do not know.)‏ This prevents students from feeling coerced to participate in the study.

15 What is Allowed: A class instructor is allowed to recruit and train a researcher, and can serve as a researcher for a different class that they do not teach. For example: Two instructors who are teaching CS1 with Robots and a traditional CS1 can act as researchers for each other's class. If they have human subjects training, instructors are allowed to view data collected from their class after grades have been issued and they are no longer in a “position of power”.

16 A Tricky Topic: Extra Credit In some situations it is allowable for an instructor to offer students incentives such as extra credit to complete a survey or participate in the study. HOWEVER: To avoid coercion, the instructor must offer the students an alternative method of obtaining the extra credit that does not force them to participate. The effort to complete the alternative method should be comparable to the effort needed to participate in the study. For Example: To obtain 5 points of extra credit, the students may either participate in the study, or write a one page (ungraded) paper about a famous robot.

17 Data and Privacy Protection Participant data should be protected from accidental disclosure. Papers must be stored in a locked filing cabinet or drawer. Electronic records must be stored in a password-protected account. Consent forms (which have students names) should be kept separate from anonymous surveys. Long term data is stored and reported only in the aggregate (e.g. 23 of 39 students continued on to CS 2 within a 2 year period).