De Anza Math Pilot Results, Evaluating a NonRandomized Trial: A Case Study of a Pilot to Increase Pre- Collegiate Math Course Success Rates Andrew LaManque, PhD Supervisor, Institutional Research De Anza College

De Anza Math Pilot Results, Overview of the Pilot Purpose of the Evaluation Results from Quantitative Analysis Demographics of Students End of Class Survey Course Success Findings Outline

De Anza Math Pilot Results, Overview of the Pilot A Partnership was formed to increase the success rates in Pre-Collegiate math courses Components: Math and Counseling faculty EnableMath – Course Software Noel Levitz – College Student Inventory Using technology, provide students timely information on their progress as well as information about their motivations

De Anza Math Pilot Results, Overview of the Pilot Developmental math sequence consists of pre, elementary, and intermediate algebra Offer 175 class sections per year of developmental math (37% of all math department courses) Historically, a 55% success rate in developmental math courses 31 full-time instructors, 45 part-time instructors Class size is 40

De Anza Math Pilot Results, Program Evaluation Given that the pilot program provides an additional learning choice for students, the question from the Dean is: “do the results demonstrate that pilot students do at least as well as other students” Using multiple forms of evaluation (e.g. student grades, student surveys) provides feedback to instructors as well as administration on implementation

De Anza Math Pilot Results, Limitations Not a randomized trial – possible instructor (although, 12 different instructors participated) and student self selection bias No independent measure of success More research is needed to determine the relative effect of each intervention Program Evaluation

De Anza Math Pilot Results, Results: Demographics of Students Twice the % of Latino students as college, similar to non-pilot sections

De Anza Math Pilot Results, Results: Demographics of Students Many first-generation college students

De Anza Math Pilot Results, Similar with regard to Gender, Initial Education Goal and Ethnicity although Math114 EnableMath had a higher percentage of minority students (54%) compared with the other sections (34%). EnableMath students are slightly older on average (45% vs 37% born prior to 1987). Results: Demographics of Students

De Anza Math Pilot Results, Results: End of Class Survey

De Anza Math Pilot Results, Results: End of Class Survey 77% Agree or Strongly Agree

De Anza Math Pilot Results, Results: End of Class Survey 61% Agree or Strongly Agree

De Anza Math Pilot Results, Results: Course Success Approx 1 of 5 students CourseGroup# SecHC * Pre-AlgebraPilot14428 Other551,755 Elem AlgebraPilot21723 Other883,045 Interm AlgebraPilot27853 Other1123,573 TotalPilot622,004 Other2558,373 * HC for the Pilot only includes students completing the CSI. De Anza College Math Pilot Sections / Students, Fall 05 - Spr 07

De Anza Math Pilot Results, Results: First Course Success CourseGroupGradesPercentGradesPercentGradesPercentGradesPercent Pre Algebra Pilot30872%7618%4410%428100% Other1,01258%39122%35220%1,755100% Elementary Algebra Pilot51471%13819%7110%723100% Other1,66055%66422%72124%3,045100% Intermediate Algebra Pilot64175%14016%728%853100% Other1,97755%67519%92126%3,573100% Includes end of term grades of students enrolled at census (3rd week). Note: Students in the Pilot group used the EnableMath Software and completed the CSI. Did Not PassWithdrewTotalPass Course Success of Fall 2005-Spring 2007 EnableMath / Noel Levitz Sections Compared to All Other Sections, De Anza College

De Anza Math Pilot Results, Results: Second Course Success First Course CourseGroupGrades% RowGrades% RowGrades% RowGrades% Row Pre to Elementary Algebra Pilot9445%5928%5727%210100% Other30555%10619%14826%559100% Elementary to Intermediate Algebra Pilot16864%4517%4919%262100% Other54566%13717%14618%828100% Intermediate Algebra to College Pilot8550%3621%4828%169100% Other30156%9718%14226%540100% Note: Considers only the first attempt of both courses. PassDid Not PassWTotal De Anza College EnableMath Pilot, Fall 2005-Winter2007 Second Course Pass Rate for Students Successful in First Course

De Anza Math Pilot Results, Results: Second Course Success

De Anza Math Pilot Results, Findings Students currently in the project want to take next class in sequence These classes are filled before other math classes Participating instructors & counselors want to continue More faculty are wanting to participate

De Anza Math Pilot Results, Findings No increased costs – taking existing resources and using them differently Regular class period / time Counseling time in the class instead of counseling center

De Anza Math Pilot Results, Findings Overall, the data suggests that there is value in offering a different approach to students, but claims of improved learning, especially the long term retention of material, can not be substantiated. Difficult to evaluate a non-randomized trial Math department might focus on improving the transition in pilot sections from Pre- Algebra to Elementary Algebra

De Anza Math Pilot Results, Questions For a copy of this presentation please Andrew LaManque at: