1. De Anza Math Pilot Results, 7-9-07 1 Evaluating a NonRandomized Trial: A Case Study of a Pilot to Increase Pre- Collegiate Math Course Success Rates Andrew LaManque, PhD Supervisor, Institutional Research De Anza College

2. De Anza Math Pilot Results, 7-9-072 Overview of the Pilot Purpose of the Evaluation Results from Quantitative Analysis Demographics of Students End of Class Survey Course Success Findings Outline

3. De Anza Math Pilot Results, 7-9-073 Overview of the Pilot A Partnership was formed to increase the success rates in Pre-Collegiate math courses Components: Math and Counseling faculty EnableMath – Course Software Noel Levitz – College Student Inventory Using technology, provide students timely information on their progress as well as information about their motivations

4. De Anza Math Pilot Results, 7-9-074 Overview of the Pilot Developmental math sequence consists of pre, elementary, and intermediate algebra Offer 175 class sections per year of developmental math (37% of all math department courses) Historically, a 55% success rate in developmental math courses 31 full-time instructors, 45 part-time instructors Class size is 40

5. De Anza Math Pilot Results, 7-9-075 Program Evaluation Given that the pilot program provides an additional learning choice for students, the question from the Dean is: “do the results demonstrate that pilot students do at least as well as other students” Using multiple forms of evaluation (e.g. student grades, student surveys) provides feedback to instructors as well as administration on implementation

6. De Anza Math Pilot Results, 7-9-076 Limitations Not a randomized trial – possible instructor (although, 12 different instructors participated) and student self selection bias No independent measure of success More research is needed to determine the relative effect of each intervention Program Evaluation

7. De Anza Math Pilot Results, 7-9-077 Results: Demographics of Students Twice the % of Latino students as college, similar to non-pilot sections

8. De Anza Math Pilot Results, 7-9-078 Results: Demographics of Students Many first-generation college students

9. De Anza Math Pilot Results, 7-9-079 Similar with regard to Gender, Initial Education Goal and Ethnicity although Math114 EnableMath had a higher percentage of minority students (54%) compared with the other sections (34%). EnableMath students are slightly older on average (45% vs 37% born prior to 1987). Results: Demographics of Students

10. De Anza Math Pilot Results, 7-9-0710 Results: End of Class Survey

11. De Anza Math Pilot Results, 7-9-0711 Results: End of Class Survey 77% Agree or Strongly Agree

12. De Anza Math Pilot Results, 7-9-0712 Results: End of Class Survey 61% Agree or Strongly Agree

13. De Anza Math Pilot Results, 7-9-0713 Results: Course Success Approx 1 of 5 students CourseGroup# SecHC * Pre-AlgebraPilot14428 Other551,755 Elem AlgebraPilot21723 Other883,045 Interm AlgebraPilot27853 Other1123,573 TotalPilot622,004 Other2558,373 * HC for the Pilot only includes students completing the CSI. De Anza College Math Pilot Sections / Students, Fall 05 - Spr 07

14. De Anza Math Pilot Results, 7-9-0714 Results: First Course Success CourseGroupGradesPercentGradesPercentGradesPercentGradesPercent Pre Algebra Pilot30872%7618%4410%428100% Other1,01258%39122%35220%1,755100% Elementary Algebra Pilot51471%13819%7110%723100% Other1,66055%66422%72124%3,045100% Intermediate Algebra Pilot64175%14016%728%853100% Other1,97755%67519%92126%3,573100% Includes end of term grades of students enrolled at census (3rd week). Note: Students in the Pilot group used the EnableMath Software and completed the CSI. Did Not PassWithdrewTotalPass Course Success of Fall 2005-Spring 2007 EnableMath / Noel Levitz Sections Compared to All Other Sections, De Anza College

15. De Anza Math Pilot Results, 7-9-0715 Results: Second Course Success First Course CourseGroupGrades% RowGrades% RowGrades% RowGrades% Row Pre to Elementary Algebra Pilot9445%5928%5727%210100% Other30555%10619%14826%559100% Elementary to Intermediate Algebra Pilot16864%4517%4919%262100% Other54566%13717%14618%828100% Intermediate Algebra to College Pilot8550%3621%4828%169100% Other30156%9718%14226%540100% Note: Considers only the first attempt of both courses. PassDid Not PassWTotal De Anza College EnableMath Pilot, Fall 2005-Winter2007 Second Course Pass Rate for Students Successful in First Course

16. De Anza Math Pilot Results, 7-9-0716 Results: Second Course Success

17. De Anza Math Pilot Results, 7-9-0717 Findings Students currently in the project want to take next class in sequence These classes are filled before other math classes Participating instructors & counselors want to continue More faculty are wanting to participate

18. De Anza Math Pilot Results, 7-9-0718 Findings No increased costs – taking existing resources and using them differently Regular class period / time Counseling time in the class instead of counseling center

19. De Anza Math Pilot Results, 7-9-0719 Findings Overall, the data suggests that there is value in offering a different approach to students, but claims of improved learning, especially the long term retention of material, can not be substantiated. Difficult to evaluate a non-randomized trial Math department might focus on improving the transition in pilot sections from Pre- Algebra to Elementary Algebra

20. De Anza Math Pilot Results, 7-9-0720 Questions For a copy of this presentation please email Andrew LaManque at: lamanqueandrew@fhda.edu