1. GATEWAY INITIATIVE Hillsborough Community College Fall 2007 Preliminary Results A Formative Evaluation

2. Formative Analysis Fall 2007 Results Preliminary Formative Analysis Fall 2007 Results To provide feedback to participating faculty as quickly as (Got Class, p. 40). To provide feedback to participating faculty as quickly as possible (Got Class, p. 40). To answer initial questions at the pilot stage: To answer initial questions at the pilot stage: Are things working the way we expected? Are things working the way we expected? What do the initial outcomes suggest? What do the initial outcomes suggest? Are there any surprises? Are there any surprises? What can we learn that will improve the process? What can we learn that will improve the process?

3. FALL 2007 INTERVENTIONS - TWO PILOT COURSES: INTERVENTIONS - TWO PILOT COURSES: MAC-1105 (College Algebra) MAC-1105 (College Algebra) Cumulative Homework Cumulative Homework In-class Group Review In-class Group Review PSY-2012 (General Psychology) PSY-2012 (General Psychology) Team-Teaching Team-Teaching Use of Clickers (Electronic Student Response System) in the classroom Use of Clickers (Electronic Student Response System) in the classroom BASELINE DATA – THREE COURSES: BASELINE DATA – THREE COURSES: ACG-2021 (Financial Accounting) ACG-2021 (Financial Accounting) CGS-1000 (Intro. To Computers & Technology) CGS-1000 (Intro. To Computers & Technology) ECO-2021 (Principles of Microeconomics) ECO-2021 (Principles of Microeconomics)

4. Data Collection Strategy Pretest and Posttest Pretest and Posttest Questions addressed identified learning outcomes Questions addressed identified learning outcomes Three scores for each student: Three scores for each student: Pretest Score Pretest Score Posttest Score Posttest Score Gain Score (Posttest Score – Pretest Score) Gain Score (Posttest Score – Pretest Score) Students’ Grades in the Course were Not Affected Students’ Grades in the Course were Not Affected Surveys of Student Characteristics Surveys of Student Characteristics Descriptive and Demographic Information Descriptive and Demographic Information Students’ Perceptions Students’ Perceptions

5. RESULTS OF ANALYSES ANOVA/ANCOVA ANOVA/ANCOVA Compared Gain Scores for Experimental and Control Groups in each intervention. Compared Gain Scores for Experimental and Control Groups in each intervention. No Significant Differences between Experimental and Control Groups No Significant Differences between Experimental and Control Groups MAC-1105 MAC-1105 PSY-2012 PSY-2012

6. PRETEST/POSTTEST COMPARISONS MAC-1105

7. PRETEST/POSTTEST COMPARISONS PSY-2012

8. SURPRISE!!! Distributions of Gain Scores revealed some “negative gains.” Distributions of Gain Scores revealed some “negative gains.” GAIN SCORE = (Posttest Score – Pretest Score) GAIN SCORE = (Posttest Score – Pretest Score) These students scored higher on the pretest than on the posttest. These students scored higher on the pretest than on the posttest.

9. Things were not working the way we expected. Negative Gain Scores Negative Gain Scores Observed in both Control and Experimental sections of the pilot courses Observed in both Control and Experimental sections of the pilot courses (MAC-1105, PSY-2012). (MAC-1105, PSY-2012). Observed in all three of the courses collecting baseline data. Observed in all three of the courses collecting baseline data. (ACG-2021, CGS-1000, ECO-2023) (ACG-2021, CGS-1000, ECO-2023)

10. MAC-1105 GAIN SCOREFREQUENCYPERCENTCUMULATIVE FREQUENCY CUMULATIVE PERCENT -2020.482 -10143.39163.87 07317.688921.55 108219.8517141.4 207217.4324358.84 306014.5330373.37 405413.0835786.44 50266.338392.74 60143.3939796.13 7092.1840698.31 8040.9741099.27 9030.73413100

11. PSY-2012 GAIN SCORE FREQUENCYPERCENTCUMULATIVE FREQUENCY CUMULATIVE PERCENT -4510.341 -4031.0341.37 -3520.6962.06 -3010.3472.41 -2510.3482.75 -2062.06144.81 -1541.37186.19 -1031.03217.22 -5165.53712.71 0299.976622.68 53512.0310134.71 103411.6813546.39 153612.3717158.76 203612.3720771.13 254415.1225186.25 ::::: 8510.34291100

12. ACG-2021 GAINSCOREFrequencyPercentCumulative FrequencyPercent -3510.441 -2031.3241.75 -1531.3273.07 -1052.19125.26 -531.32156.58 0114.822611.4 5177.464318.86 102410.536729.39 15208.778738.16 20229.6510947.81 252611.413559.21 302812.2816371.49 35219.2118480.7 ::::: 6520.88228100

13. CGS-1000 GAIN SCOREFREQUENCYPERCENTCUMULATIVE FREQUENCYCUMULATIVE PERCENT -3220.622 -2820.6241.23 -2420.6261.85 -2061.85123.7 -1682.47206.17 -12134.013310.19 -8154.634814.81 -4206.176820.99 03310.1910131.17 44814.8114945.99 83912.0418858.02 123811.7322669.75 163611.1126280.86 20257.7228788.58 4030.9332399.69 4810.31324100

14. ECO-2023 GAIN SCOREFREQUENCYPERCENTCUMULATIVE FREQUENCY CUMULATIVE PERCENT -3010.571 -2510.5721.14 -2031.7152.86 -1552.86105.71 -1031.71137.43 -5137.432614.86 0137.433922.29 5105.714928 10116.296034.29 1521128146.29 2095.149051.43 25179.7110761.14 30105.7111766.86 3514813174.86 8021.14175100

15. Because it did not affect their final grades, students were not taking the posttest seriously. Because it did not affect their final grades, students were not taking the posttest seriously. Tired of being tested Tired of being tested Ready for Winter Break Ready for Winter Break No incentive to do their best No incentive to do their best Marking answers at random (all 1s, all 5s, patterns) Marking answers at random (all 1s, all 5s, patterns) Leaving answers blank Leaving answers blank What Went Wrong?

16. Scheirer (1994) recommends using formative evaluation in a pilot situation to collect information on the feasibility of activities and their acceptance by recipients. Posttest – Not Feasible, Not Accepted Posttest – Not Feasible, Not Accepted Negative gain scores – evidence that the posttest is not accurately assessing students’ KSAs. Negative gain scores – evidence that the posttest is not accurately assessing students’ KSAs. Not a valid measure. Not a valid measure. No conclusions can be based on these results. No conclusions can be based on these results. A valid measure must be found for use in the future. A valid measure must be found for use in the future. Scheirer, M. A. (1994). Designing and using process evaluation. In J. S. Wholey, H. Hatry and K. Newcomer (Eds.), Handbook of Practical Program Evaluation (pp. 40-68). San Francisco : Josey Bass. What Was Learned

17. How Will We Improve the Process? Lead Faculty Acted Quickly and Decisively Lead Faculty Acted Quickly and Decisively MAC-1105 MAC-1105 Posttest questions imbedded in the final exam Posttest questions imbedded in the final exam PSY-2012 PSY-2012 Performance on posttest worth extra credit Performance on posttest worth extra credit Implemented in Time for Spring Posttests Implemented in Time for Spring Posttests

18. EXPECTED RESULTS Tying the posttest to the final grade appears to have corrected the problem. Tying the posttest to the final grade appears to have corrected the problem. Lead faculty grading SP08 posttests report seeing a difference. Lead faculty grading SP08 posttests report seeing a difference. We are expecting meaningful results from the revised posttest procedure as we conclude our first year of the study. We are expecting meaningful results from the revised posttest procedure as we conclude our first year of the study.

19. ABOUT OUR STUDENTS

20. STUDENTS’ TERM CREDIT LOAD CreditLoad:MAC-1105(N=760)PSY-2012(N=504)ACG-2021(N=453)CGS-1000(N=1313)ECO-2023(N=254) < 12 hrs 38%33%47%28%45% 12+ hrs 62%67%53%72%55%

21. STUDENTS RECEIVING SOME FORM OF FINANCIAL AID COURSE:YESNO MAC-1105 (N=760) 61.6%38.4% PSY-2012 (N=504) 59.4%40.6% ACG-2021 (N=453) 56.6%43.4% CGS-1000 (N=1313) 60.2%39.8% ECO-2023 (N=254) 54.6%45.4%

22. WORK HOURS per WEEK (self-report) WK-HRS:MAC-1105(N=222)PSY-2012(N=142)ACG-2021(N=216)CGS-1000(N=381)ECO-2023(N=129) 019.9%20.9%15.5%21.4%11.9% 1-106.0%3.7%2.6%5.9%3.4% 11-2015.3%15.5%10.7%17.9%11.5% 21-3027.9%29.7%24.0%25.0%23.8% 30+30.9%31.2%47.2%29.8%49.4%

23. “What were your biggest obstacles in learning (this subject)?” Perceived Obstacles to Success: MAC 1105 (n=464) PSY 2012 (n=347) ACG 2021 (n=303) CGS 1000 (n=429) ECO 2023 (n=218) Test Anxiety 37.9%35.2%20.5%18.9%22.0% Found Subject Difficult to Learn 22.4%9.5%23.1%20.3%33.5% Not Enough Time for Homework 26.1%11.5%9.2%21.2%7.3% Material Covered Too Quickly 24.1%13.8%14.8%8.6%n/a

24. “What additional resources would have helped you to be more successful ( in this class)?” COURSE: “IMPROVED STUDY SKILLS” “A STUDY SKILLS CLASS” MAC-1105 (N=464) 36.2%6.0% PSY-2012 (N=347) 42.9%7.8% ACG-2021 (N=303) 21.4%4.6% CGS-1000 (N=429) 23.1%8.6% ECO-2023 (N=218) 41.7%3.7%

25. Other Resources Identified “What additional resources would have helped...?” MAC 1105 (n=464) PSY 2012 (n=347) ACG 2021 (n=303) CGS 1000 (n=429)ECO 2023 (n=218) More group work with other students 20.7%19.6%4.0%19.4%20.6% More time in class15.1%14.7%11.6%12.8%10.1% More online tools10.6%9.8%10.9%11.0%9.2% More time one-on-one with my instructor 23.3%10.1%7.3%13.5%6.9%

26. “Which of the following helped you the most to succeed in (this class)?” COURSE: Instructor Homework/ Test Review* Working/Studying with Others MAC-1105 (n=464) 49.1%45.3%31.9% PSY-2012 (n=347) 64.8%43.2%*20.5% ACG-2021 (n=303) 60.4%47.5%15.8% ECO-2023 (n=218) 75.2%18.8%16.1%

27. PERCEIVED HELPFULNESS OF INTERVENTIONS EXPERIMENTAL SECTIONS ONLY MAC-1105 Group Test Review (n=150) Group Test Review (n=150) Helped the Most – 44.67% Helped the Most – 44.67% Other Resources that Helped – 48.67% Other Resources that Helped – 48.67% Cumulative Homework (n=98) Cumulative Homework (n=98) Helped the Most – 47.96% Helped the Most – 47.96% Other Resources that Helped – 60.20% Other Resources that Helped – 60.20%

28. PERCEIVED HELPFULNESS OF INTERVENTIONS EXPERIMENTAL SECTIONS ONLY Psy-2012 Clicker Technology (n= 80) Clicker Technology (n= 80) Helped the Most – 55.0% Helped the Most – 55.0% Other Resources that Helped – 58.75% Other Resources that Helped – 58.75% Team-Teaching (n=58 ) Team-Teaching (n=58 ) Helped the Most – 24.14 % Helped the Most – 24.14 % Other Resources that Helped – 20.69% Other Resources that Helped – 20.69%

29. LOOKING FORWARD Students’ feedback suggests that these interventions ought to be effective. Students’ feedback suggests that these interventions ought to be effective. Spring semester’s results should provide a more realistic picture of their impact on students’ learning outcomes. Spring semester’s results should provide a more realistic picture of their impact on students’ learning outcomes.