1. Part 2: Evaluating your program  You can download this presentation at:  http://faculty.smcm.edu/acjohnson/PREP/  You can download this presentation at:  http://faculty.smcm.edu/acjohnson/PREP/

2. Evaluating your program  What are your objectives for your program?  What data would let you know you’re meeting those objectives?  What data would convince your administration to keep funding the program?  What are your objectives for your program?  What data would let you know you’re meeting those objectives?  What data would convince your administration to keep funding the program?

3. Evaluating your program  Informal assessments: To let you know that the program is working; to fine- tune it as you go along  Observations of student progress, conversations with students, informal surveys  Informal assessments: To let you know that the program is working; to fine- tune it as you go along  Observations of student progress, conversations with students, informal surveys

4. Evaluating your program  Formal assessments  Of the first year  Longer-term  Formal assessments  Of the first year  Longer-term

5. Evaluating your program  The basics:  Comparison groups  Independent variables  Dependent variables  The basics:  Comparison groups  Independent variables  Dependent variables

6. Comparison groups  Historical: ESP participants vs similar students before ESP  Comparable: ESP participants vs similar students not in ESP  To the norm: ESP participants vs all non-participants  To decliners: People who rejected an invitation to ESP  Historical: ESP participants vs similar students before ESP  Comparable: ESP participants vs similar students not in ESP  To the norm: ESP participants vs all non-participants  To decliners: People who rejected an invitation to ESP

7. Independent variables  ESP participation  Race  Gender  Academic preparation (SAT scores; CCI pre-test)  Financial need  Motivation  ESP participation  Race  Gender  Academic preparation (SAT scores; CCI pre-test)  Financial need  Motivation

8. Dependent variables  CCI post-test scores  CCI growth scores  Calc grades  Raw data, % receiving A or B, % failing  Enrollment/grades in Calc II  Declaring SEM major  Graduating at all  Graduating with SEM major  CCI post-test scores  CCI growth scores  Calc grades  Raw data, % receiving A or B, % failing  Enrollment/grades in Calc II  Declaring SEM major  Graduating at all  Graduating with SEM major

9. Analyzing the data  Descriptive statistics: Simply compare the performances of the relevant groups  Are differences in grades or scores significant? Independent-samples t-tests  Are differences in percent of students doing something (getting As & Bs, graduating) significant? Chi-square  Descriptive statistics: Simply compare the performances of the relevant groups  Are differences in grades or scores significant? Independent-samples t-tests  Are differences in percent of students doing something (getting As & Bs, graduating) significant? Chi-square

10. Analyzing the data  Controlling for preparation: Divide data into groups according to some measure of preparation

11. Analyzing the data  Controlling for preparation: Construct a regression equation using all your available independent variables; see whether ESP participation is a significant predictor of the dependent variable of interest  For continuous dependent variable: OLS; for binary: Logistic regression  Controlling for preparation: Construct a regression equation using all your available independent variables; see whether ESP participation is a significant predictor of the dependent variable of interest  For continuous dependent variable: OLS; for binary: Logistic regression

12. Calculus concept inventory  Pros  This is the gold standard--did the students learn calculus? Did they learn more than other students?  This approach--at least the descriptive stats--can be used for small n  Cons  Limited number of test items--test might not be reliable or valid enough for your comfort  Requires access to all calculus students, not just ESP students  Pros  This is the gold standard--did the students learn calculus? Did they learn more than other students?  This approach--at least the descriptive stats--can be used for small n  Cons  Limited number of test items--test might not be reliable or valid enough for your comfort  Requires access to all calculus students, not just ESP students

13. Calc grades, SAT & GPA data  Pros:  Lets you control for preparation  Administrators like statistical analyses  Cons:  Someone has to like stats--might need SPSS  You have to find someone in institutional research to let you have the data  Requires a substantial n  Pros:  Lets you control for preparation  Administrators like statistical analyses  Cons:  Someone has to like stats--might need SPSS  You have to find someone in institutional research to let you have the data  Requires a substantial n

14. Examples  Fullilove & Treisman, 1990  Comparison groups:  Historical--pre-MWP African Americans  African American accepters & decliners  Preparation measures:  Special admission? Math SAT scores  Dependent variables:  Calc performance, graduation  Fullilove & Treisman, 1990  Comparison groups:  Historical--pre-MWP African Americans  African American accepters & decliners  Preparation measures:  Special admission? Math SAT scores  Dependent variables:  Calc performance, graduation

15. Examples  Johnson, 2007a  Comparison groups (all with 1st major in science):  White/Asian; Black/Latino/American Indian  Independent variables:  Financial need, predicted GPA  Dependent variables: Graduation with science/math major, grad GPA  Johnson, 2007a  Comparison groups (all with 1st major in science):  White/Asian; Black/Latino/American Indian  Independent variables:  Financial need, predicted GPA  Dependent variables: Graduation with science/math major, grad GPA

16. Expanding your program  Evidence that matriculation-to- graduation programs produce even bigger benefits:  Johnson (2007a)  Maton, Hrabowski & Schmitt (2000)  Maton & Hrabowski (2004)  Gándara (1999)  Evidence that matriculation-to- graduation programs produce even bigger benefits:  Johnson (2007a)  Maton, Hrabowski & Schmitt (2000)  Maton & Hrabowski (2004)  Gándara (1999)

17. Evaluating your program G What are your objectives for your program? G What data would let you know you’re meeting those objectives? G What data would convince your administration to keep funding the program? Evaluating your program G Informal assessments: To let you know that the program is working; to fine-tune it as you go along G Observations of student progress, conversations with students, informal surveys Evaluating your program G Formal assessments G Of the first year G Longer-term Evaluating your program G The basics: G Comparison groups G Independent variables G Dependent variables Comparison groups G Historical: ESP participants vs similar students before ESP G Comparable: ESP participants vs similar students not in ESP G To the norm: ESP participants vs all non-participants G To decliners: People who rejected an invitation to ESP Independent variables G ESP participation G Race G Gender G Academic preparation (SAT scores; CCI pre-test) G Financial need G Motivation Dependent variables G CCI post-test scores G CCI growth scores G Calc grades G Raw data, % receiving A or B, % failing G Enrollment/grades in Calc II G Declaring SEM major G Graduating at all G Graduating with SEM major Analyzing the data G Descriptive statistics: Simply compare the performances of the relevant groups G Are differences in grades or scores significant? Independent-samples t-tests G Are differences in percent of students doing something (getting As & Bs, graduating) significant? Chi-square Analyzing the data G Controlling for preparation: Divide data into groups according to some measure of preparation Analyzing the data G Controlling for preparation: Construct a regression equation using all your available independent variables; see whether ESP participation is a significant predictor of the dependent variable of interest G For continuous dependent variable: OLS; for binary: Logistic regression Calculus concept inventory G Pros G This is the gold standard--did the students learn calculus? Did they learn more than other students? G This approach--at least the descriptive stats--can be used for small n G Cons G Limited number of test items--test might not be reliable or valid enough for your comfort G Requires access to all calculus students, not just ESP students Calc grades, SAT & GPA data G Pros: G Lets you control for preparation G Administrators like statistical analyses G Cons: G Someone has to like stats--might need SPSS G You have to find someone in institutional research to let you have the data G Requires a substantial n Examples G Fullilove & Treisman, 1990 G Comparison groups: G Historical--pre-MWP African Americans G African American accepters & decliners G Preparation measures: G Special admission? Math SAT scores G Dependent variables: G Calc performance, graduation Examples G Johnson, 2007a G Comparison groups (all with 1st major in science): G White/Asian; Black/Latino/American Indian G Independent variables: G Financial need, predicted GPA G Dependent variables: Graduation with science/math major, grad GPA Expanding your program G Evidence that matriculation-to-graduation programs produce even bigger benefits: G Johnson (2007a) G Maton, Hrabowski & Schmitt (2000) G Maton & Hrabowski (2004) G Gándara (1999)