Part 2: Evaluating your program  You can download this presentation at:   You can download this presentation.

Slides:



Advertisements
Similar presentations
Data Collection Strategies An Overview. How Do We Assess ?
Advertisements

Logit & Probit Regression
Prediction with multiple variables Statistics for the Social Sciences Psychology 340 Spring 2010.
INFERENTIAL STATISTICS. Descriptive statistics is used simply to describe what's going on in the data. Inferential statistics helps us reach conclusions.
Who is Smarter? Academic or CTE Students? Jack Elliot Jim Knight The University of Arizona.
UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing Keeping Kids in School:
Interpreting published multivariate analyses (using logistic regression) Friday 11 th March 2011.
CORRELATIONAL RESEARCH II Relations in Observational and Survey Research Lawrence R. Gordon Psychology Research Methods I.
Ordinal Logistic Regression
Topics: Regression Simple Linear Regression: one dependent variable and one independent variable Multiple Regression: one dependent variable and two or.
Office of Research, Evaluation, and Assessment April 19, 2008.
Introduction to Probability and Statistics Linear Regression and Correlation.
Today Concepts underlying inferential statistics
Psyc 235: Introduction to Statistics
Inferential Statistics
Portfolio Assessment A collection of a student’s work specifically selected to tell a story about the student.
1 Predicting Success in Math: The relationship between high school courses and remedial math in college Brandon Lagerquist Northshore School District Paul.
Impact of fine arts on academic success A Comparison study EDRS 5305 Fall 2004 Dr. Teresa Cortez.
Statistics for the Social Sciences Psychology 340 Fall 2013 Tuesday, November 19 Chi-Squared Test of Independence.
Examination of Variables Related to NCLEX-RN Janis A. Franich, RN, BSN, CIC.
Scholarships on Naviance 11 th Grade Postsecondary Visits and Admissions #6.
The Learning Behaviors Scale
Does Formative Feedback Help or Hinder Students? An Empirical Investigation 2015 DEE Conference Carlos Cortinhas, University of Exeter.
The Impact of Including Predictors and Using Various Hierarchical Linear Models on Evaluating School Effectiveness in Mathematics Nicole Traxel & Cindy.
Evaluating a Research Report
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Group Quantitative Designs First, let us consider how one chooses a design. There is no easy formula for choice of design. The choice of a design should.
Gwynn Mettetal.  Discuss different ways to assess outcomes  Help you decide which methods would be best for your project.
Topics Appropriate to Experiments Projects with limited and well-defined concepts. Projects that are exploratory rather than descriptive. Studies of small.
Quantitative SOTL Research Methods Krista Trinder, College of Medicine Brad Wuetherick, GMCTE October 28, 2010.
Intro: “BASIC” STATS CPSY 501 Advanced stats requires successful completion of a first course in psych stats (a grade of C+ or above) as a prerequisite.
Logistic (regression) single and multiple. Overview  Defined: A model for predicting one variable from other variable(s).  Variables:IV(s) is continuous/categorical,
 In Chapter 10 we tested a parameter from a population represented by a sample against a known population ( ).  In chapter 11 we will test a parameter.
Louisiana Math and Science Partnership State Coordinator Meeting Washington, DC June 2008.
Classifying Designs of MSP Evaluations Lessons Learned and Recommendations Barbara E. Lovitts June 11, 2008.
MK346 – Undergraduate Dissertation Preparation Part II - Data Analysis and Significance Testing.
Using Common Core State Standards of Seventh Grade Mathematics in the Application of NXT LEGO® Robotics for CReSIS Middle School Students.
Voluntary Disclosure Not Covered in Textbook. You’re on a job interview and the interviewer knows what the distribution of GPAs are for MBA students at.
Chapter 3 should describe what will be done to answer the research question(s), describe how it will be done and justify the research design, and explain.
Experimental Research
W W W. C E S. C L E M S O N. E D U / G E / Planning Engineering Education Research Facilitator: Matthew W. Ohland.
Experimental Research Methods in Language Learning Chapter 5 Validity in Experimental Research.
Ch 9 Internal and External Validity. Validity  The quality of the instruments used in the research study  Will the reader believe what they are readying.
PREP workshop on Emerging Scholars Programs Washington, DC 18 July 2008 Angela Johnson Washington, DC 18 July 2008 Angela Johnson.
Experimentation in Computer Science (Part 2). Experimentation in Software Engineering --- Outline  Empirical Strategies  Measurement  Experiment Process.
Bonds and bridges: The relative importance of relations with peers and faculty for college student achievement Sandra Dika, PhD Assistant Research Professor.
A presentation of methods and selected results 1.
Gabriela Garcia John Briggs. Explore whether using an assessment instrument which measures non-cognitive attributes is a predictor of student success.
Chapter 10 Copyright © Allyn & Bacon 2008 This multimedia product and its contents are protected under copyright law. The following are prohibited by law:
CONFERENCE EVALUATION DATA ANALYSIS. DATA ANALYSIS  A credible amount of data has to be collected to allow for a substantial analysis  Information collected.
Comparing Two Means: One-sample & Paired-sample t-tests Lesson 13.
Experimental Research Methods in Language Learning Chapter 13 Paired-Samples and Independent- Samples T-tests.
PART 2 SPSS (the Statistical Package for the Social Sciences)
Center for Institutional Effectiveness LaMont Rouse, Ph.D. Fall 2015.
Impact of fine arts on academic success A Comparison study Deborah Jones & Debra Villalobos EDRS5305 Fall 2004 Dr. Teresa Cortez.
Educational Research Inferential Statistics Chapter th Chapter 12- 8th Gay and Airasian.
PSY 325 AID Education Expert/psy325aid.com FOR MORE CLASSES VISIT
Template provided by: “posters4research.com” Academic Performance and Persistence of Undergraduate Students at a Land-Grant Institution: A Statistical.
Experimental Design Ragu, Nickola, Marina, & Shannon.
Methods of Presenting and Interpreting Information Class 9.
Looking for statistical twins
How Can High School Counseling Shape Students’ Postsecondary Attendance? Exploring the Relationship between High School Counseling and Students’ Subsequent.
A Statistical Analysis Utilizing Detailed Institutional Data
Consequential Validity
Longitudinal Designs.
Analytics in Higher Education: Methods Overview
Propensity Score Matching Makes Program Evaluation Easy
Analysis based on normal distributions
Example on the Concept of Regression . observation
Presentation transcript:

Part 2: Evaluating your program  You can download this presentation at:   You can download this presentation at: 

Evaluating your program  What are your objectives for your program?  What data would let you know you’re meeting those objectives?  What data would convince your administration to keep funding the program?  What are your objectives for your program?  What data would let you know you’re meeting those objectives?  What data would convince your administration to keep funding the program?

Evaluating your program  Informal assessments: To let you know that the program is working; to fine- tune it as you go along  Observations of student progress, conversations with students, informal surveys  Informal assessments: To let you know that the program is working; to fine- tune it as you go along  Observations of student progress, conversations with students, informal surveys

Evaluating your program  Formal assessments  Of the first year  Longer-term  Formal assessments  Of the first year  Longer-term

Evaluating your program  The basics:  Comparison groups  Independent variables  Dependent variables  The basics:  Comparison groups  Independent variables  Dependent variables

Comparison groups  Historical: ESP participants vs similar students before ESP  Comparable: ESP participants vs similar students not in ESP  To the norm: ESP participants vs all non-participants  To decliners: People who rejected an invitation to ESP  Historical: ESP participants vs similar students before ESP  Comparable: ESP participants vs similar students not in ESP  To the norm: ESP participants vs all non-participants  To decliners: People who rejected an invitation to ESP

Independent variables  ESP participation  Race  Gender  Academic preparation (SAT scores; CCI pre-test)  Financial need  Motivation  ESP participation  Race  Gender  Academic preparation (SAT scores; CCI pre-test)  Financial need  Motivation

Dependent variables  CCI post-test scores  CCI growth scores  Calc grades  Raw data, % receiving A or B, % failing  Enrollment/grades in Calc II  Declaring SEM major  Graduating at all  Graduating with SEM major  CCI post-test scores  CCI growth scores  Calc grades  Raw data, % receiving A or B, % failing  Enrollment/grades in Calc II  Declaring SEM major  Graduating at all  Graduating with SEM major

Analyzing the data  Descriptive statistics: Simply compare the performances of the relevant groups  Are differences in grades or scores significant? Independent-samples t-tests  Are differences in percent of students doing something (getting As & Bs, graduating) significant? Chi-square  Descriptive statistics: Simply compare the performances of the relevant groups  Are differences in grades or scores significant? Independent-samples t-tests  Are differences in percent of students doing something (getting As & Bs, graduating) significant? Chi-square

Analyzing the data  Controlling for preparation: Divide data into groups according to some measure of preparation

Analyzing the data  Controlling for preparation: Construct a regression equation using all your available independent variables; see whether ESP participation is a significant predictor of the dependent variable of interest  For continuous dependent variable: OLS; for binary: Logistic regression  Controlling for preparation: Construct a regression equation using all your available independent variables; see whether ESP participation is a significant predictor of the dependent variable of interest  For continuous dependent variable: OLS; for binary: Logistic regression

Calculus concept inventory  Pros  This is the gold standard--did the students learn calculus? Did they learn more than other students?  This approach--at least the descriptive stats--can be used for small n  Cons  Limited number of test items--test might not be reliable or valid enough for your comfort  Requires access to all calculus students, not just ESP students  Pros  This is the gold standard--did the students learn calculus? Did they learn more than other students?  This approach--at least the descriptive stats--can be used for small n  Cons  Limited number of test items--test might not be reliable or valid enough for your comfort  Requires access to all calculus students, not just ESP students

Calc grades, SAT & GPA data  Pros:  Lets you control for preparation  Administrators like statistical analyses  Cons:  Someone has to like stats--might need SPSS  You have to find someone in institutional research to let you have the data  Requires a substantial n  Pros:  Lets you control for preparation  Administrators like statistical analyses  Cons:  Someone has to like stats--might need SPSS  You have to find someone in institutional research to let you have the data  Requires a substantial n

Examples  Fullilove & Treisman, 1990  Comparison groups:  Historical--pre-MWP African Americans  African American accepters & decliners  Preparation measures:  Special admission? Math SAT scores  Dependent variables:  Calc performance, graduation  Fullilove & Treisman, 1990  Comparison groups:  Historical--pre-MWP African Americans  African American accepters & decliners  Preparation measures:  Special admission? Math SAT scores  Dependent variables:  Calc performance, graduation

Examples  Johnson, 2007a  Comparison groups (all with 1st major in science):  White/Asian; Black/Latino/American Indian  Independent variables:  Financial need, predicted GPA  Dependent variables: Graduation with science/math major, grad GPA  Johnson, 2007a  Comparison groups (all with 1st major in science):  White/Asian; Black/Latino/American Indian  Independent variables:  Financial need, predicted GPA  Dependent variables: Graduation with science/math major, grad GPA

Expanding your program  Evidence that matriculation-to- graduation programs produce even bigger benefits:  Johnson (2007a)  Maton, Hrabowski & Schmitt (2000)  Maton & Hrabowski (2004)  Gándara (1999)  Evidence that matriculation-to- graduation programs produce even bigger benefits:  Johnson (2007a)  Maton, Hrabowski & Schmitt (2000)  Maton & Hrabowski (2004)  Gándara (1999)

Evaluating your program G What are your objectives for your program? G What data would let you know you’re meeting those objectives? G What data would convince your administration to keep funding the program? Evaluating your program G Informal assessments: To let you know that the program is working; to fine-tune it as you go along G Observations of student progress, conversations with students, informal surveys Evaluating your program G Formal assessments G Of the first year G Longer-term Evaluating your program G The basics: G Comparison groups G Independent variables G Dependent variables Comparison groups G Historical: ESP participants vs similar students before ESP G Comparable: ESP participants vs similar students not in ESP G To the norm: ESP participants vs all non-participants G To decliners: People who rejected an invitation to ESP Independent variables G ESP participation G Race G Gender G Academic preparation (SAT scores; CCI pre-test) G Financial need G Motivation Dependent variables G CCI post-test scores G CCI growth scores G Calc grades G Raw data, % receiving A or B, % failing G Enrollment/grades in Calc II G Declaring SEM major G Graduating at all G Graduating with SEM major Analyzing the data G Descriptive statistics: Simply compare the performances of the relevant groups G Are differences in grades or scores significant? Independent-samples t-tests G Are differences in percent of students doing something (getting As & Bs, graduating) significant? Chi-square Analyzing the data G Controlling for preparation: Divide data into groups according to some measure of preparation Analyzing the data G Controlling for preparation: Construct a regression equation using all your available independent variables; see whether ESP participation is a significant predictor of the dependent variable of interest G For continuous dependent variable: OLS; for binary: Logistic regression Calculus concept inventory G Pros G This is the gold standard--did the students learn calculus? Did they learn more than other students? G This approach--at least the descriptive stats--can be used for small n G Cons G Limited number of test items--test might not be reliable or valid enough for your comfort G Requires access to all calculus students, not just ESP students Calc grades, SAT & GPA data G Pros: G Lets you control for preparation G Administrators like statistical analyses G Cons: G Someone has to like stats--might need SPSS G You have to find someone in institutional research to let you have the data G Requires a substantial n Examples G Fullilove & Treisman, 1990 G Comparison groups: G Historical--pre-MWP African Americans G African American accepters & decliners G Preparation measures: G Special admission? Math SAT scores G Dependent variables: G Calc performance, graduation Examples G Johnson, 2007a G Comparison groups (all with 1st major in science): G White/Asian; Black/Latino/American Indian G Independent variables: G Financial need, predicted GPA G Dependent variables: Graduation with science/math major, grad GPA Expanding your program G Evidence that matriculation-to-graduation programs produce even bigger benefits: G Johnson (2007a) G Maton, Hrabowski & Schmitt (2000) G Maton & Hrabowski (2004) G Gándara (1999)