Collecting and Interpreting Quantitative Data – Introduction (Part 1) Deborah K. van Alphen and Robert W. Lingard California State University, Northridge May 15, 2009 van Alphen & Lingard

Overview Introduction Sampling Reliability and Validity Correlation May 15, 2009 van Alphen & Lingard 2

Introduction – Basic Questions How can we make assessment easier? minimize the effort required to collect data How can we learn more from the assessment results we obtain? use tools to interpret quantitative data May 15, 2009 van Alphen & Lingard

Making Assessment Easier Assess existing student work rather than creating or acquiring separate instruments. Measure only a sample of the population to be assessed. Utilize assessments at the College or University level. May 15, 2009 van Alphen & Lingard

Assessment Questions After completing an assessment, one might ask: What do the assessment results mean? Was the sample used valid (representative and large enough)? Were the results obtained valid? Were the instrument and process utilized reliable? Is a difference between two results significant? May 15, 2009 van Alphen & Lingard

Sampling – Definition of Terms Data: Observations (test scores, survey responses) that have been collected Population: Complete collection of all elements to be studied (e.g., all students in the program being assessed) Sample: Subset of elements selected from a population May 15, 2009 van Alphen & Lingard 6 6

Sampling Example Population = 1000 students Sample = 100 students There are 1000 students in our program, and we want to study certain achievements of these students. A subset of 100 students is selected for measurements. Population = 1000 students Sample = 100 students Data = 100 achievement measurements May 15, 2009 van Alphen & Lingard 7 7

Some Types of Sampling Random sample: Each member of a population has an equal chance of being selected Stratified sampling: The population is divided into sub-groups (e.g., male and female) and a sample from each sub-group is selected Convenience sampling: The results that are the easiest to get make up the sample May 15, 2009 van Alphen & Lingard

Problems with Sampling The sample may not be representative of the population. The sample may be too small to provide valid results. It may be difficult to obtain the desired sample. May 15, 2009 van Alphen & Lingard

Reliability and Validity Reliability refers to the consistency of a number of measurements taken using the same measurement method on the same subject (i.e., how good are the operational metrics and the measurement data). Validity refers to whether the measurement really measures what it was intended to measure (i.e., the extent to which an empirical measure reflects the real meaning of the concept under consideration). May 15, 2009 van Alphen & Lingard

Reliability and Validity Reliable but not valid Valid but not reliable Reliable & valid May 15, 2009 van Alphen & Lingard

Correlation Correlation is probably the most widely used statistical method to assess relationships among observational data. Correlation can show whether and how strongly two sets of observational data are related. Correlation can be used to show the reliability of an assessment instrument or the validity of an assessment result. May 15, 2009 van Alphen & Lingard

Reliability of Subjective Performance Data (6, 4) (Eval. #1) Assessment Situation Samples of students’ writing were scored by two evaluators, using the same rubric. Scatter plot: scores given to each sampled student by the two evaluators. Here we’re looking at the use of correlation to verify the reliability of some assessment data. Correlation coefficient of .93 between the two data sets is an indicator of the pair-wise reliability of the data from the two evaluators. May 15, 2009 van Alphen & Lingard

Validity of Assessment Results Assessment Situation A sample of students have scores on the WPE (assumed to be valid), and scores on the Department Writing Assessment. Scatter plot: scores obtained by each sampled student on the 2 assessments Now we’re looking at how we might verify the validity of one set of test data, assuming I have another set that is reliable. Correlation coefficient of .80 between the two data sets is an indicator of the validity of the results of the Department Writing Assessment. May 15, 2009 van Alphen & Lingard