1. Collecting and Interpreting Quantitative Data – Introduction (Part 1)
Deborah K. van Alphen and Robert W. Lingard
California State University, Northridge
May 15, 2009
van Alphen & Lingard

2. Overview Introduction Sampling Reliability and Validity Correlation
May 15, 2009
van Alphen & Lingard 2

3. Introduction – Basic Questions
How can we make assessment easier?
minimize the effort required to collect data
How can we learn more from the assessment results we obtain?
use tools to interpret quantitative data
May 15, 2009
van Alphen & Lingard

4. Making Assessment Easier
Assess existing student work rather than creating or acquiring separate instruments.
Measure only a sample of the population to be assessed.
Utilize assessments at the College or University level.
May 15, 2009
van Alphen & Lingard

5. Assessment Questions After completing an assessment, one might ask:
What do the assessment results mean?
Was the sample used valid (representative and large enough)?
Were the results obtained valid?
Were the instrument and process utilized reliable?
Is a difference between two results significant?
May 15, 2009
van Alphen & Lingard

6. Sampling – Definition of Terms
Data: Observations (test scores, survey responses) that have been collected
Population: Complete collection of all elements to be studied (e.g., all students in the program being assessed)
Sample: Subset of elements selected from a population
May 15, 2009
van Alphen & Lingard 6 6

7. Sampling Example Population = 1000 students Sample = 100 students
There are 1000 students in our program, and we want to study certain achievements of these students. A subset of 100 students is selected for measurements.
Population = 1000 students
Sample = 100 students
Data = 100 achievement measurements
May 15, 2009
van Alphen & Lingard 7 7

8. Some Types of Sampling
Random sample: Each member of a population has an equal chance of being selected
Stratified sampling: The population is divided into sub-groups (e.g., male and female) and a sample from each sub-group is selected
Convenience sampling: The results that are the easiest to get make up the sample
May 15, 2009
van Alphen & Lingard

9. Problems with Sampling
The sample may not be representative of the population.
The sample may be too small to provide valid results.
It may be difficult to obtain the desired sample.
May 15, 2009
van Alphen & Lingard

10. Reliability and Validity
Reliability refers to the consistency of a number of measurements taken using the same measurement method on the same subject (i.e., how good are the operational metrics and the measurement data).
Validity refers to whether the measurement really measures what it was intended to measure (i.e., the extent to which an empirical measure reflects the real meaning of the concept under consideration).
May 15, 2009
van Alphen & Lingard

11. Reliability and Validity
Reliable but
not valid
Valid but
not reliable
Reliable
& valid
May 15, 2009
van Alphen & Lingard

12. Correlation
Correlation is probably the most widely used statistical method to assess relationships among observational data.
Correlation can show whether and how strongly two sets of observational data are related.
Correlation can be used to show the reliability of an assessment instrument or the validity of an assessment result.
May 15, 2009
van Alphen & Lingard

13. Reliability of Subjective Performance Data
(6, 4)
(Eval. #1)
Assessment Situation
Samples of students’ writing were scored by two evaluators, using the same rubric.
Scatter plot: scores given to each sampled student by the two evaluators.
Here we’re looking at the use of correlation to verify the reliability of some assessment data.
Correlation coefficient of .93 between the two data sets is an indicator of the pair-wise reliability of the data from the two evaluators.
May 15, 2009
van Alphen & Lingard

14. Validity of Assessment Results
Assessment Situation
A sample of students have scores on the WPE (assumed to be valid), and scores on the Department Writing Assessment.
Scatter plot: scores obtained by each sampled student on the 2 assessments
Now we’re looking at how we might verify the validity of one set of test data, assuming I have another set that is reliable.
Correlation coefficient of .80 between the two data sets is an indicator of the validity of the results of the Department Writing Assessment.
May 15, 2009
van Alphen & Lingard