1. Natural Sciences Assessment
At Kennedy-King College
Fall 2014
DRAFT - FOR DISCUSSION ONLY
Prepared by Robert Rollings, spring 2015

2. With these results, we test that hypothesis.
In fall 2014, Kennedy-King College administered an assessment of natural sciences learning to 1,370 student respondents in a wide sample of classrooms in various subjects.
The test design progressed from questions requiring basic reading comprehension of science passages to questions demanding critical thinking and application of concepts.
The assessment designers hypothesized that students would perform well with basic comprehension and poorly with critical application.
With these results, we test that hypothesis.
DRAFT - FOR DISCUSSION ONLY

3. DRAFT - FOR DISCUSSION ONLY
The sample was demographically typical of KKC, except for a disproportionate representation of full-time students.
DRAFT - FOR DISCUSSION ONLY

4. DRAFT - FOR DISCUSSION ONLY
The assessment was administered in a broad array of classes, but in a disproportionate number of biology classes.
Thus we see the disproportionate number of respondents who report having taken a class in biology. We might expect higher overall performance on biology questions than if the survey were evenly distributed.
DRAFT - FOR DISCUSSION ONLY

5. DRAFT - FOR DISCUSSION ONLY
Students broadly agree with the value of science but are more likely to agree strongly that it is important for their careers than that they find it personally interesting.
DRAFT - FOR DISCUSSION ONLY

6. DRAFT - FOR DISCUSSION ONLY
About a quarter of respondents have not taken a science class at KKC, and only 20.8% of the sample had taken no college science courses at KKC or elsewhere. 37% of the sample had taken more than one course in the sciences.
If we have been achieving our SLOs, an overwhelming majority of respondents should have mastery of scientific method and basic concepts.
DRAFT - FOR DISCUSSION ONLY

7. DRAFT - FOR DISCUSSION ONLY

8. DRAFT - FOR DISCUSSION ONLY
Questions 1-5 measured ability to describe the scientific method and hypothesis testing. Most students performed well on this section.
However, a large minority of students show confusion about the basic nature of scientific inquiry.
People who believe that science leads to inarguable facts (or worse, is “just a theory”) may be unlikely to engage critical thinking when engaging scientific findings, and still less likely to perform science themselves.
Oddly, having taken a KKC science course makes no significant difference in correct response rates to this question!
DRAFT - FOR DISCUSSION ONLY

9. DRAFT - FOR DISCUSSION ONLY
Questions 6-10 measured conceptual mastery of precision, accuracy, and significant digits, then on application of a basic formula (Density=Mass/Volume) to measurements.
Respondents performed poorly on this section. KKC students, including those who have taken science classes, answered correctly concerning significant figures (Q7) at levels worse than if they had responded to the question randomly.
Most students could not read volume measurements from these figures correctly, nor could they distinguish between precision and accuracy.
This is foundational language in any quantitative work in science.
DRAFT - FOR DISCUSSION ONLY

10. DRAFT - FOR DISCUSSION ONLY
Students had a choice between section III, concerning atoms, and section IV, concerning DNA.
Section III began with two questions that required reading comprehension and proceeded to three questions that required application of concepts from the reading.
Section IV measured reading comprehension throughout.
The performance gap between questions that measured comprehension versus application is visible and profound.
These results appear to confirm the hypothesis that students have learned how to read and recall basic findings of science but learn far less how to apply concepts and use critical problem-solving skills.
DRAFT - FOR DISCUSSION ONLY

11. DRAFT - FOR DISCUSSION ONLY
Only a handful of students answered every question correctly, whereas here we see another way of expressing that most students got the first two or three questions per section correctly, and then the harder questions requiring critical application incorrectly.
The overall average of correct responses for Sections I-II was 4.40 out of 10 possible.
The standard deviation is 1.9.
DRAFT - FOR DISCUSSION ONLY

12. How well did various groups perform relative to each other?
4.40 is the overall average of questions 1-10 answered correctly.
These graphs will all be set within boundaries of 3.5 to 5.5 so that they can easily be compared against each other.
Gender & Race matter a little—but not much.
Note that women performed slightly better, breaking a stereotype concerning the natural sciences.
Part-time and Full-time students did about equally well. Students who were careless or couldn’t be bothered to fill in that blank performed worse than others.
DRAFT - FOR DISCUSSION ONLY

13. DRAFT - FOR DISCUSSION ONLY
Age mattered a surprising amount. Students over 41 fared far worse on the assessment.
Strong agreement with the importance of science correlated with higher overall performance.
DRAFT - FOR DISCUSSION ONLY

14. DRAFT - FOR DISCUSSION ONLY
Performance could also be divided between students who had done a lot of science, or science at anywhere other than KKC, and everyone else.
Possibly it should raise eyebrows that students who have taken 3 or more science classes still only averaged about half the questions correct. Critical application skills are not being learned.
However, students appear to learn the skills relevant to this assessment more in chemistry classes than in other sciences at KKC.
DRAFT - FOR DISCUSSION ONLY

15. DRAFT - FOR DISCUSSION ONLY
CONCLUSIONS
Students understand how to report back scientific findings,
but they have weaknesses in measurement, application, and critical thinking.
How does it matter in your class if students can engage empirical critical thinking skills?
Why do students not acquire these skills more frequently?
How can we help them learn these skills?
Strong agreement with the value of science correlates with better performance on the assessment.
Are there ways to increase student perceptions of the value of science?
Students aged above 40 performed significantly worse on the assessment.
Are older students out of practice with these kinds of problems?
Are different teaching methods depending on generation appropriate?
Students who had taken any class in chemistry performed significantly better on the assessment. Are several fundamental skills being included more in the chemistry curriculum than in other core science courses? Or could there be a confounding variable involved?
DRAFT - FOR DISCUSSION ONLY