Thinking and Acting Like a Scientist: Investigating the Outcomes of Introductory Science and Math Courses Kevin Eagan Jessica Sharkness Sylvia Hurtado Higher Education Research Institute, UCLA Association for Institutional Research 49th Annual Forum May 30 - June 3, 2009 Atlanta, Georgia

Background  Relatively few students earn degrees in natural science or engineering in the U.S. 15% of U.S. BA degrees are in science/engineering  Compared to 67% in Singapore, 50% in China, 47% in France, 38% in South Korea  U.S. needs more undergraduate science majors to maintain achievement and innovation in science and engineering Also need to diversify the scientific workforce and increase representation of women and minorities

Background  To graduate more bachelor’s degrees in science, U.S. needs students to choose science majors and to maintain interest in science majors National increases in proportion of freshmen indicating interest in science, technology, engineering and math (STEM) majors However, low proportion of students who intend to major in STEM actually graduate with STEM majors  One obstacle to STEM major completion: Introductory “gatekeeper” courses Mechanism for sorting students What is rewarded?

Introductory “Gatekeeper” Courses  First course in a series of courses in which knowledge is cumulative  In science – relatively high drop-out and failure rates in gatekeeper courses Large lectures Un-engaging Highly competitive Grading on a curve

Classroom Environments & Instructor Pedagogies  Classroom climates have an impact on learning and performance Competitive environments have negative impact on learning, performance, retention, self-confidence  Collaborative environments that emphasize group work can mitigate negative effects of large lectures and competitive environments Can also promote critical thinking about scientific concepts and their applications

Supportive Learning Environments and the Skills Needed for Scientific Success  Six necessary conditions for a supportive learning environment: Quality of instruction, Teacher’s interest, Social relatedness, Support of competence, Support of autonomy Engender greater self-motivation, encourages self-directed learning  Two primary pedagogical techniques in science Domain-specific learning = memorization of facts and causal relationships Domain-general learning = reasoning strategies and critical thinking skills

Additional influences on student success in STEM courses  Experiences external to the classroom environment Participation in research projects Peer Tutoring  Prior academic achievement and preparation Most significant influence on outcome of introductory courses?

Goals of Current Study  Untangle effect of prior preparation and background factors from performance assessment in introductory STEM courses  Identify how students develop in introductory courses the critical thinking dispositions necessary for science careers, and whether these dispositions are reflected in student grades

Conceptual model High School Science Achievement Demographic variables Amount of student effort expended on course Course Learning Environment & Pedagogy Tutoring & Research Participation Course Grade Ability to act and think like a scientist (Pre-Test) Ability to act and think like a scientist (Post-test) Critical thinking dispositions

Data & Sample  Data collected via online survey from students in 12 introductory science and math courses at 5 institutions  Two surveys – one at beginning of course (pre-survey) and one at end (post-survey); final analytic sample = 255  Final longitudinal sample: 70% female 34% White, 43% Asian, 8% Black, 13% Latino 86% majoring in STEM field 64% first-year students, 27% second-year

Thinking and Acting like a Scientist Loadings FactorItems*Pre-testPost-test Thinking like a scientist pre-test See connections between different areas of science & math Understand scientific concepts Identify what is known and not known in a problem Ask relevant questions0.72 Draw a picture to represent a problem or concept Make predictions based on existing knowledge Come up with solutions and explain them to others Investigate alternative solutions to a problem Understand/translate scientific terminology into non-scientific language Acting like a scientist pre-test Relate scientific concepts to real-world problems Synthesize or comprehend several sources of information Conduct an experiment Look up scientific research articles and resources Memorize large quantities of information *All items were asked as part of a questions stem that read, “Rate your ability in the following areas as it pertains to your academic learning in the sciences.” Response options were Major Strength (5), Above Average (4), Average (3), Below Average (2), Major Weakness (1) Measurement model fit statistics: χ2= (305, N=255), NNFI = 0.98, CFI = 0.98, RMSEA = 0.03, reliability = 0.82.

Variables  Independent Variables: Demographic characteristics Prior preparation Course pedagogy Classroom environment College experiences Critical thinking dispositions (CCTDI subscales)  Dependent Variables Final course grade Dispositions toward science

Analysis Plan  Identification of latent constructs (factors), representing acting like a scientist and thinking like a scientist Exploratory Factory Analysis  Confirmatory Factor Analysis (measurement model in SEM)  Structural Equation Modeling (SEM) to model how student experiences in introductory courses affect three outcomes

Structural Modeling Procedure  Added hypothesized predictors and paths to structural equation model  Used prior research, theory and Wald and LaGrange multiplier tests to add and remove paths that did not contribute to model fit When all paths were deleted from a variable, variable was removed from the model

Results (non-significant paths not shown) Female Income URM Student Felt Overwhelmed Tutored Student in HS Avg HS GPA in STEM courses Felt competition in course AP Chemistry Score Participated in HS research program BBS Major Crammed for exams HPW Engage in lab activities Sought tutoring on campus Act like a scientist Pre-test Think like a scientist Pre-test Grade in intro course Consistently got support needed HPW prof’s research project (college) Format primarily lecture Act like a scientist Post-test Think like a scientist Post-test External Support/ Science Experiences Student Effort Course Learning Environment Course employed group activities Course Pedagogy Demographics Pretests Critical thinking dispositions Open- Mindedness Critical Thinking Confidence Analyticity

Discussion of findings: Thinking and Acting Like a Scientist  CCTDI – Openmindedness negatively predicted both dispositions Relativism in conflict with objective, empirical perspective of science  CCTDI – Critical thinking self-confidence positively predicted both dispositions Indicative of students’ development of domain-general science skills  CCTDI – Analyticity positively predicted thinking like a scientist Indicative of students’ ability to think more carefully and critically during problem-solving activities  Students who felt overwhelmed scored lower on both dispositions May be indicative of naïveté about what is required of science majors

Discussion of findings: Final Course Grade  Thinking and acting like a scientist and CCTDI subscales unrelated to final grade Suggests that introductory courses focus too much on the acquisition of knowledge rather than development of higher-order thinking skills  Final grade in large part predicted by prior preparation (high school grades, research) Suggests that failure to earn top grades may merely lack the preparation necessary for success in these courses rather than science- related skills

Discussion of findings: Final Course Grade  Cramming for exams positively predicted course grades Underscores that grades reward rote memorization rather than development of higher-order thinking skills  Group work positively predicted final course grade Supports prior research that concluded that peer learning provides learning reinforcement, which may have longer-term benefits  Tutoring (receiving and providing) positively predicted final course grade Reinforces the benefits of peer-to-peer teaching and learning

Implications and Conclusions  Can we afford to cram content into courses at the expense of development of scientific skills and thinking?  Adjust grading practices so that they more accurately reflect learning rather than prior preparation  Further examination of pedagogical practices and interventions in large, lecture-based gatekeeper courses is needed

Contact Information Acknowledgments: This study was made possible by the support of the National Institute of General Medical Sciences, NIH Grant Numbers 1 R01 GMO and R01 GMO as well as the National Science Foundation, NSF Grant Number This independent research and the views expressed here do not indicate endorsement by the sponsors. Papers and reports are available for download from project website: Project Faculty and Co-PIs: Sylvia Hurtado Mitchell Chang Graduate Research Assistants: Kevin Eagan Lorelle Espinsoa Christopher Newman Jessica Sharkness Minh Tran Paolo Velasco Administrative Staff: Aaron Pearl