AScILS Assessing Science Inquiry and Leadership Skills Research Questions: How do activities implemented by support programs (especially research and mentoring) influence: - skills in science inquiry and scientific team leadership and membership - beliefs in efficacy and collective efficacy regarding these skills - stage-appropriate education and career outcomes? Are these influences similar for minority and non-minority students?

“Inquiry” in the context of AScILS Lisa Hunter Associate Director, Education & Human Resources, Center for Adaptive Optics AScILS Research Team Member Inquiry Strand Advisor/Mentor Focus Group, 9/13 & 9/15/05

What do we mean by “inquiry” The processes or methods that scientists use to gain an understanding of the world, and to create a body of knowledge (“research”)

Which “Science Inquiry Skills?” CfAO Survey Intern Advisor/Mentor Meeting (9/13 & 9/15) Education & Assessment Lit. AScILS Alumni Survey AScILS qualitative Studies (interviews) Team Members’ Prior Experience Current AScILS Inquiry Strand Focus: Which science inquiry skill(s) or knowledge should be assessed?

Elements of scientific inquiry Skills or processes (what scientists do) Knowledge about scientific inquiry (what scientists know)

Scientific Inquiry Processes Chinn, et. al. “Cognitive processes” America’s Lab Report “Scientists’ activities” Generating research questions Posing a research question Designing studiesFormulating hypotheses Making observationsDesigning investigations Explaining resultsMake observations, gather & analyze data Developing theoriesBuilding and revising scientific models or theories Studying research reportsEvaluating, testing, or verifying models 1. Chinn, Clark, et. al. (2002). Epistemilogically Authentic Inquiry in Schools. Science Education, 86:2. 2. National Research Council (2005). America’s Lab Report: Investigations in High School Science. Washington, DC: The National Academies Press.

Knowledge about scientific inquiry or the nature of science Lederman, et. al “Nature of science aspects” Tentativeness (of scientific knowledge) Empirical basis Subjectivity (influenced and driven by presently accepted theories) Creativity Observations and inference Social and cultural influences Nature of theories (how and why they change) Lederman, N.G. et. al. (2002). Journal of Research in Science Teaching, 39(6),

Why we are interested in “inquiry” Many (10 of 14) of the AScILS study programs include research experiences, and is one of the 3 criteria for inclusion as a study program Students engage in authentic research  expect gains in inquiry (or research) skills and understandings about inquiry NOTE: Student researchers clearly gain specific technical skills, but this is outside the AScILS focus

How can we assess interns’ gains in inquiry skills and/or understandings about inquiry?

If we could assess inquiry skills, there are many interesting questions Do students gain inquiry skills and understandings about inquiry more in a research experience than in classes? Do gains depend on the type of research experience? Is there a connection between gains in inquiry skills and students’ confidence (or self- efficacy)? Students’ identity as a scientist? Persistence in science?

What is known? Knowledge base on assessing inquiry skills gained from authentic research is very limited: –Papers reporting self-reported gains from interviews and surveys “Thinking and working like a scientist” is a major gain/benefit from research experiences* –Papers reporting on how research experiences impacted students understandings about the nature of science –Significant amount of work on assessing inquiry skills at K- 12 classroom level *Seymour, Elaine, et. al. (2004). Establishing the Benefits of Resarch Experiences for Undergraduates: First Findings from a Three-Year Study. Science Education, 88:

AScILS Inquiry Strand Determine the most fruitful inquiry skill to assess Design a performance assessment (or simulation) to assess those skills

Which “Science Inquiry Skills?” CfAO Survey Intern Advisor/Mentor Meeting (9/13 & 9/15) Education & Assessment Lit. AScILS Alumni Survey AScILS qualitative Studies (interviews) Team Members’ Prior Experience Current AScILS Inquiry Strand Focus: Which science inquiry skill(s) or knowledge should be assessed?

Scientific Inquiry Processes Chinn, et. al. “Cognitive processes” America’s Lab Report “Scientists’ activities” Generating research questions Posing a research question Designing studiesFormulating hypotheses Making observationsDesigning investigations Explaining resultsMake observations, gather & analyze data Developing theoriesBuilding and revising scientific models or theories Studying research reportsEvaluating, testing, or verifying models

Scientific Inquiry Processes Chinn, et. al. “Cognitive processes” America’s Lab Report “Scientists’ activities” Engineering processes Generating research questions Posing a research question Identifying a problem or design need Designing studiesFormulating hypothesesDesigning solutions within constraints Making observationsDesigning investigationsImplementing Explaining resultsMake observations, gather & analyze data Testing Developing theoriesBuilding and revising scientific models or theories Explaining solution, including limitations and tradeoffs Studying research reportsEvaluating, testing, or verifying models Studying research reports

As a result of doing research with your group, which skill(s) are most important for your students to develop? Chinn, et. al. “Cognitive processes” America’s Lab Report “Scientists’ activities” Engineering processes Generating research questions Posing a research question Identifying a problem or design need Designing studiesFormulating hypothesesDesigning solutions within constraints Making observationsDesigning investigationsImplementing Explaining resultsMake observations, gather & analyze data Testing Developing theoriesBuilding and revising scientific models or theories Explaining solution, including limitations and tradeoffs Studying research reportsEvaluating, testing, or verifying models Studying research reports

ASSESSMENT BASICS AScILS Style Jerome Shaw Assistant Professor Education Department

Source: National Research Council. (2001). Knowing what students know: The science and design of educational assessment. Committee on the Foundations of Assessment. Pelligrino, J., Chudowsky, N., and Glaser, R., editors. Board on Testing and Assessment, Center for Education. Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.

ASSESSMENT MANTRA STANDARDS – what? TASKS – how? RUBRICS – how well?

Scientific Inquiry Processes “Cognitive Processes” Chinn et al. “Scientists’ activities” America’s Lab Report Engineering processes Generating research questionsPosing a research questionIdentifying a problem or design need Designing studiesFormulating hypothesesDesigning solutions within constraints Making observationsDesigning investigationsImplementing Explaining resultsMake observations, gather & analyze data Testing Developing theoriesBuilding and revising scientific models or theories Explaining solutions, including limitations and tradeoffs Studying research reportsEvaluating, testing, or verifying models Studying research reports

Source: Sandoval, W. A., & Reiser, B. J. (2004). Explanation-driven inquiry: Integrating conceptual and epistemic scaffolds for scientific inquiry. Science Education, 88, Explanation Rubric: Sample Criteria  Thoroughness and Clarity of Explanations  Use of Data  Ruling Out Alternative Explanations  Documenting the Limitations of Your Explanations

Scoring Guide: Using Evidence Score Response uses objective reason(s) based on relevant evidence to argue for or against a choice. 4 Response accomplishes level 3, AND goes beyond in some relevant way, e.g., questioning or justifying the source, validity, and/or quantity of evidence. 3 Provides major objective reasons AND supports each with relevant and accurate evidence. 2 Provides some objective reasons AND some supporting evidence, BUT at least one reason is missing and/or part of the evidence is incomplete. 1 Provides only subjective reasons (opinions) for choice; uses unsupported statements; OR uses inaccurate or irrelevant evidence from the activity. Source: Science Education for Public Understanding Program (1995) as presented in… National Research Council. (2001). Classroom assessment and the National Science Education Standards. Committee on Classroom Assessment and the National Science Education Standards. J. Myron Atkin, Paul Black, and Janet Coffey (Eds.). Center for Education, Division of Behavioral and Social Sciences Education. Washington, DC: National Academy Press.

Which “Science Inquiry Skills?” CfAO Survey Intern Advisor/Mentor Meeting (9/13 & 9/15) Education & Assessment Lit. AScILS Alumni Survey AScILS qualitative Studies (interviews) Team Members’ Prior Experience Current AScILS Inquiry Strand Focus: Which science inquiry skill(s) or knowledge should be assessed?