2018 NSSME+: Briefing Book Slides Horizon Research, Inc. Chapel Hill, NC

Acknowledgement These materials are based upon work conducted by Horizon Research, Inc. and supported by the National Science Foundation under Grant No. DGE-1642413. Any opinions, findings, and conclusions or recommendations expressed are those of the author and do not necessarily reflect the views of Horizon Research, Inc. or the National Science Foundation.

Study Overview

About the 2018 NSSME+ The 2018 NSSME+ is the sixth in a series of surveys dating back to 1977. It is the only survey specific to STEM education that provides nationally representative results.

Endorsing Organizations American Association of Chemistry Teachers American Association of Physics Teachers American Federation of Teachers Association of Mathematics Teacher Educators American Society for Engineering Education Association of State Supervisors of Mathematics Association for Science Teacher Education Council of State Science Supervisors Computer Science Teachers Association National Association of Biology Teachers National Association of Elementary School Principals National Association of Secondary School Principals National Council of Supervisors of Mathematics National Council of Teachers of Mathematics National Earth Science Teachers Association National Education Association National Science Education Leadership Association National Science Teachers Association

About the Study Two-stage random sample that targeted: 2,000 schools (public and private) Over 10,000 K–12 teachers Purposefully oversampled teachers of advanced mathematics, chemistry, and physics to allow for disaggregated results for each group Six instruments: School coordinator questionnaire Science program questionnaire Mathematics program questionnaire High school computer science teacher questionnaire Science teacher questionnaire Mathematics teacher questionnaire

Topics Addressed Characteristics of the science/mathematics/ computer science teaching force: demographics preparation for teaching beliefs about teaching and learning perceptions of preparedness Instructional practices Factors that shape teachers’ decisions about content and pedagogy Use of instructional materials Opportunities teachers have for professional growth How instructional resources are distributed

Excellent response rate: 1,273 schools agreed to participate 86 percent of program representatives 78 percent of sampled teachers Sampling and analysis techniques used provide nationally representative estimates. Results should be presented as percentage of schools, teachers, classes in the nation (not “in the sample”).

For More Information on the 2018 NSSME+ http://horizon-research.com/NSSME/2018-nssme

Using the Briefing Book

Organization The Briefing Book follows the structure of the Report of the 2018 NSSME+: Banilower, E. R., Smith, P. S., Malzahn, K. A., Plumley, C. L., Gordon, E. M., & Hayes, M. L. (2018). Report of the 2018 NSSME+. Chapel Hill, NC: Horizon Research, Inc.

Three sets of slides, one for science, one for mathematics, and one for computer science, were created for each chapter of the report that included results from the study. When the results would fit, all three grade ranges (elementary, middle, and high) are represented on a single slide. When the grade ranges are on separate slides, note that the slides may show the data for each grade range in different orders.

For each set of slides presenting findings, the Briefing Book first contains an overview slide that is not intended for use in presentations. The overview slide shows the original data table from the report. Numbers in parentheses in the tables are standard errors. Note that several tables in the technical report contain science, mathematics, and computer science data. The science slides show only the science portion of those tables, the mathematics slides show only the mathematics portion, and the computer science slides show only the computer science portion. In some cases, select data from a table were chosen to be shown in a graph.

The overview slide notes include: Where in the report the table appears; An indication of whether the table shows data from individual survey items, composite variables, or other variables derived from the survey data; The original questionnaire items the data are based on; and The text from the report that was written about the data table.

Conventions used in displaying original items: Instructions that were included with questionnaire items are represented by a darkened circle:  Sub-items for questionnaire items are represented by bulleted letters: a., b., c., etc. Response options for items with sub-items are shown in parentheses: () “Select one” response options are represented by an open circle:  “Select all that apply” response options are represented by a check box:  Free-response items are represented by a blank line: _____ In some cases, only some sub-items were used to create a table; sub-items that were not used in the table are included but crossed out (strikethrough).

Composite Variables Sets of related questionnaire items were combined to create several composite variables related to key constructs. Composite variables, which are more reliable than individual survey items, were computed to have a minimum possible value of 0 and a maximum possible value of 100. Higher composite scores indicate more of the construct being measure by the composite. Overview slides that contain composite variable data include a more in-depth description of how composites were calculated.

Equity Factors Chapters 2–7 in the technical report provide data on several key indicators, disaggregated by one or more equity factors: the prior achievement level of students in the class, the percentage of students in the class from race/ethnic groups historically underrepresented in STEM (HUS), the percentage of students in the school eligible for free/reduced-price lunch (FRL), school size, community type, and region. For school size and FRL, each school was classified into one of four quartiles. Similarly, each randomly selected class was classified into one of the four categories based on the proportion of students identified as HUS.

Accurately Communicating Findings Results from the 2018 NSSME+ are nationally representative—thus, it is appropriate to talk about schools, teachers, and classes in the nation. It is not appropriate to talk about schools, teachers, and classes “in the sample.”

Accurately Communicating Findings It is important to pay attention to figure/axis titles: Schools vs. Classes vs. Teachers Because of the sample design, you cannot interchange “classes” and “teachers.”

Accurately Communicating Findings The scale on a figure axis affects the interpretation of the figure: Magnifying small differences Obscuring important differences In the briefing book, almost all figures are on a 0–100 scale. Exceptions are when the bars are supposed to add to 100 percent.

Accurately Communicating Findings Be wary of making mountains out of molehills. Although some results point to differences among groups (e.g., by grade level), it will be important to consider whether a finding is educationally significant as well as statistically significant.

Accurately Communicating Findings Not all apparent “differences” are statistically significant. If the “difference” isn’t statistically significant, it’s not a difference—the result could be due to sampling error.

Statistical Significance Unfortunately, there’s not a simple way for you to test statistical significance.

Option 1 Look in the 2018 NSSME+ report If we wrote about a difference in a report, that difference was tested and is statistically significant. Other two options are approximations.

Option 2 An approximation of whether two numbers are statistically different is to add the standard errors and compare that to the difference between the two numbers. If the difference is greater than the sum of standard errors, it is likely statistically significant.

Option 3 If you want to be somewhat more precise, and put in a little more effort… We have an Excel spreadsheet calculator you can use for comparing percentages. Though again, if the difference is borderline statistically significant, it may not be educationally significant.