1. 2018 NSSME+: Briefing Book Slides
Horizon Research, Inc.
Chapel Hill, NC

2. Acknowledgement
These materials are based upon work conducted by Horizon Research, Inc. and supported by the National Science Foundation under Grant No. DGE Any opinions, findings, and conclusions or recommendations expressed are those of the author and do not necessarily reflect the views of Horizon Research, Inc. or the National Science Foundation.

3. Study Overview

4. About the 2018 NSSME+
The 2018 NSSME+ is the sixth in a series of surveys dating back to 1977.
It is the only survey specific to STEM education that provides nationally representative results.

5. Endorsing Organizations
American Association of Chemistry Teachers
American Association of Physics Teachers
American Federation of Teachers
Association of Mathematics Teacher Educators
American Society for Engineering Education
Association of State Supervisors of Mathematics
Association for Science Teacher Education
Council of State Science Supervisors
Computer Science Teachers Association
National Association of Biology Teachers
National Association of Elementary School Principals
National Association of Secondary School Principals
National Council of Supervisors of Mathematics
National Council of Teachers of Mathematics
National Earth Science Teachers Association
National Education Association
National Science Education Leadership Association
National Science Teachers Association

6. About the Study Two-stage random sample that targeted:
2,000 schools (public and private)
Over 10,000 K–12 teachers
Purposefully oversampled teachers of advanced mathematics, chemistry, and physics to allow for disaggregated results for each group
Six instruments:
School coordinator questionnaire
Science program questionnaire
Mathematics program questionnaire
High school computer science teacher questionnaire
Science teacher questionnaire
Mathematics teacher questionnaire

7. Topics Addressed
Characteristics of the science/mathematics/ computer science teaching force:
demographics
preparation for teaching
beliefs about teaching and learning
perceptions of preparedness
Instructional practices
Factors that shape teachers’ decisions about content and pedagogy
Use of instructional materials
Opportunities teachers have for professional growth
How instructional resources are distributed

8. Excellent response rate: 1,273 schools agreed to participate
86 percent of program representatives
78 percent of sampled teachers
Sampling and analysis techniques used provide nationally representative estimates.
Results should be presented as percentage of schools, teachers, classes in the nation (not “in the sample”).

9. For More Information on the 2018 NSSME+

10. Using the Briefing Book

11. Organization
The Briefing Book follows the structure of the Report of the 2018 NSSME+: Banilower, E. R., Smith, P. S., Malzahn, K. A., Plumley, C. L., Gordon, E. M., & Hayes, M. L. (2018). Report of the 2018 NSSME+. Chapel Hill, NC: Horizon Research, Inc.

12. Three sets of slides, one for science, one for mathematics, and one for computer science, were created for each chapter of the report that included results from the study. When the results would fit, all three grade ranges (elementary, middle, and high) are represented on a single slide. When the grade ranges are on separate slides, note that the slides may show the data for each grade range in different orders.

13. For each set of slides presenting findings, the Briefing Book first contains an overview slide that is not intended for use in presentations. The overview slide shows the original data table from the report. Numbers in parentheses in the tables are standard errors. Note that several tables in the technical report contain science, mathematics, and computer science data. The science slides show only the science portion of those tables, the mathematics slides show only the mathematics portion, and the computer science slides show only the computer science portion. In some cases, select data from a table were chosen to be shown in a graph.

14. The overview slide notes include:
Where in the report the table appears;
An indication of whether the table shows data from individual survey items, composite variables, or other variables derived from the survey data;
The original questionnaire items the data are based on; and
The text from the report that was written about the data table.

15. Conventions used in displaying original items:
Instructions that were included with questionnaire items are represented by a darkened circle: 
Sub-items for questionnaire items are represented by bulleted letters: a., b., c., etc.
Response options for items with sub-items are shown in parentheses: ()
“Select one” response options are represented by an open circle: 
“Select all that apply” response options are represented by a check box: 
Free-response items are represented by a blank line: _____
In some cases, only some sub-items were used to create a table; sub-items that were not used in the table are included but crossed out (strikethrough).

16. Composite Variables
Sets of related questionnaire items were combined to create several composite variables related to key constructs.
Composite variables, which are more reliable than individual survey items, were computed to have a minimum possible value of 0 and a maximum possible value of 100.
Higher composite scores indicate more of the construct being measure by the composite.
Overview slides that contain composite variable data include a more in-depth description of how composites were calculated.

17. Equity Factors
Chapters 2–7 in the technical report provide data on several key indicators, disaggregated by one or more equity factors:
the prior achievement level of students in the class,
the percentage of students in the class from race/ethnic groups historically underrepresented in STEM (HUS),
the percentage of students in the school eligible for free/reduced-price lunch (FRL),
school size,
community type, and
region.
For school size and FRL, each school was classified into one of four quartiles. Similarly, each randomly selected class was classified into one of the four categories based on the proportion of students identified as HUS.

18. Accurately Communicating Findings
Results from the 2018 NSSME+ are nationally representative—thus, it is appropriate to talk about schools, teachers, and classes in the nation. It is not appropriate to talk about schools, teachers, and classes “in the sample.”

19. Accurately Communicating Findings
It is important to pay attention to figure/axis titles:
Schools vs. Classes vs. Teachers
Because of the sample design, you cannot interchange “classes” and “teachers.”

20. Accurately Communicating Findings
The scale on a figure axis affects the interpretation of the figure:
Magnifying small differences
Obscuring important differences
In the briefing book, almost all figures are on a 0–100 scale. Exceptions are when the bars are supposed to add to 100 percent.

21. Accurately Communicating Findings
Be wary of making mountains out of molehills. Although some results point to differences among groups (e.g., by grade level), it will be important to consider whether a finding is educationally significant as well as statistically significant.

22. Accurately Communicating Findings
Not all apparent “differences” are statistically significant. If the “difference” isn’t statistically significant, it’s not a difference—the result could be due to sampling error.

23. Statistical Significance
Unfortunately, there’s not a simple way for you to test statistical significance.

24. Option 1
Look in the 2018 NSSME+ report If we wrote about a difference in a report, that difference was tested and is statistically significant.
Other two options are approximations.

25. Option 2
An approximation of whether two numbers are statistically different is to add the standard errors and compare that to the difference between the two numbers. If the difference is greater than the sum of standard errors, it is likely statistically significant.

26. Option 3
If you want to be somewhat more precise, and put in a little more effort… We have an Excel spreadsheet calculator you can use for comparing percentages.
Though again, if the difference is borderline statistically significant, it may not be educationally significant.