1. New Survey Questionnaire Indicators in PISA and NAEP
Jonas Bertling Director, NAEP Survey Questionnaires, ETS
RWJF & ETS Workshop
Understanding the opportunities for collaboration between the health and education sectors in a culture of health
Princeton, May 5, 2016

2. Main talking points today
Survey questionnaires in educational large-scale assessments started collecting broader types of contextual variables reflecting the increased policy interest in “noncognitive” skills and additional outcomes.
PISA broadened the range of constructs captured with the student questionnaires and introduced health and well-being facets in PISA 2018 will include an optional well-being questionnaire.
NAEP started innovating their survey questionnaires and will introduce a series of new contextual modules (including noncognitive indices) in 2017.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Copyright © 2016 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS)

3. Main talking points today (continued)
NAEP 2017 contextual modules can also be mapped to PISA draft well-being framework.
PISA conceptualizes noncognitive variables explicitly as additional outcomes; NAEP currently models them on the predictor side only.
Challenges fall especially into three categories: Measurement, Perceived Intrusiveness and Respondent Burden.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

4. Changing role of large-scale assessments
How do subgroups perform?
How do students approach learning situations and utilize learning opportunities?
How do learning opportunities differ across the Nation?
How are students in the Nation doing in terms of other outcomes than achievement?
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

5. Shift in policy interest
Measure contextual variables
Measure constructs of own interest
Describe test-taker populations
Improve estimates of achievement (conditioning model)
Report additional outcomes
Measure trend
Compare subgroups
Evaluate policies
Accountability
Figure from Bertling, Borgonovi, & Almonte (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

6. Examples
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

7. Survey questionnaires data put achievement results into context …
… and can add value to large-scale assessments by capturing important additional variables.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

8. PISA Programme for International Student Assessment
Organisation for Economic Co-operation and Development (OECD)

9. PISA broadened the range of constructs captured with survey questionnaires since the first cycle in 2000
Since its early cycles, PISA has included questions capturing students’ attitudes and sense of belonging to school
PISA 2012 introduced new domain-general student attitudes and behaviors (e.g., perseverance, openness for problem solving)
PISA 2015 added additional noncognitive outcomes also covering health and well-being
This trend continues with PISA 2018
Figure from Bertling, Borgonovi, & Almonte (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

10. PISA 2015 Questionnaire Framework
Figure from OECD (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

11. Well-being-related constructs in PISA 2015
General life satisfaction
Participation in physical education classes at school
Engagement in moderate and vigorous physical exercise
School-related well-being (sense of belonging, bullying, test anxiety)
A larger number of additional health- and well-being-related questions (e.g., BMI, dietary habits, enjoyment of physical activity, etc.) were piloted but not taken to operational largely due to timing constraints.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

12. Well-being questionnaire in PISA 2018
DRAFT*
OECD is planning an optional questionnaire devoted to student well-being for PISA in addition to other noncognitive factors captured with the regular student questionnaires.
Developed draft framework for PISA Module
*Note: The PISA 2018 well-being framework is not a public document and currently remains a DRAFT version, is subject to all the normal OECD copyright provisions, and should clearly be acknowledged with the OECD as owner and author.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

13. NAEP National Assessment of Educational Progress
National Center for Education Statistics (NCES)

14. Innovations underway for the NAEP Survey Questionnaires
Expand constructs to include more contextual factors critical for student achievement I.
Improve reporting of contextual factors for more robustness & policy relevance
II.
Improve pre-testing and questionnaire design for digital delivery to maximize validity and keep student burden low
III.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

15. NAEP Contextual Variables
Opportunity-to-Learn (OTL) Factors:
(COGNITIVE ASSESSMENT)
(SURVEY QUESTIONNAIRES)
“Noncognitive” Student Factors:
Achievement
Domain-general factors (“Core”)
Subject-specific factors
*not measured with the NAEP survey questionnaires
Students’ exposure to opportunities to acquire relevant knowledge and skills, both at school and outside of school.
Skills, strategies, attitudes and behaviors that are distinct from content knowledge and academic skills (e.g., motivational and self-regulatory variables)
OTL at School
BUT: The term “noncognitive” is not intended to be interpreted literally. All aspects of human thought and behavior are related to cognition!
Cognitive Ability*
“Noncognitive” Student Factors
OTL Outside of School
Basic Demographics
Balanced reporting approach based on indices and stand-alone questions
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

16. Implementation for NAEP 2017 Core, Reading, Writing and Mathematics:
Item Development
Pre-testing in Cognitive Labs
Large-scale Pilot Testing
Operational Administration
Noncogs
Domain-general (Core)
OTL Outside of School
OTL At School
“Noncognitive” Student Factors
Basic Demographics
Subject-specific 1 2 3 4 5 6 7
Grit/Perseverance (including self-control)
Desire for Learning
Self-Efficacy/Confidence
Achievement Goals
Interest & Enjoyment
Components of SES
Technology Use
School Climate
OTL
at School
Out-of-School Activities
Resources for Learning and Instruction
Organization of Instruction
Teacher Preparation
5 Core Modules
Additional Subject-specific Modules
Outside of School
2014 
2016
2017
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

17. Mapping NAEP to PISA 2018 WB Framework
DRAFT*
Grit/Perseverance,
Desire for Learning,
Self-Efficacy
School Climate
SES
*Note: The PISA 2018 well-being framework is not a public document and currently remains a DRAFT version, is subject to all the normal OECD copyright provisions, should clearly be acknowledged with the OECD as owner and author.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

18. Difference between PISA and NAEP approaches
Additional Outcomes
Achievement Predictors
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

19. Perceived Intrusiveness Respondent Burden
Challenges
Measurement
Perceived Intrusiveness
Respondent Burden
Figure from Bertling, Borgonovi, & Almonte (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

20. Summary
Survey questionnaires in educational large-scale assessments started collecting broader types of contextual variables reflecting the increased policy interest in “noncognitive” skills and additional outcomes.
PISA broadened the range of constructs captured with the student questionnaires and introduced health and well-being facets in PISA 2018 will include an optional well-being questionnaire.
NAEP started innovating their survey questionnaires and will introduce a series of new contextual modules (including noncognitive indices) in 2017.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

21. Summary (continued)
NAEP 2017 contextual modules can also be mapped to PISA draft well-being framework.
PISA conceptualizes noncognitive variables explicitly as additional outcomes; NAEP currently models them on the predictor side only.
Challenges fall especially into three categories: Measurement, Perceived Intrusiveness and Respondent Burden.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016

22. Thank you! Questions? Comments? jbertling@ets.org
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Copyright © 2016 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS)