1. Contextual questionnaires Overview of PISA instruments
PISA for Development
Technical Strand 1:
Contextual questionnaires
Overview of PISA instruments
(constructs, indices and variables)
EDU/DCD

2. PISA for Development Contextual questionnaires
Purpose of PISA contextual questionnaires
How are they used?
How they are developed
What will PISA for Development seek to do?
Questions for discussion
This presentation presents an overview of the PISA for Development initiative and it is meant to support discussions and engagement among development partners and country Ministry officials.
In addition to this presentation, additional materials include:
Narrative description of Project
Logical Framework of Project
Timeline presentation (PREZI and pdf file)
Two-page summary brief

3. PISA for Development Purpose of PISA contextual questionnaires
Student and school questionnaires are part of the core assessment design of PISA
Countries may also opt for additional questionnaires:
Parents, Teachers, Health, Time Use, ICT familiarity, Career…
Student questionnaires: ~ 30 minute* stand- alone document (students respond after test questions)
School questionnaires: ~ 30 minute stand-alone document (school authority responds before or on day of testing)
* Rotated questionnaires in 2012 to cover additional material
This series of slides provide an overview of PISA – focusing on what it is designed to assess. Despite the widespread reference to the “rankings” in PISA, deeper knowledge of the actual PISA assessment is quite limited.

4. PISA for Development Purpose of PISA contextual questionnaires
Contextualise the performance results in reading, mathematics and science*
Allow analyses of equity issues across sub-groups and populations
Provide indications of possible policy interventions
Some information collected is used to scale cognitive assessment data
Country-specific questions can be included for additional analysis
* Additional domains such as financial literacy and problem-solving
This series of slides provide an overview of PISA – focusing on what it is designed to assess. Despite the widespread reference to the “rankings” in PISA, deeper knowledge of the actual PISA assessment is quite limited.

5. PISA for Development Purpose of PISA contextual questionnaires
Have been used in 74 countries/economies
In more than 40 languages
Publicly available
PISA 2009 Student Questionnaire
PISA 2012 Student Questionnaire
PISA 2012 School Questionnaire
PISA 2009 School Questionnaire
This series of slides provide an overview of PISA – focusing on what it is designed to assess. Despite the widespread reference to the “rankings” in PISA, deeper knowledge of the actual PISA assessment is quite limited.

6. PISA Contextual Questionnaires How they are used
This is the PISA Analytic Framework that guides data collection and analysis
Outputs and Outcomes impact of learning
Policy Levers shape educational outcomes
Antecedents contextualise or constrained policy
Individual learner
Quality and distribution of knowledge & skills
Individual attitudes, engagement and behaviour
Socio-economic background of learners
Instructional settings
Quality of instructional delivery
Teaching, learning practices and classroom climate
Student learning, teacher working conditions
Levels of an education system
This slide presents the PISA Analytic Framework on which the data collection analysis is based.
The PISA analytic framework includes distinct levels and domains through which to assess an educational system:
4 levels: individual learner, instructional setting, schools and country or system level.
3 domains on which PISA collects and reports on data are Outputs and Outcomes, Policy Levers and Antecedents.
This basic framework will be reviewed as part of the PISA for Development pilot in order to address if and how it may need to be extended to include aspects that may be relevant (currently not in the framework) considering contexts in developing countries.
Schools (other institutions)
Output and performance of institutions
The learning environment at school
Community and school characteristics
Country or system
Social & economic outcomes of education
Structures, resource allocation and policies
National educ., social and economic context

7. PISA Contextual Questionnaires How they are used
This is how the PISA Analytic Framework is operationalised for data collection and analysis
Main data collection
Student Questionnaire, School Questionnaires and System Level*
Outcomes
Processes
Antecedents
Students
General recurring variables (all cycles)
Domain recurring variables (main domain every 3 years)
Extension variables (specific cycles)
System-level data
Classrooms
Schools
Country
Variables included in instruments:
This slide shows how the PISA Analytic Framework is operationalised through the data collection instruments.
Extended data collection procedures will inform the pilot trial with regards to how to maintain common constructs for cross-national comparability (and reporting) with additional (enhanced) data collection.
* Options include questionnaires geared for Teachers, Parents, Health, Time Use, ICT familiarity, Career

8. PISA Contextual Questionnaires System-level data collection
Standards and procedures for data collection in PISA
Questionnaires for non-OECD economies
Panel of experts to complete some data collection
These are examples of the types of information collected at the system level:
Ratio: students / teaching staff
Spending /student
Instruction time (by age)
Teachers’ salaries (by education, experience)
Evaluation & accountability
Educational stratification
Decision-making at different levels
School choice
Changes in education policy
Others (…)
Sample of relevant areas to address in developing contexts
Within-country disparities
Quality, supply and distribution of educational materials and resources
(Extra-)tuition
Parental support
Contractual arrangements of school teaching staff

9. PISA Contextual Questionnaires School questionnaires
School resources, policies and practices
This is the type of information collected through the school questionnaires, that can then be cross-referenced with other sources
These are the constructs that will be reviewed as part of the technical work of PISA for Development
Grades served
Public-Private
Funding sources
Community type
Enrollment (boys, girls)
Average class size
Non-native speakers
Grouping/tracking
Teachers, by qualifications
Teacher shortages
Physical building limitations
Instructional materials
Teacher attitudes, expectations, relations
Computer/network/web access
Absenteeism, disruptions
Teacher morale
Student morale
Teaching environment
Time for special programmes
Non-native speaker offerings
Admissions requirements
Assessment types
Uses of assessment
Assessment of teachers
Decision making authority
External decision makers

10. PISA Contextual Questionnaires Student questionnaires
Student attitudes, behaviours and approaches to learning
General and domain-related processes
Sense of belonging at school
Student-teacher relations at school
Disciplinary climate in classes
Teacher support in subject-specific classes
Out-of-school activities (homework, etc...)
Self and domain-related cognitions
Anxiety towards “study subject”
Instrumental motivation to learn “study subject”
Interest in and enjoyment of “study subject”
Self-efficacy in “study subject”
Self-concept in “study subject”
Learning strategies
Preference for competitive learning situations
Preference for co-operative learning situations
Control strategies
Perseverance
Others
Health status, Time use
(…)
These are examples of the types of
non-cognitive outcomes
on which PISA collects information
These are the constructs that will be reviewed as part of the technical work of PISA for Development
This slide makes the link between current PISA instruments and the enhancements that will be addressed through the PISA for Development.

11. PISA Contextual Questionnaires Examples of analysis
… How are questionnaire data used for relevant analysis and comparative policy insights?
(a few examples from PISA results)

12. Examples of analysis Range of performance among non-OECD countries
PISA Results for different years
Source: OECD (2010), PISA 2009 Results. Volume 5, Table V.2.1
ACER (2012), PISA 2009 Plus Results, Table A.2.
GNI per capita from 2011, World Bank Indicators. See:
Shanghai:

13. Schools with a similar socio-economic profile
Mexico
Examples of analysis School performance and socio-economic background
PISA 2009 Results
School performance and students’ socio-economic background within schools
Private school
Public school in rural area
Public school in urban area
Student performance and schools’ socio-economic background
Schools with a similar socio-economic profile
Student performance
This slide shows PISA 2009 results for the lowest-performing OECD country – Mexico.
These PISA 2009 results for Mexico show that even within “poor performers” that appear at the bottom of the “league tables”, PISA provides differentiation and relevant analysis.
PISA provides results in terms of performance in reading, mathematics and science on the PISA scales. On this chart, the vertical axis (y-axis) shows increasing levels of performance as one goes from bottom to top. On the bottom, the x-axis, a measure of socio-economic advantage and disadvantage is shown – as one moves from left to right, the relative advantage of students increases. The bubbles shown are the schools that participated in PISA 2009 in Mexico and their results relative to both cognitive outcomes (in this case reading, vertical scale) and socio-economic status (horizontally along the bottom).
The size of the bubbles represent the relative size of the students enrolled in the school. The dark blue bubbles, therefore, represent the relative position of the schools that participated in Mexico in PISA As you can see, the general tendency in terms of the relationship between socio-economic status of students in the schools and the average performance is correlated – the more advantaged students tends to result in higher average performance of schools. However, let’s take a closer look. (Animation).
Among schools that have a relatively similar socio-economic profile of students, shown by the vertical band, there is a considerable amount of variation in performance, and as we move right towards more advantaged students in schools, we see the variation continues. Why is it that some schools have much higher performance averages with students from similar neighbourhoods?
Looking at a cross-section of schools that have similar performance (horizontal band), even below the country average, we see that among these schools that can be considered to be “under-performing” there is also a considerable amount of variation in terms of the relative advantage and disadvantage of their students. Some of these schools, towards the left side of the chart, show results that are similar to schools with students from more privileged backgrounds (towards the right).
You see that some of them do very well even by OECD standards while others do quite poorly. If Mexican schools would achieve what Mexican schools show can be achieved, Mexico’s overall performance would be significantly higher. Similarly, you can look at the schools where rich parents sent their children, all marked in dark bubbles which tells you that these are private schools. But not all of them are of good quality.
Advantage
PISA Index of socio-economic background
Disadvantage

14. PISA Index of socio-economic background
Kyrgyzstan
Examples of analysis School performance and socio-economic background
PISA 2009 Results
School performance and students’ socio-economic background within schools
Private school
Public school in rural area
Public school in urban area
Student performance and schools’ socio-economic background
Student performance
OECD average
This slide shows results for the partner country Kyrgyzstan
that participated in PISA 2006 and These results are from PISA 2009.
This slide is the same type of figure as the previous one for Mexico. However, even when looking at these two very different countries – both low performing in terms of overall PISA results – there are several interesting points that the PISA data show:
Even between countries that can be considered “low performing”, PISA results show striking differences in terms of the relationship between socio-economic background of students and how this is associated with student performance.
PISA results also clearly underscore differences in variance within and between schools, as well as the relative performance of public, private and urban/rural schools. There is also clearly a correlation in terms of size of rural area schools compared to urban schools which also tend to be higher performing.
Again, as in the case of Mexico, PISA data for Kyrgyzstan shows that even between schools that serve students from similar socio-economic backgrounds, there is a large variation in performance. And among schools that serve very different types of students, there can be similar results.
Advantage
PISA Index of socio-economic background
Disadvantage

15. Examples of analysis Policy findings from contextual information
Policies and practices
Learning climate
Discipline
Teacher behaviour
Parental pressure
Teacher-student relationships
Dealing with heterogeneity
Grade repetition
Prevalence of tracking
Expulsions
Ability grouping (all subjects)
Standards /accountability
Nat. examination
Standardised tests
Posting results
Governing schools
School autonomy (content)
Choice and competition
Private schools
Managing resources
Prioritising pay
Student-staff ratios
Length of pre-school
Policy
System   R
School 
Equity E
Examples of analysis Policy findings from contextual information
Policy briefs on these issues can be found in the PISA in Focus series
Let me briefly summarise the influences that we have measured in PISA.

16. Commitment to universal achievement
Goals, gateways, instructional systems
Capacity at point of delivery
Incentives and accountability
Resources where they yield most
A learning system
Coherence
Based on the analysis of cognitive outcomes and contextual information
Examples of analysis Policy insights using contextual information
Lessons from PISA from successful education systems (equitable and improvers)
Key Messages:
- Findings from different cycles of PISA and trends over time
This is the type of analysis that can be tailored for a particular country – Strong Performers/Successful reformers series of publications (show examples)
The analysis for countries participating in PISA for Development will be tailored based on the relevance and policy priorities established by partners.
Part of the value added will come from looking closely at how some of these and other policy issues actually present themselves in countries – and the pointers to possible policies and interventions.
Original notes on slide:
First, there is no question that most nations declare that education is important. But the test comes when these commitments are weighed against others. How do countries pay teachers, compared to other highly-skilled workers? How are education credentials weighed against other qualifications when people are being considered for jobs? Would you want your child to be a teacher? How much attention do the media pay to schools and schooling? What we have learned from PISA is that in high performing systems political and social leaders have persuaded citizens to make choices that show they value education more than other things.
But placing a high value on education is only part of the equation. Another part is belief in the possibilities for all children to achieve success. In some countries, students are separated into different tracks at an early age, reflecting a notion shared by teachers, parents and citizens that only a subset of the nation’s children can or need to achieve world class standards. Our analysis shows that systems that track students in this way, based differing expectations for different destinations, tend to be fraught with large social disparities.
By contrast, the best performing systems deliver strong and equitable learning outcomes across very different cultural and economic contexts. In Finland, Japan, Singapore, Shanghai-China and Hong Kong-China, parents, teachers and the public at large share the belief that all students are capable of achieving high standards and need to do so, and they provide great examples for how public policy can support the achievement of universal high standards.

17. PISA Contextual Questionnaires How they are developed
‘Crowd sourcing’ and collaboration
PISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies…
… guided by governments on the basis of shared policy interests
Cross-national relevance and transferability of policy experiences
Emphasis on validity across cultures, languages and systems
Frameworks built on well-structured conceptual understanding of assessment areas and contextual factors
Continuous review and updating of assessment frameworks (cycles)
Guided by participating countries and various expert groups formed by leading international experts in different areas related PISA
Technical Advisory Group, Reading Expert Group, Mathematics EG, Science EG, Questionnaire EG, and international contractors
SMEG (subject matter expert group)
REG (reading)
MEG (math)
CPEG (collaborative problem solving)
SEG (science)
QEG (questionnaire)
EL (environmental)
FL (financial)
How many people are involved?
Major domains - 6 Core members and 8 Extended members
Minor domains - 2 Core members and 2 Extended members

18. PISA Contextual Questionnaires How they are developed
PISA 2012 Questionnaires
Student questionnaire: Rotated design (total of 65 questions – 25 core and 3 blocks of 27 questions)
School questionnaire: 33 questions (5 additional questions on financial education at school)
Parental questionnaire: 25 questions
ICT questionnaire: 12 questions
Education careers: 14 questions
PISA 2012 Mathematics Framework
What mathematical content knowledge can we expect of 15-year-old students?
What processes do students engage in when solving contextual mathematical problems, and what capabilities do we expect students to be able to demonstrate as their mathematical literacy grows?
In what contexts is mathematical literacy able to be observed and assessed?
Linkages between cognitive and non-cognitive outcomes

19. PISA Contextual Questionnaires How they are developed
Purpose of the PISA Field Trial (for Main Study)
Selection of cognitive test items to be included in the PISA Main Study instruments (e.g. countryXcountry DIF, gender DIF)
Validation of psychometric equivalence of translated instruments (and adaptations from source versions)  issues addressed
Identify constructs from contextual questionnaires (indices and variables) that are associated with performance for inclusion in the Main Study
Determine cross-national validity of the questions in the questionnaire
Technical standards ensure that data from participating countries are internationally comparable

20. PISA Contextual Questionnaires What will PISA for Development seek to do?
Adapt and enhance contextual instruments to “better fit” diverse contexts found in developing countries while maintaining the comparability with the main PISA scales and international results
Adapt existing constructs (and indices and variables)
Identify and introduce new constructs (indices and variables)
Confirm and validate enhancements through field trials and main data collection in participating countries

21. Examples of variables and indices
PISA Contextual Questionnaires What will PISA for Development seek to do?
Examples of variables and indices
Student Questionnaire
Section 1: ABOUT YOU
Section 2: YOUR FAMILY AND YOUR HOME
Section 3: YOUR READING ACTIVITIES
Section 4: LEARNING TIME
Section 5: YOUR SCHOOL
Section 6: YOUR SCHOOL LESSONS AND SUBJECTS
Section 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTS
Section 8: YOUR VIEWS ON <BROAD SCIENCE>
Section 9: CAREERS AND <BROAD SCIENCE>
Section 10: YOUR MATHEMATICS EXPERIENCES

22. PISA Contextual Questionnaires Student questionnaires
Examples of variables and indices
Student Questionnaire
SECTION 2: YOUR FAMILY AND YOUR HOME
2009-ST08: Family structure
2009-ST09: Mother’s main job (ESCS, HISEI, BMMJ)
2009-ST10: Mother’s education (ESCS, HISCED, PARED, MISCED)
2009-ST11: Mother’s qualifications (ESCS, HISCED, PARED, MISCED)
2009-ST12: Mother’s employment status (ESCS, HISEI, BMMJ)
2009-ST13: Father’s main job (ESCS, HISEI, BMFJ)
2009-ST14: Father’s education (ESCS, HISCED, PARED, FISCED)
2009-ST15: Father’s qualifications (ESCS, HISCED, PARED, FISCED)
2009-ST16: Father’s employment status (ESCS, HISEI, BFMJ)
2009-ST17: Country of birth for student and parents (IMMIG)
2009-ST18: Age at arrival in country of test
2009-ST19: Language spoken at home
2009-ST20: Home resources (ESCS, HOMEPOS, WEALTH, HEDRES, CULTPOS)
2009-ST21: Family wealth (ESCS, HOMEPOS, WEALTH)
2009-ST22: Books in home (ESCS, HOMEPOS)

23. PISA Contextual Questionnaires Student questionnaires – examples of indices
PISA index of Economic, Social and Cultural Status
(ESCS)
Comparability across countries (not a country-specific income indicator)
WEALTH
HISEI
Highest occupational status of parents
CULTPOSS
Principal component Analysis
The reliability of the index ranged from 0.41 to 0.81
Scale indices are the variables constructed through the scaling of multiple items. Scaled using a weighted maximum likelihood estimate (WLE).
Simple indices are the variables that are constructed through the arithmetic transformation or recoding of one or more items, in
exactly the same way across assessments.
The index of home possessions (HOMEPOS) comprises all items on the indices of WEALTH, CULTPOSS and HEDRES, as well as books in the home recoded into a four-level categorical variable (0-10 books, or books, or books, more than 500 books).
The index of home possessions, obtained by asking students whether they had a desk at which they studied at home, a room of their own, a quiet place to study, educational software, a link to the Internet, their own calculator, classic literature, books of poetry, works of art (e.g. paintings), books to help them with their school work, a dictionary, a dishwasher, a DVD player or VCR, three other country-specific items and the number of cellular phones, televisions, computers, cars and books at home.
HOMEPOS
Home possessions
HEDRES
# of books in home
PARED
Highest educational level of parents

24. PISA Contextual Questionnaires Student questionnaires – examples of indices
PISA index of Economic, Social and Cultural Status
(ESCS)
Index of family wealth
a room of their own, a link to the Internet, a dishwasher (treated as a country-specific item), a DVD player, and three other country-specific items (some items in ST20), and their responses on the number of cellular phones, televisions, computers, cars and the rooms with
a bath or shower (ST21).
Index of cultural possessions
classic literature, books of poetry and works of art (ST20)
WEALTH
CULTPOSS
HEDRES
# of books in home
Index of home educational resources
For example, availability of a desk, a quiet place to study, a computer that students can use for schoolwork, educational software, books to help students, technical reference books and dictionary (some items in ST20)
Principal component Analysis
The reliability of the index ranged from 0.41 to 0.81
Scale indices are the variables constructed through the scaling of multiple items. Scaled using a weighted maximum likelihood estimate (WLE).
Simple indices are the variables that are constructed through the arithmetic transformation or recoding of one or more items, in
exactly the same way across assessments.
The index of home possessions (HOMEPOS) comprises all items on the indices of WEALTH, CULTPOSS and HEDRES, as well as books in the home recoded into a four-level categorical variable (0-10 books, or books, or books, more than 500 books).
The index of home possessions, obtained by asking students whether they had a desk at which they studied at home, a room of their own, a quiet place to study, educational software, a link to the Internet, their own calculator, classic literature, books of poetry, works of art (e.g. paintings), books to help them with their school work, a dictionary, a dishwasher, a DVD player or VCR, three other country-specific items and the number of cellular phones, televisions, computers, cars and books at home.

25. 2006-ST17: Self-efficacy in science (SCIEEFF)
PISA Contextual Questionnaires Student questionnaires – examples of indices
Self-efficacy in science and mathematics
2006-ST17: Self-efficacy in science (SCIEEFF)
2003-ST31: Self-efficacy in mathematics (MATHEFF)
 Stays the same in PISA 2012
Relationship between constructs, indices and variables (questions)
Students’ self-belief is another context variable that shows a strong correlation with student’s performance in the subjects. Similar to the level of engagement with the subjects, self-belief can be an important part of improving the performance. Self-belief is measured for mathematics and science in PISA 2003 and 2006, but not for reading in PISA 2009.
Self-belief is measured in terms of self-efficacy (how much students believe in their own ability to handle tasks effectively and overcome difficulties) and self-concept (how much students believe in their own academic abilities).

26. Self-efficacy in science
PISA Contextual Questionnaires Student questionnaires – examples of indices
Self-efficacy in science
2006-ST17: Self-efficacy in science (SCIEEFF)
06-ST17
Q42
How easy do you think it would be for you to perform the following tasks on your own?
(Please darken only one circle in each row.)
I could do this easily
I could do this with a bit of effort
I would struggle to do this on my own
I couldn’t do this a)
Recognise the science question that underlies a newspaper report on a health issue ●1 ●2 ●3 ●4 b)
Explain why earthquakes occur more frequently in some areas than in others c)
Describe the role of antibiotics in the treatment of disease d)
Identify the science question associated with the disposal of garbage e)
Predict how changes to an environment will affect the survival of certain species f)
Interpret the scientific information provided on the labels of food items g)
Discuss how new evidence can lead you to change your understanding about the possibility of life on Mars h)
Identify the better of two explanations for the formation of acid rain
Students’ self-belief is another context variable that shows a strong correlation with student’s performance in the subjects. Similar to the level of engagement with the subjects, self-belief can be an important part of improving the performance. Self-belief is measured for mathematics and science in PISA 2003 and 2006, but not for reading in PISA 2009.
Self-belief is measured in terms of self-efficacy (how much students believe in their own ability to handle tasks effectively and overcome difficulties) and self-concept (how much students believe in their own academic abilities).

27. Instrumental motivation to learn science
PISA Contextual Questionnaires Student questionnaires – examples of indices
Instrumental motivation to learn science
2006-ST35: Instrumental motivation to learn science (INSTSCIE)
06-ST35
Q46
How much do you agree with the statements below?
(Please darken only one circle in each row.)
Strongly agree
Agree
Disagre e
Strongly disagree a)
Making an effort in my science class(es) is worth it because this will help me in the work I want to do later on ●1 ●2 ●3 ●4 b)
What I learn in my science class(es) is important for me because I need this for what I want to study later on c)
I study science because I know it is useful for me d)
Studying science is worthwhile for me because what I learn will improve my career prospects e)
I will learn many things in my science class(es) that will help me get a job

28. PISA for Development How will we do this?
Identify constructs in need of adaptation and important constructs that are missing
Obtain input and advice from experts in the field (grounded in developing-country contexts)
Obtain input and guidance from participating countries
Draw on existing work and empirical evidence
Test the constructs (indices and variables) to ensure comparability with PISA scales

29. Contextual questionnaires
PISA for Development
Technical Strand 1:
Contextual questionnaires

30. PISA for Development Questions for discussion
Are the intentions and rationale of this strand of work clear and understood?
Which components of the PISA questionnaires appear to demonstrate the greatest need of adaptation?
Where do you see the largest hurdle to overcome?
What existing studies and evidence should contribute to this strand of work?
For example:
- Capturing a wider range of socio-economic and cultural contexts
International and national development of the PISA context questionnaires
- Measuring the knowledge and skills of children not in school
- Ensuring the relevance and validity of the skills that are measured
- Balance between country-specific contexts and international comparability