IEA INTERNATIONAL CIVIC AND CITIZENSHIP EDUCATION STUDY (ICCS) A new comparative study of civic and citizenship education AERA Annual Meeting New York, 2008

AERA Annual Meeting March 2008 Symposium Outline Introduction Concept and Design Assessing Knowledge, Background & Perceptions Collecting School and Teacher Data Studying CCE in the European Context Citizenship Competencies in the Latin American Region Discussant General Discussion

AERA Annual Meeting March 2008 Purposes of study International comparison of outcomes of civic and citizenship education at Grade 8 Responses to challenges in civic and citizenship education in changed contexts since CIVED in 1999

AERA Annual Meeting March 2008 Instrumentation International cognitive test International student questionnaire –Background –Perceptions Teacher questionnaire School questionnaire National Contexts Survey Regional student instruments –European –Latin American –Asian (under development)

AERA Annual Meeting March 2008 Participating countries

AERA Annual Meeting March 2008 ICCS Current Status 38 countries participating Field trial –October 07 to January 08 Field trial data –19,800 students (30 per school) –10,500 teachers (16 per school) –668 schools –29 countries

AERA Annual Meeting March 2008 Next steps Analysis of Field Trial data Revision of instruments Final forms Main Survey –Southern hemisphere (end 2008) –Northern hemisphere (early 2009) Data compilation and analysis (2009) Reporting (2010)

AERA Annual Meeting March 2008

Concept and Design of the International Civic and Citizenship Study Julian Fraillon and Wolfram Schulz (ACER) AERA Annual Meeting New York, 2008

AERA Annual Meeting March 2008 Contents ICCS Overview The ICCS Assessment Framework Summary Introduction ICCS Assessment Framework links to CIVED ICCS Assessment Framework Structure ICCS Assessment Framework Domains - Detail

AERA Annual Meeting March 2008 ICCS Overview The purpose of the International Civic and Citizenship Education Study (ICCS) is to investigate, in a range of countries, the ways in which young people are prepared and consequently ready and able to undertake their roles as citizens. The study will report on: –student achievement in a test of knowledge and conceptual understandings in civics and citizenship. –data about student activities, dispositions, and attitudes related to civic and citizenship education. –contextual data that will help to explain variation in the outcome variables.

AERA Annual Meeting March 2008 ICCS Overview The study builds on the previous IEA studies of civics and citizenship education. The study is underpinned by six research questions that address the degree to which Grade 8 (or equivalent)* students are ready and able to undertake their roles as citizens AND the contextual factors that play a role in this readiness. * Mean age of student not less than 13.5 years

AERA Annual Meeting March 2008 ICCS Assessment Framework The Assessment Framework has been developed in consultation with experts and iterative review with ICCS country representatives. The Assessment Framework comprises two main components 1.The civics and citizenship framework outlines the aspects to be addressed when collecting the outcome measures through the cognitive test and the student perceptions questionnaire. 2.The contextual framework provides a mapping of context factors that might influence outcome variables and explain their variation.

AERA Annual Meeting March 2008 ICCS AF – Links to CIVED Key conceptualisation of student learning: 1.The student as the central agent in their civic world, with both an influence on and being influenced by their multiple connections with their civic communities. 2.Young people learn about civics and citizenship through their interactions with their multiple civic communities and not only through formal classroom instruction.

AERA Annual Meeting March 2008 ICCS AF – Links to CIVED Conceptually similar survey design matrices that link question types to civics and citizenship content. Some secure trend items from CIVED as a concrete scaling link between the two studies.

AERA Annual Meeting March 2008 ICCS AF - Structure The civics and citizenship framework consists of: –Four content domains –Four affective/behavioural domains –Two cognitive domains

AERA Annual Meeting March 2008 ICCS AF – Content Domains The four content domains in the ICCS Civics and Citizenship Framework are: 1.Civic society and systems 2.Civic principles 3.Civic participation 4.Civic identities.

AERA Annual Meeting March 2008 ICCS AF – Content Domains

AERA Annual Meeting March 2008 ICCS AF – Affective Behavioural Domains The four affective-behavioural domains in the ICCS Civics and Citizenship Framework are: 1.Value beliefs 2.Attitudes 3.Behavioural intentions 4.Behaviours.

AERA Annual Meeting March 2008 ICCS AF – Cognitive Domains The two cognitive domains in the ICCS framework are: 1.Knowing 2.Reasoning and analyzing.

AERA Annual Meeting March 2008 ICCS AF Design Matrix Content Domain 1: Civic society and systems Content Domain 2: Civic principles Content Domain 3: Civic participation Content Domain 4: Civic identities Cognitive Domains Knowing IIIIIIIV Analysing and reasoning VVIVIIVIII Affective- behavioural Domains Value beliefs ABCD Attitudes EFGH Behavioural intentions I Behaviours J

AERA Annual Meeting March 2008 ICCS AF Contextual Domains Wider community Educational system History and culture School/classroom: Characteristics Composition Resources Home environment: Family background Social group Indicators related to: Civic society and systems Civic principles Civic participation Civic identities Wider community Educational policies Political events School/classroom: Instruction Governance Student: Socialization & learning Home environment: Communication Activities AntecedentsProcessesOutcomes Student: Characteristics

AERA Annual Meeting March 2008 ICCS AF Contextual Domains Level of...AntecedentsProcessesOutcomes National and other communities NCQ & other sources: Democratic history Structure of education NCQ & other sources: Intended curriculum Political developments StT & StPQ & StBQ: Test results Student perceptions Student behaviours School/classroomScQ & TQ: School characteristics Resources ScQ & TQ: Implemented curriculum Policies and practices StudentStBQ: Gender Age StBQ: Learning activities Practiced engagement Home environmentStBQ: Parent SES Ethnicity Language Country of birth StBQ: Communication Peer-group activities

AERA Annual Meeting March 2008 Additional information The paper: OR The paper and/or the Assessment Framework:

AERA Annual Meeting March 2008

Assessing Student Knowledge, Background and Perceptions in the International Civic and Citizenship Study AERA Annual Meeting New York, March 2008 Wolfram Schulz (ACER) Falk Brese (IEA DPC)

AERA Annual Meeting March 2008 Contents of presentation Overview of areas assessed with student instruments in ICCS General principle for scaling and analysis Review of measurement invariance Some preliminary results

AERA Annual Meeting March 2008 Student instruments Student cognitive test –One hour testing time –Six rotated booklets –Multiple-choice, true/false and open-ended items Student questionnaire –Questions on student background and items measuring student perceptions –Different formats (with and without “don’t know” category) –Three forms with different combinations of item batteries

AERA Annual Meeting March 2008 Cognitive Test Four content domains –Civic society and systems –Civic principles –Civic participation –Civic identities Two cognitive domains –Knowing –Reasoning and analysing Cluster with CIVED link items –Multiple-choice items

AERA Annual Meeting March 2008 Student questionnaire (perceptions) Four content domains (as above) –Civic society and systems –Civic principles –Civic participation –Civic identities Affective-behavioural domains: –Value beliefs –Attitudes –Behavioural intentions –Behaviours

AERA Annual Meeting March 2008 Value beliefs Democratic values –Set of modified CIVED items Citizenship values –Items already used in CIVED Students’ acceptance of socially undesirable behaviour

AERA Annual Meeting March 2008 Attitudes (1) Students' self-cognitions related to Civics and Citizenship –Interest in political events and social issues –Self-concept regarding political participation (internal efficacy) –Citizenship self-efficacy –Perceptions of distinctiveness compared to others in the country –Sense of belonging to communities

AERA Annual Meeting March 2008 Attitudes (2) Students' attitudes towards rights and responsibilities –Attitudes towards gender rights –Attitudes toward rights of ethnic/racial groups –Attitudes toward immigrants –Attitudes towards policies to avert threats to democracy

AERA Annual Meeting March 2008 Attitudes (3) Students' attitudes towards institutions –Trust in institutions –Responsiveness of the political system (external efficacy) –Confidence in student participation at school –Attitudes towards one’s nation –Satisfaction with country’s achievements

AERA Annual Meeting March 2008 Behavioural intentions Preparedness to participate in forms of civic protest –Legal –Illegal Behavioural intentions regarding future political participation as adult –Electoral participation –Active political participation Behavioural intentions regarding future participation as a young person

AERA Annual Meeting March 2008 Behaviours Involvement in civic-related participation in the community –Only minority of students is active in the community! Involvement in civic-related activities at school

AERA Annual Meeting March 2008 Student questionnaire: background (1) Context of schools and classrooms –Classroom climate for civic and citizenship education at school –Perceptions about students’ influence on decision-making at school –Student perceptions of school climate

AERA Annual Meeting March 2008 Student questionnaire: background (2) Context of the home environment –Parental socio-economic status Parental occupation Parental education Household possessions –Cultural/ethnic background Country of birth (students and parents) Language use at home Self-reports on ethnicity (country-specific)

AERA Annual Meeting March 2008 Student questionnaire: background (3) Context of the home environment –Family composition –Indicators of social interaction Discussion of social/political issues Media information Social activities

AERA Annual Meeting March 2008 Student questionnaire: background (4) Context of the individual student –Age –Gender –Expected educational level with reference to ISCED classification –Out-of-school activities

AERA Annual Meeting March 2008 Field trial data analysis All ICCS instruments were tested in international field trial (October to December 2007) Samples of about 600 students (from about 25 schools) per country Data from 29 countries in current (preliminary analysis)

AERA Annual Meeting March 2008 Purposes of field trial data analysis Inform on item selection for the main study Inform on most appropriate formats for questionnaire items Inform on selection of constructs –Predictive validity –Cross-country validity

AERA Annual Meeting March 2008 Test data analysis Item Response Theory –One-parameter Rasch model Review of item characteristics –Scalability of items –Dimensionality of items Review of differential item difficulty –Gender DIF –Item-by-country interaction Review of link items –Relative item difficulty Review of test length

AERA Annual Meeting March 2008 Analysis of Coder reliability 10 percent of cognitive items are open-ended and need to be coded Necessary to –inform the development of scoring rubrics –inform the translation verification of scoring rubrics –examine the extent of between country differences in coders

AERA Annual Meeting March 2008 Questionnaire data analysis Review percentages in each category and missing responses Analyse the dimensional structure of item batteries Exploratory and confirmatory factor analyses Determine scaling properties of items and constructs IRT scaling (Partial Credit Model) Review relationships between constructs and between constructs and test results

AERA Annual Meeting March 2008 Ways of Construct Validation in International Studies Assessing item dimensionality through confirmatory factor analysis and compare fit across countries Testing parameter invariance –by constraining item loadings (SEM) –by reviewing item-by-country interaction (IRT) –by constraining item parameters (IRT) Analysing relationships with reference variables

AERA Annual Meeting March 2008 Some first results Field trial results are encouraging and indicate a high quality of most of the item material Cognitive test items have generally good scaling properties Most of the questionnaire items have good measurement properties

AERA Annual Meeting March 2008 Match of test and abilities

AERA Annual Meeting March 2008 Questionnaire scale reliabilities (1)

AERA Annual Meeting March 2008 Questionnaire scale reliabilities (2)

AERA Annual Meeting March 2008 Multiple regression models for pooled sample

AERA Annual Meeting March 2008 The paper: The paper and/or the Assessment Framework:

AERA Annual Meeting March 2008

Collecting School and Teacher Data in International Civic and Citizenship Study Bruno Losito and Gabriella Agrusti (Universita Roma Tre) AERA Annual Meeting New York, 2008

AERA Annual Meeting March 2008 Civic and citizenship education in ICCS –Knowledge and understanding –Attitudes –Behaviors and behavioral intentions –Value beliefs

AERA Annual Meeting March 2008 ICCS Research Question What aspects of schools and education systems are related to achievement in and attitudes to civics and citizenship including: a)curriculum or program content structure and delivery b)teaching practices such as those that encourage higher order thinking and analysis in relation to civics and citizenship c)aspects of school organisation including opportunities to contribute to conflict resolution, participate in governance processes, and be involved in decision making. ScQ and TcQ Rationale

AERA Annual Meeting March 2008 Relevance of the school environment for CCE (school as a democratic learning environment) Whole school student experience Actual possibility to exercise citizen rights and responsibilities within the school School culture and classroom climate ScQ and TcQ Rationale

AERA Annual Meeting March 2008 School heads and teachers as “key factors” in CCE Teachers of all subjects (not only CCE reated subjects teachers) – international option ScQ and TcQ Rationale

AERA Annual Meeting March 2008 ScQ Content School questionnaire content, constructs and variables Five research areas –School characteristics – School autonomy – School community relationships – School environment – CCE at school (implementation of CCE at school level)

AERA Annual Meeting March 2008 ScQ Content Demographics School characteristics –Type of school – School enrolment – Number of teachers School autonomy (in management and in educational planning)

AERA Annual Meeting March 2008 ScQ Content School-community relationships –resources available to students in the local community – issue of social tension within the local community – issues of social tension within the school School environment – Teachers, parents participation – Sense of belonging to the school (teachers, students, non teaching staff)

AERA Annual Meeting March 2008 ScQ Content CCE at school (aims of CCE, responsibilities in CCE

AERA Annual Meeting March 2008 TcQ Content Teacher questionnaire content, constructs and variables Teacher characteristics (demographics, work experience) Teachers confidence in teaching methods and approaches

AERA Annual Meeting March 2008 TcQ Content CCE at school -Aims of CCE -CCE teaching and learning activities -Teachers’ confidence in teaching CCE topics -Improvement of CEE

AERA Annual Meeting March 2008 TcQ Content School environment – teachers’ participation in school governance – importance given to students’ opinions – students’ opportunity to participate in CCE activities – teachers’ participation in CCE related activities –Students’ sense of belonging to the school

AERA Annual Meeting March 2008 TcQ Content School community relationships Teaching strategies Classroom climate

AERA Annual Meeting March 2008 Development process –Involvement of NRCs (country contexts) Uses of school and teacher data – reporting data at the country level – aggregated school level data Descriptive and predictive levels

AERA Annual Meeting March 2008

AERA Annual Meeting March 2008 Studying Civic and Citizenship Education in the European context David Kerr and Joana Lopes NFER, England AERA Annual Meeting New York, ICCS – A new comparative study of civic and citizenship education

AERA Annual Meeting March 2008 European context Democratic deficit Fall of Soviet regimes Major events security and cohesion Rapid movement of peoples European enlargement Growing interest in civic and citizenship education

AERA Annual Meeting March 2008 European context Interest in civic and citizenship education: Creating and using evidence base Identifying and measuring outcomes Developing effective policies and practices

AERA Annual Meeting March 2008 Purpose of European module ICCS Assessment Framework To investigate specific, Europe-related issues

AERA Annual Meeting March 2008 European Context in ICCS International instruments  European items National Contexts Survey (NCS)  European countries European module

AERA Annual Meeting March 2008 EM development Consultative, iterative process Influences:  Individual European countries  Council of Europe  European Commission  Centre for research in Lifelong Learning (CRELL)

AERA Annual Meeting March 2008 Process of construction of EM 2006/07: Definition of prioritiesMay’07: Map of constructs & Version 1June’07: Draft instrument & Version 2July’07: Pilot (Version 3)Aug’07: Field Trial (Version 4)

AERA Annual Meeting March 2008 Knowledge Test  European institutions and laws  particularly European Union (EU)-related Knowledge self-assessment

AERA Annual Meeting March 2008 Questionnaire Affective-behavioural domain Value Beliefs Content Domain Equal opportunities within European countries Civic principles Political and economical organisation of European countries Civic society and systems Enlargement of the European Union Civic society and systems

AERA Annual Meeting March 2008 Questionnaire Affective-behavioural domain Attitudes Content Domain European self-identityCivic identities Openness to other European countries/cultures Civic principles Learning European languages Civic principles Freedom of travel, settlement and work within Europe Civic principles

AERA Annual Meeting March 2008 Questionnaire Affective-behavioural domain Behaviours Content Domain Participation in European- level groups or events Civic participation Interpersonal communication behaviours Civic participation

AERA Annual Meeting March 2008 Questionnaire Knowledge of European foreign languages (self-assessed) Confidence self-assessment

AERA Annual Meeting March 2008 Next steps Current Field Trial analysis Jun’08 Version 5; NRCs meeting 2009 Main study (Version 6) 2010 Reporting of outcomes

AERA Annual Meeting March 2008 Outcomes International report European regional report (likely)  European module  International instruments - European items  National contexts survey report – European countries

AERA Annual Meeting March 2008

The Evaluation of Citizenship Competencies in the Latin American Region Fernando Reimers (Harvard University) AERA Annual Meeting New York, 2008

DISCUSSION IEA International Civic and Citizenship Education Study (ICCS) Judith Torney-Purta University of Maryland AERA Annual Meeting New York, 2008

ICCS: A new comparative study of civic and citizenship education Questions and Discussion ?? !! ** ##