Reporting International Test Data Ch. 3 Measuring Civic Knowledge and Understanding ICCS NRC meeting Madrid, Feb 2010

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge Instrument Described scale Results

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge - Instrument 80 item (reduced to 79 - CI2HRM2 removed) 73 MCQ, 6 constructed response 62 new, 17 CIVED trend Items showing extreme item by country interaction (>1.5 logits) removed from scaling for estimates of test scores in individual countries Report to include example release items illustrating range of item types and content (including % correct by country and scale location)

NRC Meeting Madrid February, 2010 rp = 0.62 (relative shift item difficulties = ln(0.62/0.36) ≈0.49 logits Measuring Civic and Citizenship Knowledge – Described Scale

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Described Scale Item map (ordered item descriptors)Item map

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Described Scale Item map (ordered item descriptors)Item map L1 L2 L3 B1 0.8 logits

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Described Scale 2 Items 22 Items 16 Items 39 Items

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Described Scale Level 1 Students working at Level 1 demonstrate familiarity with equality, social cohesion and freedom as principles of democracy. They relate these broad principles to everyday examples of situations in which protection of or challenge to the principles are demonstrated. Students also demonstrate familiarity with fundamental concepts of the individual as an active citizen: they recognise the necessity for individuals to obey the law; they relate individual courses of action to likely outcomes; and they relate personal characteristics to the capacity of an individual to effect civic change.

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Described Scale Level 2 Students working at Level 2 demonstrate familiarity with the broad concept of representative democracy as a political system. They recognise ways in which institutions and laws can be used to protect and promote a society's values and principles. They recognise the potential role of citizens as voters in a representative democracy, and generalise principles and values from specific examples of policies and laws (including human rights). Students demonstrate understanding of the influence that active citizenship can have beyond the local community. They generalise the role of the individual active citizen to broader civic societies and the world.

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Described Scale Level 3 Students working at Level 3 make connections between the processes of social and political organisation and influence, and the legal and institutional mechanisms used to control them. They generate accurate, hypotheses on the benefits, motivations and likely outcomes of institutional policies and citizens' actions. They integrate, justify and evaluate given positions, policies or laws based on the principles that underpin them. Students demonstrate familiarity with broad international economic forces and the strategic nature of active participation.

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Described Scale Level 1: broad concepts; “big ideas”; mechanistic Specificity Level 2: interconnectedness of systems; wider sphere of citizen influence Evaluation Level 3: holistic; strategic

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Results Comparison of means Means range from 380 to 576 –Levels 1 to 3

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Results Comparison of means Multiple comparison

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Results Level Level boundary Items (N) Items (%) Students All (%) Students Max (%) Students Min (%) > > >395 B Comparison of levels

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Results LevelNo. countries with max % students in level B12 Comparison of levels

NRC Meeting Madrid February, 2010 Measuring Civic and Citizenship Knowledge – Results LevelNo. countries > 60% students in levels B Comparison of levels

NRC Meeting Madrid February, 2010 Gender differences in achievement International means –Girls 511 –Boys 489 –Difference is statistically significant Mean for girls higher in all countries –Statistically significantly higher in 32 countries Range of mean differences 2 to 48 (median =22) Measuring Civic and Citizenship Knowledge – Results

NRC Meeting Madrid February, 2010 Trend Comparisons Calculated only on the basis of 17 trend items (different metric to ICCS) Dimensionality conflating results for complete scaling from both ends – CIVED through representativeness of trend items –ICCS through broadening of framework and constructs Measuring Civic and Citizenship Knowledge – Results

NRC Meeting Madrid February, 2010 Comparisons over time 17 countries used same translations of CIVED items for comparison CIVED population mean = 500 ICCS population mean = 484 –Difference is statistically significant 5 countries (stat sig one country) 11 countries (stat sig six countries) Measuring Civic and Citizenship Knowledge – Results