Proposed Cognitive Test for the Main Study National Research Coordinators Meeting Windsor, June 2008

NRC Meeting Windsor June 2008 Feedback from group discussions, comments, recommendations Contents ≅ 50min Group discussion of specific recommendations ≅ 60min ≅ 70 min First impressions Summary of recommendations Possible outcomes Item deletions, minor changes.

NRC Meeting Windsor June 2008 First impressions Items worked well, can comprise single scale (NRC 2, p10) CIVED trend set can be scaled with and used for single (NRC 2 pp16,17) Test time per item seemed appropriate according to ‘not reached’ – still to be investigated further No pattern of gender DIF by item type or substance (NRC 2, p12) Some patterns of ‘regional’ item-by-country interaction (NRC 2, p15)

NRC Meeting Windsor June 2008 First impressions (cont.) Open-ended response items showed higher ‘non-attempts’ and slightly more variability (item-by-country interaction and opportunity for inter-rater reliability issues) (NRC 2, p17) Targeting good (NRC 2, p9)p9

NRC Meeting Windsor June 2008 Open-ended response item scoring guides Revision: –Structural changes informed by suggestions from countries* –Addition of “real” student responses (thank you to Julie Nelson at NFER) –Addition of relevant scoring codes on the advice of countries* * Feedback from NRCs through questions, voluntary reports and field-trial operations survey. – Finalisation of scoring guides at the marker training seminar in Hamburg

NRC Meeting Windsor June 2008 Summary of Item Recommendations RecommendationMCQ Include unchanged65 + 6* Include only with change11 Delete3 + 1* Include and consider possible change2 Possible deletion or inclusion with change0 Possible delete2 + 5* Total90 OE Total *T/F

NRC Meeting Windsor June 2008 Possible Outcomes Shorten test to 45 minutes Increase number of clusters to 77 Delete two open-ended response items and approximately 15 MCQ to maintain test-taking rate. Some updates on meeting documents resulting from further analyses and PAC meeting. –True/False items (NRC 5, pp 25-26)True/False items –PAC recommendations/suggestions (tabled document)

NRC Meeting Windsor June 2008 Latest News Some updates on meeting documents resulting from further analyses and PAC meeting. –True/False items (NRC 5, pp 25-26)True/False items –PAC recommendations/suggestions (tabled document)

NRC Meeting Windsor June 2008 Part 2 Group discussions of items

NRC Meeting Windsor June 2008 Specific Conclusions Differences in adult and target grade student concepts of ‘Human Rights’ Problems of assessing understanding of representative democracy using MCQ Item-by-item recommendations and comments

NRC Meeting Windsor June 2008 Context of Group Discussions Assessment framework developed in consultation with NRCs and other experts before the field-trial. All countries invited to submit items before field-trial All items made available for online feedback by NRCs before the field-trial All items revised on the basis of the online feedback for the field-trial and approved for inclusion at the NRC meeting in Rome.

NRC Meeting Windsor June 2008 Context for Group Discussions Field-trial empirical data collected from 31 countries and approx. 19, 000 students Recommendations for changes to instrument are based on empirical (and some qualitative) data from the field trial.

NRC Meeting Windsor June 2008 Context for Group Discussions The only changes that can be made to the cognitive instrument now are: 1.Deletion of items 2.Minor changes to items only if we are very confident that a change will fix a problem (and not introduce a new problem) 3.Amendments to the scoring guides

NRC Meeting Windsor June 2008 Context for Group Discussions All items with “good” data are to be included in the MS instrument Groups to consider each of the ISC recommendations and make one of three recommendations for each: 1.Accept definitive ISC recommendation 2.Reject definitive ISC recommendation (delete item) 3.Offer alternative “solution” to ISC recommendation

NRC Meeting Windsor June 2008 Session 2 – Until lunch 1.30 minutes – finish group discussion 2.60 minutes – BRIEF presentation from each group NOTE: Please submit all your comments and feedback to us (me) and we will include them in our deliberations regarding the final instrument.

NRC Meeting Windsor June 2008 Part 3 Feedback from group discussions, comments, recommendations

NRC Meeting Windsor June 2008 Booklet rotation Clust. Booklet #Pos 1Pos 2Pos 3 A1ABD B2BCE C3CDF D4DEG E5EFA F6FGB G7GAC

NRC Meeting Windsor June 2008 Not reached for Latin American Countries

NRC Meeting Windsor June 2008 Recommended delete Possible delete

NRC Meeting Windsor June 2008 Recommended delete Possible delete

NRC Meeting Windsor June 2008 Recommended delete Possible delete

NRC Meeting Windsor June 2008 True/False Items Composite model 1 Yuck! 

NRC Meeting Windsor June 2008 True/False Items Composite model 2 Also yuck! 

NRC Meeting Windsor June 2008 True/False Items 36: laws can be made by an elected. Disc: 0.18

NRC Meeting Windsor June 2008 True/False Items 37: judges make legal decisions without being influenced by the government. Disc: 0.16

NRC Meeting Windsor June 2008 True/False Items 38: the religion of the majority is the only legal religion. Disc: 0.36

NRC Meeting Windsor June 2008 True/False Items 38: the religion of the majority is the only legal religion.

NRC Meeting Windsor June 2008 True/False Items 39: people have the right to criticise the government. Disc: 0.31

NRC Meeting Windsor June 2008 True/False Items 39: people have the right to criticise the government.

NRC Meeting Windsor June 2008 True/False Items 40: only adults who pay income tax are allowed to vote. Disc: 0.15

NRC Meeting Windsor June 2008 True/False Items 41: all adult voters have the right to vote on new laws in the national. Disc: 0.18