1. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany ICCS 2009 Design, Databases and Materials

2. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Content Survey Design Status of Preparation International Database Structure: File Types, Countries, Variables, Missing Data Documentation: Codebooks, Instruments, IDB User Guide Seminar Material Analysis (Do‘s and Dont‘s)

3. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Content Survey Design Status of Preparation International Database Structure: File Types, Countries, Variables, Missing Data Documentation: Codebooks, Instruments, IDB User Guide Seminar Material Analysis (Do‘s and Dont‘s)

4. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – Objective ICCS 2009 is intended to investigate the ways in which young people are prepared to undertake their roles as citizens. It will report on: Student achievement (civic knowledge) Student disposition and attitudes Techers‘ teaching and class management practices School governance and climate National context of civic and citizenship education Asian, European and Latin American regional issues 4

5. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – Surveys Student Survey Student Achievement Booklets Student Questionnaire Teacher Survey Teacher Questionnaire School Survey School Questionnaire National Context Survey National Context Questionnaire 5

6. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – Target Populations Students: All students enrolled in the grade that represents eight years of schooling, counting from the first year of ISCED Level 1, providing the mean age at the time of testing is at least 13.5 years (usually grade 8). Students older than 17 years are not part of the target population. Teachers: All teachers teaching regular school subjects to students of the target grade (regardless of the subject or the number of hours taught) during the ICCS testing period, and have been employed at school since the beginning of the school year. 6

7. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – Participating Countries 38 countries from around the world 7 Source: Schulz et al., 2010, ICCS 2009: International Report, IEA, Amsterdam

8. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – Sample Structure Stratified multi-stage probability sampling design Random samples were required for each population Minimum sample sizes: 150 schools 1 intact class per school, aiming for an achieved sample size of 3,000 target grade students 15 target grade teachers (or all if number of target grade teachers was less or equal to 20) 8

9. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – Sample Structure More than 5.300 schools, 140.000 target grade students, and 62.000 target grade teachers 9

10. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – Sample Structure Reporting of data from the student survey need to be cautious for some countries and reflect the following issues: HKG and NLD did not meet sampling requirements. BFL, CZE, DNK, NZL, NOR, CHE, THA met guidelines for sampling participation rates only after replacement schools were included. ENG nearly satisfied guidelines for sample participation only after replacement schools were included. GTM, KOR, PRY surveyed the same cohort of students but at the beginning of the next school year. SVK national desired population does not cover all of international desired population. 10

11. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – Sample Structure Reporting of data from the teacher survey need to be cautious for some countries and reflect the following issues: AUT, BFL, DEN, ENG, HKG, LUX, NZL, NOR, and CHE did not meet sampling requirements. CZE, SWE, THA met guidelines for sampling participation rates only after replacement schools were included. IRL nearly satisfied guidelines for sample participation only after replacement schools were included. SVK national desired population does not cover all of international desired population. GRC and NLD data from the teacher survey could not be reported. All the above is annotated in the international reports. 11

12. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – Sample Structure More information about the sampling design will be detailed in chapters 2 and 10 of the ICCS 2009 Technical Report 12

13. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Design – CIVED Link Data from 17 countries could be linked to data from the IEA CIVED study from 1999. To cover a similar target population, four countries surveyed an additional grade (grade 9): GRC, NOR, SWE, SVN 13

14. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Content Survey Design Status of Preparation International Database Structure: File Types, Countries, Variables, Missing Data Documentation: Codebooks, Instruments, IDB User Guide Seminar Material Analysis (Do‘s and Dont‘s)

15. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Status of Preparation Data itself confirmed by countries Data Processing at the IEA DPC Data preparation done Derived variables added Weights added 15

16. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Status of Preparation Changes since last data send out (v3.1, April 2010) Current data version 3.2 Scale SCSTUDOP (Principals' perceptions of student opportunities to participate in community activities) in School data file (ICG) changed so that category “not offered at school” is now coded “0” instead of receiving a missing code 3 new scales added to Student Questionnaire data file (ISG): NISB: National Index of Socioeconomic Background PAREDYRS: Highest educational level in approximate years of schooling EXPEDYRS: Expected years of schooling (recoded from SISCED) 16

17. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Status of Preparation Timeline for future data send out (November 23 sent out to NRCs) January 2011: put on IEA website for public download, including Technical Report and IDB User Guide 17

18. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Content Survey Design Status of Preparation International Database Structure: File Types, Countries, Variables, Missing Data Documentation: Codebooks, Instruments, IDB User Guide Seminar Material Analysis (Do‘s and Dont‘s)

19. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany International Database - Content Country specific files Data files (labeled SPSS format, SAS format, national variables where applicable) Documentation Codebooks (WinDEM format) Technical Report IDB User Guide Data and documentation will be published in January 2011 on IEA website: www.iea.nlwww.iea.nl 19

20. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany International Database - Files Types ICGSchool Questionnaire File ITGTeacher Questionnaire File ISGStudent Questionnaire File ISAStudent Achievement File ISRStudent Reliability File ISEEuropean Module Student File ISLLatin American Module Student File ISSAsian Module Student File 20

21. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany International Database - Files Types Additional grade data (GRC, NOR, SWE, SVN) JSGStudent Questionnaire File JSAStudent Achievement File JSRStudent Reliability File JSEEuropean Module Student File 21

22. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Cases included: School questionnaires that were returned (CPART = 3) Teacher questionnaires that were returned (TPART = 3) AND where schools met the minimum within school teacher participation rate of 50% Student data that were returned (SPART = 3) AND where schools met the minimum within school student participation rate of 50% NOTE: A student was counted participating if either achievement or questionnaire data was available (regional module data only was not sufficient) Indicator variable: INICS09 = 1 22 International Database - Files Types

23. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Note that the student files and also the teacher file may contain cases from schools where the school questionnaire has not been returned. 23 International Database - Files Types

24. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany International Database - Data Structure From the survey ID variables (record, scorer, puncher, linking) and, for students, booklet rotation Listing and tracking variables (e.g., ITPARTR) Response variables and, for students, constructed response scores 24

25. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany International Database - Data Structure From the data processing and weighting stage Participation (SPART, TPART, CPART) and adjudication flags (INICS09) Stratification information (IDSTRATE/I) Design and estimation weights Variables needed for variance estimation (JKZONES, JKREPS) Meta data (VERSION, IDCNTRY, DPCDATE) 25

26. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany With the information in the NAFs national adaptations could be reflected and data were recoded to fit international database structure in the majority of cases Full back-translation into English will be important for readers of the report and also for secondary research (especially if more than only country specific terms were adapted) International Database - National Adaptations 26

27. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Main goal: document the comparability or non- comparability of country data Documentation of important cultural and other national adaptations that affect the meaning of a variable International Database - National Adaptations 27

28. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany If national adaptations made recoding of data necessary, this has been documented in the National Adaptations Reports If international comparability is not achievable, data for international variable(s) were set to not administered (but country data were still available as national variable) If possible, data for not administered variable(s) were gang punched to a certain value E.g., ITLANG has been gang punched to ‘1’ if only one language was used for all instruments in a country International Database - National Adaptations 28

29. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany National Adaptations Reports will inform and support secondary analysis Will be published as part of the ICCS 2009 complementary publication (IDB User Guide) International Database - National Adaptations 29

30. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany International Database - Variables Identification variables (ID…) Tracking variables (IT…) Survey instrument variables Weighting variables Scale scores 30

31. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Identification Variables Unique Identifiers IDSCHOOL School ID 4 digits Used in all files 31

32. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Identification Variables IDCLASS Class ID 6 digits First 4 digits identify school Last two digits used to identify class within school Used in the student files (ISG, ISA, ISR, ISE, ISL, ISS) 32

33. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Identification Variables IDTEACH Teacher ID 6 digits Used in teacher file (ITG) IMPORTANT NOTE: Teachers are NOT linked to classes and NOT directly linked to students! 33

34. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Identification Variables IDSTUD Student ID 8 digits First 6 digits include the corresponding class (IDCLASS) Used in student data files 34

35. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Tracking Variables CPART TPART SPART  Final Participation Indicators ITLANG  Administration Language (1, 2,..., 5) IDCNTRY  Country ID for Reporting VERSION  Data Sendout Version (currently 32) DPCDATE  Date Exported (currently 20101116) 35

36. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Variables IC2G01A…IC2G23  survey instrument variables in school data file (ICG) IT2G01A... IT2G29I  survey instrument variables in teacher data file (ITG) 36

37. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Variables CI2COM1…CI106M1  survey instrument variables in student achievement data file (ISA) IS2G03... IS2G18G  student background variables in student questionnaire data file (ISG) IS2P19A... IS2P36F  student perception variables in student questionnaire data file (ISG) 37

38. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Variables EST01A…EST12  student achievement variables in European module data file (ISE) ESP01A…ESP13D  student perception variables in European module data file (ISE) LST01…LST16  student achievement variables in Latin American module data file (ISL) LSP01A…LSP12I  student perception variables in Latin American module data file (ISL) ASP01A…ASP08E  student perception variables in Asian module data file (ISS) 38

39. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Survey Variables All survey instrument variables that have been adapted for national purposes, have been recoded to fit the international scheme National variables will be provided to country, but not be part of international database 39

40. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Variables for Weighting and VE School data file TOTWGTC  Final school weight – school study JKZONEC  Jackknife zones – school study JKREPC  Jackknife replicate code – school study 40

41. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Variables for Weighting and VE Teacher data file TOTWGTT  Final teacher weight – teacher study JKZONET  Jackknife zones – teacher study JKREPT  Jackknife replicate code – teacher study 41

42. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Variables for Weighting and VE Student data files TOTWGTS  Final student weight – student study JKZONES  Jackknife zones – student study JKREPS  Jackknife replicate code – student study 42

43. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Content Knowledge Variables PV1CIV…PV5CIV: Civic knowledge plausible values Centered to mean of 500 and standard deviation of 100 scale points (for equally weighted countries) NWLCIV: National civic knowledge score (WLE) 43

44. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Scales & Derived Variables For derived indices, see the list of indices (C:\ICCS2009\Documentation\ICCS2009_Indices.xls)list of indices Example: RELIG Students’ belonging to a religion Dummy coded to ‘No religion’ (code 0) – ‘Religion’ (code 1) 44

45. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Scales & Derived Variables For derived scales, see the scale documentation (C:\ICCS2009\Documentation\ICCS2009_Scales.doc)scale documentation Example: POLDISC Students' discussion of political and social issues outside of school IRT WLE scores with mean of 50 and standard deviation of 10 for equally weighted countries Items included IS2G13A IS2G13D IS2G13F IS2G13G 45

46. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Missing Values Special codes represent missing data Codes contain information on nature of missing data Codes and levels of missing data should be checked before usage 46

47. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany (SPSS)‏ 9 99... Missing Values – “Omitted” No Response 47

48. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany (SPSS)‏ 7 97... Missing Values – “Invalid” Invalid Response 48

49. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Question not reached Question omitted (SPSS)‏ 6 96 … Missing Values – “Not Reached” (SPSS)‏ 2 49 (SPSS)‏ 9 99 …

50. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany (SPSS)‏ System- missing. Missing Values – “Not Administered” Nationally not administered variables 50

51. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany (SPSS)‏ 5 95... Question 2 does not apply according to filter question Missing Values – “Logically Not Applicable” 51

52. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Codebooks Contain information related to the structure of file types including the Source Format Descriptive labels Response option codes for all variables One codebook file is provided for each of the instruments/ file types Available in.txt format that can be opened with text editor All dictionary information can be found directly in the SPSS file 52

53. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Codebooks 53

54. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Content Survey Design Status of Preparation International Database Structure: File Types, Countries, Variables, Missing Data Documentation: Codebooks, Instruments, IDB User Guide Seminar Material Analysis (Do‘s and Dont‘s)

55. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Seminar Material Agenda Seminar agenda Codebooks For each of the file types (ICG, ITG, ISA, ISG, ISR, ISE, ISL, ISS) Data SPSS data (one file per country and file type) 55

56. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Seminar Material Documentation List of Indices Scale documentation Draft chapters of IDB User Guide  All these is DRAFT information (do not cite or quote). Information will be published in the IDB User Guide. 56

57. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Seminar Material Instruments School Questionnaire Teacher Questionnaire Student Questionnaire Regional Module Questionnaires European (questionnaire part only) Latin American (questionnaire part only) Asian Presentations All presentations of this seminar 57

58. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Seminar Material Software 7-Zip: compressing and archiving files (freeware) Adobe Reader 9: reading PDF docuements IEA IDB Analyzer: analyzing IEA data Work SPSS script needed for seminar hands-on training AddIns_ICCS Support material and examples for hands-on training of statisticial significance tests HLM_Training Support material and examples for hands-on training of HLM sample analysis 58

59. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Content Survey Design Status of Preparation International Database Structure: File Types, Countries, Variables, Missing Data Documentation: Codebooks, Instruments, IDB User Guide Seminar Material Analysis (Do‘s and Dont‘s)

60. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Analyzing ICCS 2009 Data ICCS 2009 is a survey and collected observational data We know what schools, teachers, and students characteristics are, what they think, and the context in which this occurs We have current cross-sectional information, while the reason or effect might have occurred a while back The data has invaluable descriptive power We can make statements about associations and correlations, not causation 60

61. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Analyzing ICCS 2009 Data ICCS 2009 is NOT an experiment We do not control the assignment of students and teachers to ‘treatment’ groups Events have already happened and we record what has happened We can not establish causality, or direct effects 61

62. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Analyzing ICCS 2009 Data School variables ONLY Use ICG file Teacher variables ONLY Use ITG file Student variables ONLY Use ISA, ISG, ISE, ISL, and/or ISS file 62

63. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Analyzing ICCS 2009 Data Student variables using school level attributes (combined) Merge corresponding files (i.e. disaggregate school variables to students) Supported by the IEA IDB Analyzer Use the student weight (TOTWGTS) Mind that there might be students of a school of which the school questionnaire was not returned or vice versa, so watch levels of missing data 63

64. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Analyzing ICCS 2009 Data School variables using aggregated student level characteristics Not supported by the IEA IDB Analyzer If at all, use with caution (ecological fallacy) 64

65. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Analyzing ICCS 2009 Data Teacher variables using school level attributes (combined) Merge corresponding files (i.e. disaggregate school variables to teachers) Supported by the IEA IDB Analyzer Use the teachers weight (TOTWGTT) Mind that there might be teachers of a school of which the school questionnaire was not returned or vice versa, so watch levels of missing data 65

66. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Analyzing ICCS 2009 Data Teacher data should not be merged with student data There is no direct linkage between teachers and students Teachers teaching in the school a student is attending might have never seen that student or taught a lesson to the student 66

67. ICCS 2009 IDB Seminar – Nov 24-26, 2010 – IEA DPC, Hamburg, Germany Thanks for your attention!