3rd NRC Meeting, 9-12 June 2008, Windsor ICCS 2009 Main Survey Field Operations

3rd NRC Meeting, 9-12 June 2008, Windsor 2 Field Operations - Overview National Context Questionnaire Main Survey Operations Overview On-line Data Collection Support Materials Main Survey Timeline Next Steps

3rd NRC Meeting, 9-12 June 2008, Windsor 3 National Context Questionnaire Stage 2 of National Contexts Survey will start in March 2009 and will continue until October/November 2009 Survey will be conducted on-line (similar to stage 1) NRCs will only need to update information from stage 1 NOTE: There are separate logins (username and password) for the sources available on-line: –ICCS study homepage –WebFTP server (2-stage login) –National Context Questionnaire –Survey Activities Questionnaire

3rd NRC Meeting, 9-12 June 2008, Windsor Main Survey Operations Overview

3rd NRC Meeting, 9-12 June 2008, Windsor 5 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 6 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 7 Verification of Adaptations Translation Verification Adapt Instruments & Document Adaptations Correct Adaptations & Tranlsate Instruments Correct Instruments & Assemble Booklets Layout Verification Correct Instruments (if necessary) Printing & Send Out

3rd NRC Meeting, 9-12 June 2008, Windsor 8 NRCISC/IEA Sec./DPC time Time planning before administration Layout verification Implement changes & assemble booklets Translation verification Adapt instruments 2 weeks 1 week 3 weeks 4 weeks Start data collection16 weeks Instruments send out Printing & send out Translate instruments Verification of adaptations 1 week 4 weeks

3rd NRC Meeting, 9-12 June 2008, Windsor 9 due dateNRC/ISC/IEAtime Time planning before administration - SH Layout verification Implement changes & assemble instruments Translation verification Adapt & translate instruments 1 week 2 weeks 1 week Instruments send out July 11, 2008 August 22, 2008 September 19, 2008 September 26, 2008 October 10, 2008 October 31, 2008 Start data collection Printing & send out 3 weeks October 31, 2008 Adapt instruments Verification of adaptations July 25, 2008 July 18, weeks

3rd NRC Meeting, 9-12 June 2008, Windsor Meanwhile…

3rd NRC Meeting, 9-12 June 2008, Windsor 11 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 12 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 13 Contact Schools and get within School Information Perform Within-School Sampling Prepare List of Schools and send it to DPC Recruit and Train Test Administrators DPC provides list of sampled Schools Recruit and Train Scorers Translate School Coordinator Manual Translate Test Administrator Manual

3rd NRC Meeting, 9-12 June 2008, Windsor 14 School Contact If not yet done, send sampling frame (list of schools) to the IEA DPC Sampling of schools has been or will be done by the IEA DPC List of sampled schools has been or will be provided by the IEA DPC Contact schools, ensure participation Identify liaison person (School Coordinator) Provide School Coordinator Manual Record status for each sampled school in WinW3S Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible, also combining the two to only one document)

3rd NRC Meeting, 9-12 June 2008, Windsor 15 Within-School Sampling (Students) Two levels: students and teachers Identification of classes on Class Listing Forms –Class name –Number of students Random within-school sample of classes Identification of students within sampled classes on Student Listing Forms Send list of sampled students to School Coordinator on Student Tracking Form WinW3S serves as survey administration tool

3rd NRC Meeting, 9-12 June 2008, Windsor 16 Within-School Sampling (Teachers) Identification of eligible target grade teachers on Teacher Listing Forms Within-school random sample of target grade teachers Send Teacher Tracking Form to School Coordinator

3rd NRC Meeting, 9-12 June 2008, Windsor 17 Within-School Sampling WinW3S will support NRCs with –Storing school contact information –Listing of classes / teachers / students –Sampling of classes / teachers –Assignment of booklets –Survey administration –Tracking of participation  Will be subject to Data Management Seminar on July 8-11, 2008 in Hamburg

3rd NRC Meeting, 9-12 June 2008, Windsor 18 Assigning Questionnaires School Questionnaire Teacher –Every teacher will receive one questionnaire Student –Achievement Booklet (one of seven rotated forms) –Questionnaire (one form, no rotated forms in main survey) –Regional Module Questionnaire  WinW3S will assign forms and provide labels for questionnaires

3rd NRC Meeting, 9-12 June 2008, Windsor 19 NRCSchools Time Planning before Administration time Contacts schools Decision to participate Sends listing forms List students/teachers Samples teachers Sends tracking forms Sends questionnaires Start data collection Name School Coordinator Samples classes List classes/provide number of teachers

3rd NRC Meeting, 9-12 June 2008, Windsor 20 School ContactInstrumentstime Time Planning before Administration Contact schools Send listing forms Sample teachers Send tracking forms Send questionnaires Implement changes & assemble booklets Adapt & translate instruments Printing Data collection Sample classes

3rd NRC Meeting, 9-12 June 2008, Windsor 21 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 22 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 23 Monitor Test Administrators Ad hoc Support Arrange for Material Receipt Train Scorers Train Data Entry Staff

3rd NRC Meeting, 9-12 June 2008, Windsor 24 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 25 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 26 Organize Material for (direct) Data Entry Organize Material for Scoring and Reliability Scoring Check if Information and Material is complete Get back to School if necessary Enter Participation Information

3rd NRC Meeting, 9-12 June 2008, Windsor 27 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 28 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 29 Adapt International Codebooks to National Questionnaires Use National Adaptation Forms Use adapted Codebooks for Data Entry in WinDEM

3rd NRC Meeting, 9-12 June 2008, Windsor 30 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 31 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 32 Conduct and Monitor Scoring Conduct and Monitor Reliability Scoring Organize Material for Data Entry

3rd NRC Meeting, 9-12 June 2008, Windsor 33 Scoring All open ended items have to be scored according to scoring guide Reliability scoring for open ended items for ~700 booklets (100 booklets per rotated form), ideally at the beginning of the scoring process, so that measures can be undertaken to improve reliability of scoring Occupations of parents/guardians have to be scored according to ISCO 88  Subject to Marker Training on July 7-8, 2008, in Hamburg

3rd NRC Meeting, 9-12 June 2008, Windsor 34 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 35 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 36 Complete Participation Information Verify Data Enter Data

3rd NRC Meeting, 9-12 June 2008, Windsor 37 Data Entry Data entry software (WinDEM) and international codebooks for the MS are provided by the DPC on August 18, 2008 National staff should be trained on data entry, even if all or parts are being outsourced Computer workstations need to be set up National Centers enter and verify data from paper questionnaires

3rd NRC Meeting, 9-12 June 2008, Windsor 38 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 39 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor 40 Set of Final Instruments National Codebooks and Documentation (NAF v. IV) Verified Data sets Tracking Forms Reliability Scoring Data

3rd NRC Meeting, 9-12 June 2008, Windsor 41 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

3rd NRC Meeting, 9-12 June 2008, Windsor On-line Data Collection

3rd NRC Meeting, 9-12 June 2008, Windsor 43 On-line Data Collection (ODC) Used for all questionnaires that National Centers should answer –National Context Questionnaire –Survey Activities Questionnaire International Option for –School Questionnaire –Teacher Questionnaire Analysis of split half design during field trial showed that the use of the on-line mode in general is feasible All nine countries participating in ODC option during the FT received individual feedback Only countries that tried out ODC in FT can use ODC option for MS

3rd NRC Meeting, 9-12 June 2008, Windsor 44 On-line Data Collection in MS True mixed mode design Goal: –Reduce response burden by offering respondents preferred mode –Save time on data entry and data processing –Improve quality of data Teachers and school principals should be assigned to the preferred mode –Knowlegde of NRC about feasibility/preferences –School coordinator could contactS school principal and teachers Ensure fall-back to paper questionnaire

3rd NRC Meeting, 9-12 June 2008, Windsor 45 Codebook Adaptation Instrument Preparation Sampling and Preparing Administration Data Entry and Verification Administration Receiving Material Data Sendout Codebook Adaptation Prepare ODC Monitor ODC

3rd NRC Meeting, 9-12 June 2008, Windsor 46 School contactInstrumentstime Time planning before administration ODC Contact schools Sample students / teachers Send tracking forms Send questionnaires Implement changes & assemble booklets Adapt & translate instruments Printing Data collection Printing Prepare ODC Send quest./cover letters Send listing forms Sample classes

3rd NRC Meeting, 9-12 June 2008, Windsor 47 On-line Data Collection (ODC) Based on concept of cultures, one setup per language and country Participants receive “cover letter” –Label with login information provided by WinW3S –Culture specific URL –Personal username and password –Sent to school coordinator for distribution Same administration period as paper questionnaires NRCs can monitor completion status of on-line questionnaires NEW: Status of on-line respondents can be exported to WinW3S If necessary, NRCs instruct School Coordinators to remind teachers/school principals to complete the questionnaire No data entry at National Center needed

3rd NRC Meeting, 9-12 June 2008, Windsor Support Material

3rd NRC Meeting, 9-12 June 2008, Windsor 49 Support Material Survey Operations Procedures Manual (MS) –Unit 1: School contact and material preparation –Unit 2: Within school sampling and test administration School Coordinator Manual Test Administrator Manual National Quality Control Monitor Manual Within School Sampling Software (WinW3S)

3rd NRC Meeting, 9-12 June 2008, Windsor 50 Support Material Survey Operations Procedures Manual (MS) –Unit 3: On-line data collection, scoring and data entry IEA SurveySystem Designer Scoring Guide Codebooks Windows Data Entry Manager (WinDEM)  Distribution of all materials and software packages through secure section of study website at

3rd NRC Meeting, 9-12 June 2008, Windsor Main Survey Timeline

3rd NRC Meeting, 9-12 June 2008, Windsor 52 Main Survey Timeline - SH - Receive verified final instruments - Start printing and distribution of questionnaires - Send final instruments and NAF v. IV to IEA DPC - Set up on-line questionnaires October 10, 2008 Submit assembled instruments and NAF version III for layout verification to ACER September 26, 2008 Receive verified instrumentSeptember 19, 2008 Submit translated questionnaires and NAF version II for international translation verification to IEA Sec. August 22, 2008 Receive feedback on national adaptationsJuly 25, 2008 Submit national adaptations in NAF version I to ISCJuly 18, 2008 Receive instruments & NAFJuly 11, 2008

3rd NRC Meeting, 9-12 June 2008, Windsor 53 Main Survey Timeline - SH Final deadline for submitting data to DPCMarch 27, Scoring of open-ended items and occupations - Data entry and verification November – March, 2009 Proposed MS administration period (subject to country conditions) October – December, 2008 Submit on-line questionnaires definition files for verification to DPC October 17, 2008 Give feedback about FT procedures using the on-line Survey Activities Questionnaire April, 2009

3rd NRC Meeting, 9-12 June 2008, Windsor 54 Main Survey Timeline – NH - Receive verified final instruments - Start printing and distribution of questionnaires - Send final instruments and NAF v. IV to IEA DPC - Set up on-line questionnaires February, 2008 Submit assembled instruments and NAF version III for layout verification to ACER February, 2008 Translation verificationJanuary, 2008 Translate questionnaires and prepare NAF version II for international translation verification at IEA Sec. December, 2008 Receive feedback on national adaptationsNovember, 2008 Submit national adaptations in NAF version I to ISCNovember, 2008 Receive instruments & NAFJuly 11, 2008

3rd NRC Meeting, 9-12 June 2008, Windsor 55 Main Survey Timeline - NH Final deadline for submitting data to DPCAugust 28, Scoring of open-ended items and occupations - Data entry and verification April – August, 2009 Proposed MS administration period (subject to country conditions) March – May, 2009 Submit on-line questionnaires definition files for verification to DPC February, 2008 Give feedback about FT procedures using the on-line Survey Activities Questionnaire September, 2009

3rd NRC Meeting, 9-12 June 2008, Windsor Next Steps

3rd NRC Meeting, 9-12 June 2008, Windsor 57 Next Steps Prepare team at national center and plan for –School contact –Adaptation & translation of instruments –Test administrators –Scorers (CCE items and occupations) –Data management (data entry and verification) Be prepared to receive Main Survey instruments on July 11, 2008

3rd NRC Meeting, 9-12 June 2008, Windsor 58 Next Steps Marker Training on July 7-8, 2008 in Hamburg Data Management Seminar on July 8-11, 2008 in Hamburg

3rd NRC Meeting, 9-12 June 2008, Windsor Contact Information

3rd NRC Meeting, 9-12 June 2008, Windsor 60 Contact Information Field Operations / Data Management: IEA Data Processing and Research Center Falk Brese Michael Jung

3rd NRC Meeting, 9-12 June 2008, Windsor Thank you for your attention!