1st NRC Meeting, October 2006, Amsterdam 1 ICCS 2009 Field Operations

1st NRC Meeting, October 2006, Amsterdam 2 Field Operations Overview National Context Questionnaire Item Pilot Field Trial Operations Overview Online Data Collection Support Materials Field Trial Timeline Next Steps Contact Information

1st NRC Meeting, October 2006, Amsterdam 3 National Context Questionnaire Language will be English only NRCs will have to gather information from several sources Data for National Context Survey will be collected online NRCs can login as often as required Questionnaire will be online end of 2006/beginning of 2007 –NRCs will receive with individual access information

1st NRC Meeting, October 2006, Amsterdam 4 Item Pilot Submit Items to ACER via at Review Items and commenting on study website End of reviewing process mid January 2007

1st NRC Meeting, October 2006, Amsterdam 5 Field Trial Operations Overview

1st NRC Meeting, October 2006, Amsterdam 6 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 7 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 8 Adapt Questionnaires & Document Adaptations Translation Verification Layout Verification Translate Questionnaires & Verifiy within Country Correct & Assemble Questionnaires Correct Questionnaires (if necessary) Print Send Out

1st NRC Meeting, October 2006, Amsterdam 9 NRCIEA Sec./ISC/DPCtime Time planning before administration Layout Verification Implement changes & assemble Instruments Translation Verification Translate Instruments 2 weeks 4 weeks 3 weeks 2 weeks 4 weeks Start data collection15 weeks Instruments Send Out Printing

1st NRC Meeting, October 2006, Amsterdam 10 due dateNRC/ISC/IEAtime Time planning before administration Layout Verification Implement Changes & Assemble Instruments Translation Verification Translate Instruments 4 weeks 3 weeks 2 weeks 4 weeks Instruments Send Out 2. July July August September October October 2007 Start data collection Print Instruments 2 weeks 15. October 2007

1st NRC Meeting, October 2006, Amsterdam 11 Meanwhile…

1st NRC Meeting, October 2006, Amsterdam 12 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 13 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 14 Contact Schools and get within School Information Perform Within-School Sampling Prepare List of Schools and send it to DPC Recruit and Train Test Administrators DPC provides list of sampled Schools Recruit and Train Scorers Translate School Coordinator Manual Translate Test Administrator Manual

1st NRC Meeting, October 2006, Amsterdam 15 School Contact Send sampling frame (list of schools) to DPC Sampling of schools will be done by DPC preferably List of sampled schools will be provided by DPC Contact schools, ensure participation Identify liaison person (School Coordinator) Provide School Coordinator Manual Record status for each sampled school in WinW3S Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible)

1st NRC Meeting, October 2006, Amsterdam 16 Within-School Sampling (Students) Two levels: students and teachers Identification of classes on Class Listing Forms –Class name –Number of students Random within-school sample of classes Identification of students within sampled classes on Student Listing Forms Send list of sampled students to School Coordinator on Student Tracking Form WinW3S serves as survey administration tool

1st NRC Meeting, October 2006, Amsterdam 17 Within-School Sampling (Teachers) Identification of eligible teachers on Teacher Listing Forms Within-school random sample of teachers Send Teacher Tracking Form to School Coordinator

1st NRC Meeting, October 2006, Amsterdam 18 Within-School Sampling WinW3S will support NRCs with –Listing of classes / teachers / students –Sampling of classes / teachers –Assignment of forms –Survey administration –Tracking of participation  Will be subject to Data Management Seminar

1st NRC Meeting, October 2006, Amsterdam 19 Assigning Questionnaires School Questionnaire Teacher –Every Teacher will receive one questionnaire Student –Booklet –Questionnaire (perception and background part in one form) –Regional Modules instrument(s)  WinW3S will assign forms and provide labels for questionnaires

1st NRC Meeting, October 2006, Amsterdam 20 NRCSchool Time Planning before Administration time Contact Schools Decision to participate Send Listing Forms List Students/Teachers Sample Teachers Send Tracking Forms Send Questionnaires Start data collection Name School Coordinator Sample Classes List Classes

1st NRC Meeting, October 2006, Amsterdam 21 School ContactInstrumentstime Time Planning before Administration Contact Schools Send Listing Forms Sample Teachers Send Tracking Forms Send Questionnaires Implement changes & assemble Quest. Translate Instruments Printing Data collection Sample Classes

1st NRC Meeting, October 2006, Amsterdam 22 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 23 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 24 Adapt Codebooks To National Questionnaires Document Adaptations in National Adaptaion Forms

1st NRC Meeting, October 2006, Amsterdam 25 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 26 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 27 Monitor Test Administrators Ad hoc Support Arrange for Material Receipt Train Scorers Train Data Entry Staff

1st NRC Meeting, October 2006, Amsterdam 28 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 29 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 30 Organize Material for (direct) Data Entry Organize Material for Scoring and Reliability Scoring Check if Information and Material is complete Get back to School if necessary Enter Participation Information

1st NRC Meeting, October 2006, Amsterdam 31 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 32 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 33 Conduct and Monitor Scoring Conduct and Monitor Reliability Scoring Organize Material for Data Entry

1st NRC Meeting, October 2006, Amsterdam 34 Scoring All open ended items have to be scored according to scoring guide Reliability scoring for open ended items for a certain amount of booklets  Subject to Scoring Training Seminar in 2007

1st NRC Meeting, October 2006, Amsterdam 35 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 36 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 37 Complete Participation Information Verify Data Enter Data

1st NRC Meeting, October 2006, Amsterdam 38 Data Entry Data entry software (WinDEM) and international codebooks for the FT are provided by the DPC in time National staff should be trained on data entry, even if all or parts are being outsourced Computer workstations need to be set up National Centers enter and verify data from paper questionnaires Adapt codebooks to reflect national conditions (e.g. ISCED levels)

1st NRC Meeting, October 2006, Amsterdam 39 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 40 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 41 Set of Final Instruments National Codebooks and Documentation (NAF) Verified Data sets Tracking Forms Reliability Coding Data

1st NRC Meeting, October 2006, Amsterdam 42 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

1st NRC Meeting, October 2006, Amsterdam 43 Online Data Collection

1st NRC Meeting, October 2006, Amsterdam 44 Online Data Collection (ODC) Used for all questionnaires that National Centers should answer –National Context Questionnaire –Survey Activities Questionnaire International Option for –School Questionnaire –Teacher Questionnaire Countries that decide to use ODC in Main Study are obliged to conduct Field Trial in a split half design Ensure fall-back to paper questionnaire True mixed mode design Goal: reduce response burden –Ask schools for preferred mode (in Main Study) –Mode assignment for all questionnaires of one type possible

1st NRC Meeting, October 2006, Amsterdam 45 Codebook Adaptation Instrument Preparation Sampling and Preparing Administration Data Entry and Verification Administration Receiving Material Data Sendout Codebook Adaptation Prepare ODC Monitor ODC

1st NRC Meeting, October 2006, Amsterdam 46 School contactInstrumentstime Time planning before administration Contact Schools Sample Students / Teachers Send Tracking Forms Send Questionnaires Implement changes & assemble Quest. Translate Instruments Printing Data collection Printing Prepare ODC Send Quest./Cover Letters Send Listing Forms Sample Classes

1st NRC Meeting, October 2006, Amsterdam 47 Online Data Collection (ODC) - Facts Based on concept of cultures All languages supported (character sets, text directions,…) Participants receive “cover letter” –Label with login information provided by WinW3S –Culture specific URL –Personal username and password –Sent to school coordinator for distribution Participants can log in and out as often as desired Same administration period as paper questionnaires

1st NRC Meeting, October 2006, Amsterdam 48 Support Material

1st NRC Meeting, October 2006, Amsterdam 49 Support Material Survey Operations Manual (FT) –Part 1: Survey Operations Overview, Dates and Activities –Part 2: Survey Administration School Coordinator Manual Test Administrator Manual Within School Sampling Software (WinW3S) + Manual IEA SurveySystem Designer + Manual National Quality Control Monitor Manual –Part 3: Data Entry Scoring Guide Codebooks Windows Data Entry Manager (WinDEM) + Manual  Distribution of all materials and software packages through study website only

1st NRC Meeting, October 2006, Amsterdam 50 FT Timeline

1st NRC Meeting, October 2006, Amsterdam 51 Field Trial Timeline Receive instruments & NAF by 2 July 2007 Submit translated questionnaires with NAF v1 for international translation verification by 30 July 2007 Receive verified instruments by 27 August 2007 Submit for layout verification by 17 September 2007 Start printing and distribution of questionnaires by 1 October 2007 Proposed FT administration period (subject to country conditions): 15 October – 16 November 2007 Submit data to DPC by end of 2007

1st NRC Meeting, October 2006, Amsterdam 52 Next Steps

1st NRC Meeting, October 2006, Amsterdam 53 Next Steps Send items to ISC via to: Review and comment on items at until Mid January 2007 Complete National Context Questionnaire beginning of 2007 Set up team at national center and plan for –Staff –Translation –School Contact –Test Administrators –Scorers –Data Management (Data Entry and Verification) NRC meeting in June 2007 Data Management Seminar in July 2007 Be prepared to receive Field Trial instruments in July 2007

1st NRC Meeting, October 2006, Amsterdam 54 Contact Information

1st NRC Meeting, October 2006, Amsterdam 55 Contact Information Field Operations / Data Management: IEA Data Processing Center Falk Brese Michael Jung

1st NRC Meeting, October 2006, Amsterdam 56 Thanks for your attention!