1. Data Management Seminar, 8-11th July 2008, Hamburg ICCS 2009 Main Survey Field Operations

2. Data Management Seminar, 8-11th July 2008, Hamburg Field Operations - Overview Main Survey Operations Overview On-line Data Collection Support Materials Main Survey Timeline Next Steps Contact Information

3. Data Management Seminar, 8-11th July 2008, Hamburg Main Survey Operations Overview

4. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

5. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

6. Data Management Seminar, 8-11th July 2008, Hamburg Verification of Adaptations Translation Verification Adapt Instruments & Document Adaptations Correct Adaptations & Tranlsate Instruments Correct Instruments & Assemble Booklets Layout Verification Correct Instruments (if necessary) Printing & Send Out Instrument Preparation

7. Data Management Seminar, 8-11th July 2008, Hamburg NRCISC/IEA Sec./DPCtime Time planning before administration Layout verification Implement changes & assemble booklets Translation verification Adapt instruments 2 weeks 1 week 2 weeks 4 weeks Start data collection15 weeks Instruments send out Printing & send out Translate instruments Verification of adaptations 1 week 4 weeks

8. Data Management Seminar, 8-11th July 2008, Hamburg due dateNRC/ISC/IEAtime Time planning before administration - SH Layout verification Implement changes & assemble instruments Translation verification Adapt & translate instruments 1 week 2 weeks 1 week Instruments send out July 11, 2008 August 22, 2008 September 19, 2008 September 26, 2008 October 10, 2008 October 24, 2008 Start data collection Printing & send out 2 weeks October 24, 2008 Adapt instruments Verification of adaptations July 25, 2008 July 18, 2008 4 weeks

9. Data Management Seminar, 8-11th July 2008, Hamburg Meanwhile…

10. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

11. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

12. Data Management Seminar, 8-11th July 2008, Hamburg Contact Schools and get Within-School Information Perform Within-School Sampling Prepare List of Schools and send it to DPC Recruit and Train Test Administrators DPC provides list of sampled Schools Recruit and Train Scorers Translate School Coordinator Manual Translate Test Administrator Manual Recruit and Train School Coordinators Sampling & Preparing Survey Admin.

13. Data Management Seminar, 8-11th July 2008, Hamburg School Contact If not yet done, send sampling frame (list of schools) to the IEA DPC Sampling of schools has been or will be done by the IEA DPC List of sampled schools has been or will be provided by the IEA DPC Contact schools, ensure participation Identify liaison person (School Coordinator) Provide School Coordinator Manual Record status for each sampled school in WinW3S

14. Data Management Seminar, 8-11th July 2008, Hamburg School Contact Adaptation and translation of School Coordinator and Test Administrator Manual (no verification on international level, modifications to some extend possible, also combining the two to only one document)

15. Data Management Seminar, 8-11th July 2008, Hamburg Within-School Sampling (Students) Two levels: students and teachers Identification of classes on Class Listing Forms –Class name –Number of students Random within-school sample of classes Identification of students within sampled classes on Student Listing Forms Send list of sampled students to School Coordinator on Student Tracking Form WinW3S serves as survey administration tool

16. Data Management Seminar, 8-11th July 2008, Hamburg Within-School Sampling (Teachers) Identification of eligible target grade teachers on Teacher Listing Forms Within-school random sample of target grade teachers Send Teacher Tracking Form to School Coordinator

17. Data Management Seminar, 8-11th July 2008, Hamburg Within-School Sampling WinW3S will support NRCs and DMs with –Storing school contact information –Listing of classes / teachers / students –Sampling of classes / teachers –Assignment of booklets –Survey administration –Tracking of participation  Will be subject today and tomorrow

18. Data Management Seminar, 8-11th July 2008, Hamburg Assigning Questionnaires School Questionnaire Teacher –Every teacher will receive one questionnaire Student –Achievement Booklet (one of seven rotated forms) –Questionnaire (one form, no rotated forms in main survey) –Regional Module Questionnaire  WinW3S will assign forms and provide labels for questionnaires

19. Data Management Seminar, 8-11th July 2008, Hamburg NRCSchoolstime Contacts schools Decision to participate Sends listing forms List students/teachers Samples teachers Sends tracking forms Sends questionnaires Start data collection Name School Coordinator Samples classes List classes/provide number of teachers Time Planning before Administration

20. Data Management Seminar, 8-11th July 2008, Hamburg School ContactInstrumentstime Contact schools Send listing forms Sample teachers Send tracking forms Send questionnaires Implement changes & assemble booklets Adapt & translate instruments Printing Data collection Sample classes Time Planning before Administration

21. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

22. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

23. Data Management Seminar, 8-11th July 2008, Hamburg Monitor Test Administrators Ad hoc Support Arrange for Material Receipt Train Scorers Train Data Entry Staff Survey Administration

24. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

25. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

26. Data Management Seminar, 8-11th July 2008, Hamburg Organize Material for (direct) Data Entry Organize Material for Scoring and Reliability Scoring Check if Information and Material is complete Get back to School if necessary Enter Participation Information Receiving Material

27. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

28. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

29. Data Management Seminar, 8-11th July 2008, Hamburg Adapt International Codebooks to National Questionnaires Use National Adaptation Forms Use adapted Codebooks for Data Entry in WinDEM Codebook Adaptation

30. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

31. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

32. Data Management Seminar, 8-11th July 2008, Hamburg Conduct and Monitor Scoring Conduct and Monitor Reliability Scoring Organize Material for Data Entry Conduct and Monitor Reliability Scoring Conduct and Monitor Scoring Retrain or Discuss items Scoring

33. Data Management Seminar, 8-11th July 2008, Hamburg Scoring All open ended items have to be scored according to scoring guide Ideally at the beginning of the scoring process, the first incoming booklets should be scored for reliability, so that measures can be undertaken to improve reliability of scoring (e.g., discuss and retrain problematic items) Reliability scoring required for open ended items for ~700 booklets (100 booklets per rotated form) Occupations of parents/guardians have to be scored according to ISCO 88  Has been subject to Marker Training yesterday and today

34. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

35. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

36. Data Management Seminar, 8-11th July 2008, Hamburg Complete Participation Information in WinW3S Verify Data Enter Data in WinDEM Data Entry Enter Double Punching Data in WinDEM

37. Data Management Seminar, 8-11th July 2008, Hamburg Data Entry Data entry software (WinDEM) and international codebooks for the MS are provided by the DPC on August 18, 2008 National staff should be trained on data entry, even if all or parts are being outsourced Computer workstations need to be set up National Centers enter data and double punching data in WinDEM Before send out, National Centers need to verify data from paper questionnaires using verification checks provided in WinDEM  Will be subject tomorrow through Thursday

38. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

39. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

40. Data Management Seminar, 8-11th July 2008, Hamburg Set of Final Instruments National Adaptation Forms (NAF) version IV Exported WinW3S database National Codebooks Verified Data sets Reliability Scoring Data Data Submission DocumentationData Tracking FormsDouble Punching Data

41. Data Management Seminar, 8-11th July 2008, Hamburg Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring

42. Data Management Seminar, 8-11th July 2008, Hamburg On-line Data Collection

43. Data Management Seminar, 8-11th July 2008, Hamburg On-line Data Collection (ODC) Used for all questionnaires that National Centers should answer –National Context Questionnaire –Survey Activities Questionnaire International Option for –School Questionnaire –Teacher Questionnaire Analysis of split half design during field trial showed that the use of the on-line mode in general is feasible All nine countries participating in ODC option during the FT received individual feedback Only countries that tried out ODC in FT can use ODC option for MS

44. Data Management Seminar, 8-11th July 2008, Hamburg On-line Data Collection in MS True mixed mode design Goal: –Reduce response burden by offering respondents preferred mode –Save time on data entry and data processing –Improve quality of data Teachers and school principals should be assigned to the preferred mode –Knowlegde of NRC about feasibility/preferences –School coordinator could contactS school principal and teachers Ensure fall-back to paper questionnaire

45. Data Management Seminar, 8-11th July 2008, Hamburg Codebook Adaptation Instrument Preparation Sampling and Preparing Administration Data Entry and Verification Administration Receiving Material Data Sendout Codebook Adaptation Prepare ODC Monitor ODC

46. Data Management Seminar, 8-11th July 2008, Hamburg School contactInstrumentstime Time planning before administration ODC Contact schools Sample students / teachers Send tracking forms Send questionnaires Implement changes & assemble booklets Adapt & translate instruments Printing Data collection Printing Prepare ODC Send quest./cover letters Send listing forms Sample classes

47. Data Management Seminar, 8-11th July 2008, Hamburg On-line Data Collection (ODC) Based on concept of cultures, one setup per language and country Participants receive “cover letter” –Label with login information provided by WinW3S –Culture specific URL –Personal username and password –Sent to school coordinator for distribution Same administration period as paper questionnaires

48. Data Management Seminar, 8-11th July 2008, Hamburg On-line Data Collection (ODC) NRCs can monitor completion status of on-line questionnaires If necessary, NRCs instruct School Coordinators to remind teachers/school principals to complete the questionnaire No data entry at National Center needed

49. Data Management Seminar, 8-11th July 2008, Hamburg Support Material

50. Data Management Seminar, 8-11th July 2008, Hamburg Support Material Survey Operations Procedures Manual (MS) –Unit 1: School contact and material preparation –Unit 2: Within school sampling and test administration School Coordinator Manual Test Administrator Manual National Quality Control Monitor Manual Within School Sampling Software (WinW3S)

51. Data Management Seminar, 8-11th July 2008, Hamburg Support Material Survey Operations Procedures Manual (MS) –Unit 3: On-line data collection, scoring and data entry IEA SurveySystem Designer Scoring Guide Codebooks Windows Data Entry Manager (WinDEM)  Distribution of all materials and software packages through secure section of study website at http://iccs.acer.edu.au

52. Data Management Seminar, 8-11th July 2008, Hamburg Main Survey Timeline

53. Data Management Seminar, 8-11th July 2008, Hamburg Main Survey Timeline - SH - Receive verified final instruments - Start printing and distribution of questionnaires - Send final instruments and NAF v. IV to IEA DPC - Set up on-line questionnaires October 10, 2008 Submit assembled instruments and NAF version III for layout verification to ACER September 26, 2008 Receive verified instrumentSeptember 19, 2008 Submit translated questionnaires and NAF version II for international translation verification to IEA Sec. August 22, 2008 Receive feedback on national adaptationsJuly 25, 2008 Submit national adaptations in NAF version I to ISCJuly 18, 2008 Receive instruments & NAFJuly 11, 2008

54. Data Management Seminar, 8-11th July 2008, Hamburg Main Survey Timeline - SH Final deadline for submitting data to DPCMarch 27, 2009 - Scoring of open-ended items and occupations - Data entry and verification November – March, 2009 Proposed MS administration period (subject to country conditions) October – December, 2008 Submit on-line questionnaires definition files for verification to DPC October 17, 2008 Give feedback about MS procedures using the on-line Survey Activities Questionnaire April, 2009

55. Data Management Seminar, 8-11th July 2008, Hamburg Main Survey Timeline – NH - Receive verified final instruments - Start printing and distribution of questionnaires - Send final instruments and NAF v. IV to IEA DPC - Set up on-line questionnaires February, 2008 Submit assembled instruments and NAF version III for layout verification to ACER February, 2008 Translation verificationJanuary, 2008 Translate questionnaires and prepare NAF version II for international translation verification at IEA Sec. December, 2008 Receive feedback on national adaptationsNovember, 2008 Submit national adaptations in NAF version I to ISCNovember, 2008 Receive instruments & NAFJuly 11, 2008

56. Data Management Seminar, 8-11th July 2008, Hamburg Main Survey Timeline - NH Final deadline for submitting data to DPCAugust 28, 2009 - Scoring of open-ended items and occupations - Data entry and verification April – August, 2009 Proposed MS administration period (subject to country conditions) March – May, 2009 Submit on-line questionnaires definition files for verification to DPC February, 2008 Give feedback about MS procedures using the on-line Survey Activities Questionnaire September, 2009

57. Data Management Seminar, 8-11th July 2008, Hamburg Next Steps

58. Data Management Seminar, 8-11th July 2008, Hamburg Next Steps National centers should plan and prepare for –School contact –Adaptation & translation of instruments –Test administrators –Scorers (CCE items and occupations) –Data management (data entry and verification) Be prepared to receive Main Survey instruments on July 11, 2008

59. Data Management Seminar, 8-11th July 2008, Hamburg Contact Information

60. Data Management Seminar, 8-11th July 2008, Hamburg Contact Information Field Operations / Data Management: IEA Data Processing and Research Center Falk Brese Michael Jung iccs@iea-dpc.de

61. Data Management Seminar, 8-11th July 2008, Hamburg Thank you for your attention!