1. Data Management Seminar, 9-12th July 2007, Hamburg 11 ICCS 2009 – On-line Data Collection A General Introduction

2. Data Management Seminar, 9-12th July 2007, Hamburg 2 Content History Potential Key Features of the IEA SurveySystem Integration with Survey Operations Guidelines Restrictions Timeline

3. Data Management Seminar, 9-12th July 2007, Hamburg 3 History Of Project Interest for On-line Data Collection from within and outside IEA in 2003/2004 (SITES, PISA CBAS, PISA 2009). 1st implementation in the IEA SITES 2006 (Second Information Technology in Education Study), which had an near-ideal match between research domains and methodology. Currently used for OECD TALIS.

4. Data Management Seminar, 9-12th July 2007, Hamburg 4 History Of Project Staged development and implementation in SITES: –Technical try-out in ~10 countries (Spring 2005) –Split-sample design in field trial in 17 countries (Summer 2005) –FT analysis showed Feasibility: ODC methodology and procedures work in the context of an international, educational survey Validity: no contraindication that the two modes yield comparable data, thus allowing the implementation of both modes in and across countries. Problems were identified with respect to drop-out and reliable administration of fallback paper questionnaires.

5. Data Management Seminar, 9-12th July 2007, Hamburg 5 Potential On-line Data Collection (ODC) provides an alternative channel for respondents. One-step printing, postage, data collection and entry! Options not available on paper. Reduced workload for data entry and preparing datasets at National Center! For returned data processing steps are collapsed or eliminated and it requires little or no verification as particular sources of error are eliminated. But, this means added complexity and increased workload for NRCs and IEA DPC. More tasks go on in sequence and in parallel.

6. Data Management Seminar, 9-12th July 2007, Hamburg 6 SurveySystem - Key Features Hierarchical, structure oriented survey model that keeps all information, including text passages, translations, and adaptations in files that can be easily send by email. Three application components: –‘Designer’ to flexibly define survey components (e.g., questionnaire sections, questions etc.), translate text passages and preview; –‘Web’ for respondents around the world to complete their personalized questionnaire; –‘Monitor’ to audit and track participation in real-time.

7. Data Management Seminar, 9-12th July 2007, Hamburg 7 SurveySystem - Key Features Based on the concept of “cultures”, for example ‘fr-CH’ stands for: French in Switzerland. Hosted and maintained by IEA DPC.

8. Data Management Seminar, 9-12th July 2007, Hamburg 8 ‘Designer’ Off-line application that runs on Windows PCs in National Centers (no internet connection required). Works from English survey definition files received from IEA DPC. Role based: ‘Administrator’, ‘Manager’ and ‘Translator’. Used to create, delete, disable and edit survey components and their properties. Allows for translation of all text passages and “system resources” (e.g., ‘Next’-button or validation messages). Integrated preview system (built-in web server).

9. Data Management Seminar, 9-12th July 2007, Hamburg 9 ‘Designer’ Screenshot

10. Data Management Seminar, 9-12th July 2007, Hamburg 10 ‘Web’ – “Respondent-friendly Design” Information collected on specific content domains, but we are not assessing computer literacy!  Web interface and usage has to be simple. Standard features: login, save-and-resume, page-by- page navigation. Accessible from any place at any time. No setups. No downloads. Includes a Table of Contents with visual feedback about which questions have been answered or not

11. Data Management Seminar, 9-12th July 2007, Hamburg 11 ‘Web’ – Technical Details Works with ‘thin client’ (internet browser) with minimum requirements: –800x600 screen size, 256 colours –HTML 4.0 (standard since 1998) for tables –‘Dir’ attribute and bi-directional Unicode support –Cascading Style Sheets v1 support Supported browsers that meet these requirements (~99,9%): –Internet Explorer 5.0+, Netscape 5.0+, Mozilla/Firefox 1.0+, Safari 1.0, Opera 7.0+, all regardless of platform No “fancy” technologies like cookies, JavaScript, or pop- ups required!

12. Data Management Seminar, 9-12th July 2007, Hamburg 12 ‘Web’ – Login Screenshot

13. Data Management Seminar, 9-12th July 2007, Hamburg 13 ‘Web’ – Matrix Question Screenshot

14. Data Management Seminar, 9-12th July 2007, Hamburg 14 ‘Web’ – Table of Contents Screenshot

15. Data Management Seminar, 9-12th July 2007, Hamburg 15 Real-time on-line accessible information about participation and completion status Provides summary statistics (participation rates) Filter can be set to highlight recent actvities, e.g. respondents who logged in the last two days ‘Monitor’

16. Data Management Seminar, 9-12th July 2007, Hamburg 16 ‘Monitor’ Screenshot

17. Data Management Seminar, 9-12th July 2007, Hamburg 17 ‘Monitor’ Screenshot

18. Data Management Seminar, 9-12th July 2007, Hamburg 18 ‘Monitor’ Screenshot

19. Data Management Seminar, 9-12th July 2007, Hamburg 19 Implications for Survey Operations Before and during survey administration, additional tasks are necessary in parallel and sequence to the paper instrument preparation and administration. The following slides will give you a visual, side-by-side and detailed overview of these steps.

20. Data Management Seminar, 9-12th July 2007, Hamburg 20 ‘Paper’ administration with questionnaires only. Use Teacher Listing/Tracking Forms. Use National Adaptation Forms. SurveySystem not used. Manually adapt WinDEM codebooks to reflect national adaptations. Use WinW3S labels. Entire data entry with WinDEM. Track participation with WinW3S based on Teacher Tracking Forms. ‘Paper’ and ‘On-line’ administration (split design in FT) with questionnaires AND cover letters. Use Teacher Listing/Tracking Forms. Use National Adaptation Forms. Convert and adapt questionnaires with SurveySystem AFTER translation/layout verification. Export codebooks (no alterations allowed after export!). Use WinW3S labels for both modes. Enter ‘paper’ data only with WinDEM. (‘on-line’ data merged at IEA DPC) Track participation with WinW3S, based on Teacher Tracking Forms and ‘Monitor’ reports. Mixed Paper/On-line Paper only

21. Data Management Seminar, 9-12th July 2007, Hamburg 21 Instrument Preparation Sampling and Preparing Survey Administration Data Entry and Verification SURVEYSURVEY Codebook Adaptation Receiving Material Data Sendout Scoring Done by SurveySystem. Convert for ODC after process is complete for paper instruments. Determine and track administration mode. Monitor participation on-line and follow- up. No data to enter, just participation status. Send “cover letters” instead of questionnaires. No data to send. It’s at the IEA DPC already.

22. Data Management Seminar, 9-12th July 2007, Hamburg 22 Integration – Sampling School sampling is not affected in terms of coverage and selection Within-School Sampling –WinW3S allows you to assign the questionnaire type (paper or on-line) per default for principals and/or teachers –WinW3S allows you to track the questionnaire type for individuals that require the other type (e.g., a school that was supposed to fill all questionnaires on-line, but in which some teachers requested paper questionnaires - FT only on-line to paper)

23. Data Management Seminar, 9-12th July 2007, Hamburg 23 Integration – Instruments Instrument Preparation –International definition files prepared by IEA DPC. –National adaptations to structure within countries. –“Cut and Paste” conversion AFTER paper translation, within country and international translation verification, and layout verification.

24. Data Management Seminar, 9-12th July 2007, Hamburg 24 Integration – Instruments Instrument Verification –Within country content comparison and optical check (final paper versions vs. on-line) –NRCs send localized definitions to IEA DPC –Final optical / logical checks against paper and NAF –Approval or request for corrections from IEA DPC –IEA DPC sets up survey for the Internet. Codebook Adaptation –WinDEM codebooks are automatically created!

25. Data Management Seminar, 9-12th July 2007, Hamburg 25 Integration – Administration On-line respondents, usually entire schools, receive “cover letters” instead of questionnaires. Same WinW3S labels used. Credentials created with WinW3S are distributed to respondents together with a culture-specific and typo- safe login address, for instance: http://www.iccs2009.net/odc/71792 Administration period applies to both modes.

26. Data Management Seminar, 9-12th July 2007, Hamburg 26 Integration – Data Entry and Verification Trivial: No need to enter data! Track participation in WinW3S from downloadable ‘Monitor’ reports (“returned, data available” vs. “no data available”) and teacher tracking forms (for other participation indicators, like “left school permanently”). ‘On-line’ data merged with ‘paper’ data from WinDEM at IEA DPC. Verification: Done at the IEA DPC for the FT and MS.

27. Data Management Seminar, 9-12th July 2007, Hamburg 27 Guidelines Overall goal: highest possible response rates, both on questionnaire as well as variable level. Decision and responsibility of usage lies with NRCs, who are generally free to decide about type and scope of implementation, i.e. as the default mode, for principals and/or teachers, or for specific schools or individuals. Individual restrictions in terms of computer/Internet penetration, poor on-line response rates in previous surveys or other obstacles should be taken into account. Eventually, the results of the field trial in terms of response rate and drop-out should become the basis for decision.

28. Data Management Seminar, 9-12th July 2007, Hamburg 28 Restrictions While participation is optional, participation in the FT is mandatory if you want to use ODC in the main run. On-line data collection is administered with the IEA SurveySystem ONLY. The IEA DPC can not manage the verification and data conversion for other systems.

29. Data Management Seminar, 9-12th July 2007, Hamburg 29 Restrictions On-line mode may become the default mode in your country for the MS, but it is MANDATORY to provide paper questionnaires to individuals/schools without Internet access, equipment, or that simply refuse. For the MS National Centers should therefore check with schools whether the on-line mode is acceptable for them BEFORE send-out of materials. In other words, school coordinators should not need to ask for paper questionnaires if they prefer them. Enquiries should take place during the first contact/notification of schools or together with the send-out of teacher listing forms.

30. Data Management Seminar, 9-12th July 2007, Hamburg 30 Field Trial Timeline Documentation of software is part of the ICCS 2009 Survey Operations Procedures Manual Unit 3. International definition files distributes together with SurveySystem software (and if applicable a short list of deviations from the paper version) on August 20, 2007 or earlier. “Copy-and-Paste” conversion may start AFTER translation AND layout verification for paper questionnaires is completed, starting October 15, 2007.

31. Data Management Seminar, 9-12th July 2007, Hamburg 31 Field Trial Timeline Submit zipped definition files to IEA DPC within one week after layout verification approval, latest by October 22, 2007. Allow at least one week for review and paper/on-line comparison at IEA DPC. Begin paper and on-line administration from November 5, 2007.

32. Data Management Seminar, 9-12th July 2007, Hamburg 32 Summary Participation is optional and should be guided by the level of confidence that participants have with the mode. FT participation is mandatory and vital to collect experience for the main data collection implementation. Fall-back paper questionnaires are mandatory (ODC can not be the exclusive mode for the time being).

33. Data Management Seminar, 9-12th July 2007, Hamburg 33 Summary More tasks going on in sequence and parallel in terms of preparation and monitoring. Challenge to assign and track the correct questionnaire mode (on-line or paper) to individuals in schools. Some may agree to participate on-line, but others will require paper versions. Advantages gained from reduced printing and shipping costs, less workload during data entry and preparation. Software and hosting will be provided by the IEA DPC.

34. Data Management Seminar, 9-12th July 2007, Hamburg 34 Thank you for your attention!