Assessment of the Swiss National Objectives in Education Ingo Barkow, Catharina Wasner University of Applied Sciences HTW Chur, Switzerland EDDI16 – 8th Annual European DDI User Conference December 6, 2016, Cologne

Surveys about the national objectives in education in Switzerland Overview Surveys about the national objectives in education in Switzerland Tasks of HTW Chur: IT support and data management A tool for survey management Where DDI helps Where DDI does not help Other metadata standards to be considered

Background: «EDK» and «HarmoS» In Switzerland, the main responsibility for education and culture lies with the cantons. They coordinate their work at the national level. The 26 cantonal ministers of education (called “directors of education”) together form a political body to carry out this work: the Swiss Conference of Cantonal Ministers of Education (EDK). Legally binding, intercantonal agreements (known as concordats) form the foundation for the work of the EDK. One of these agreements is the HarmoS concordat, the Intercantonal Agreement on the Harmonization of Compulsory Education. It came into effect in 2009 and is aimed at harmonizing some cornerstones of the educational system nationwide.

The Swiss National Objectives in Education and «ÜGK» The Swiss National Objectives in Education are some of these cornerstones of the HarmoS concordat. They were developed until 2011 and describe which core skills students in all cantons should have obtained after 2, 6 and 9 years of school in languages, math and natural sciences. They are now part of all curriculums in all cantons. In 2013 it was decided how to assess these objectives and if the harmonization has been successful. Therefor the Swiss National Core Skills Assessment Programme (in German: ÜGK – Überprüfung der Grundkompetenzen) was initiated. Samples in each canton: all in all 25.000 students First survey in spring 2016: math, 9th school year Second Survey in spring 2017: languages, 6th school year The cantons are thinking about institutionalizing these assessments and coordinating them with the PISA assessments 15 Partners conducting theses surveys

Tasks of HTW Chur in «ÜGK» Leading the work package IT support and data management IT support: Coordinating and organizing the hardware with the service provider Coordinating the use and set up of software and servers with DIPF Optimizing and developing the tools for the survey management Data management: Introducing and optimizing the data management Introducing metadata standards for educational data Consulting for FORS regarding the preservation of educational data and metadata

A Tool for Survey Management Will be developed for PISA and ÜGK Development starts probably in spring 2017 The tool should support the fieldwork and data management in an adequate manner: Support the administration and field monitoring during the data collection process Optimize the data documentation process by generating documentation und metadata as a byproduct without additional effort during the data collection process Tool bases on a modified version of Rogatus Survey 2.0 (specifically the case management system) Current version only supports CAPI – will be modified to use case “data collections in schools” Should be able to use different metadata standards for preservation

Metadata-driven research process Research project information stored in survey management system Items enriched with metadata during design process and stored in item bank Data collection using survey / case management system thus creating paradata / log files Storing of all metadata, paradata and data in research data facility Metadata and data are linked to existing publication (enhanced publication) All items and results are available for re-use in research data facility Research data facility ensures long-time preservation by rollover processes for format changes Here are examples for all processes

Metadata-driven Survey Management Software: IAB Metadata Portal (TBA21) Metadata Standard: DDI Lifecycle 3.2

Metadata-driven Item Design & Item Banking Software: TIPO Item Portal (TBA21/DIPF) / TAO (o.a.t. S.A.) Metadata Standard: QTI v2.1

Metadata-driven Data Collection Process (Mobile) Software: Rogatus Survey / Aitema Mobile Client (TBA21) Metadata Standard: DDI Lifecycle 3.2 (not for paradata)

Metadata-driven Data Dissemination Software: Rogatus Survey / Administrator Panel (TBA21) Metadata Standard: DDI Lifecycle 3.2

Metadata-driven Publication Process Software: Invenio 0.99 (CERN) Metadata Standards: MARC21 / DataCite

Metadata-driven Secondary Data Analysis / Retrospective Software: FORSbase (FORS) Metadata Standard: DDI Lifecycle 3.2

Where DDI can help Research process in the educational sciences is very similar to the social sciences Most large-scale studies (e.g. PISA, PIAAC, PIRLS, TIMMS) use a mix of questionnaires and cognitive items DDI Lifecycle v3.2 can be used in “ÜGK” to store survey information and questionnaire design including routing It can be the standard to forward these metadata and data from the data collection towards the data archive Currently it can be used for the questionnaires as well as very simple cognitive items (items with a simple stimulus like a picture plus a multiple choice battery)

Where DDI currently cannot help DDI Lifecycle v3.2 is lacking the following features for computer-based assessment Layout and screen size to avoid changes in item difficulty Scoring rules Response domains like “point-and-click”, “hotspot” or “highlighting” Use of paradata / logfile data analysis e.g. for diagnostic assessment Statistical parameters (e.g. Cronbach‘s Alpha) have to be stored with the cognitive item to support adaptive testing Support for more complex item types like simulations Very important are also the modifications due to the shift to DDI Lifecycle 4.0 Does DDI4 contain the same routing capabilities like DDI3.2 so a questionnaire or test can be rendered directly? Otherwise DDI will be useless to survey organisations

PIAAC Problem Solving Item DIPF PowerPoint-Präsentation

Other metadata standards in ÜGK Some interactions from IMS Questionnaire and Test Interoperability (QTI) ChoiceInteraction (multiple choice items with one or more correct answers) OrderInteraction (sorting answers in a list) AssociateInteraction (building pairs of answers) MatchInteraction (assigning correct responses to a matrix) GapMatchInteraction (matrix with open answers) InlineChoiceInteraction (choice of different answers within a text) TextEntryInteraction (open text boxes within different media) ExtendedTextInteraction (longer text box for free text passages) HotTextInteraction (choices of several text answers within a text) HotSpotInteraction (clickable areas in a graphic) SelectPointInteraction (marking an area in a graphic) GraphicOrderInteraction (marking hotspots with numbers) GraphicAssociateInteraction (building relations between hotspots in graphics) GraphicGapMatchInteraction (graphics with text boxes to enter data) PositionObjectInteraction (similar to HotSpot, but a predefined graphic can be used) SliderInteraction (offers a slider bar for a numeric answer) DrawingInteraction (offers a sketchboard for drawing) UploadInteraction (data created by another tool can be attached as an answer)

Conclusion DDI 3.2 is well suited for questionnaires, but is lacking some features to be the leading standard for educational sciences More work on paradata and response domains is needed if this domain is supposed to be included as well Nevertheless it is the standard of choice for many parts of the ÜGK survey

Thank you for your attention!