1. Assessment of the Swiss National Objectives in Education
Ingo Barkow, Catharina Wasner
University of Applied Sciences HTW Chur, Switzerland
EDDI16 – 8th Annual European DDI User Conference
December 6, 2016, Cologne

2. Surveys about the national objectives in education in Switzerland
Overview
Surveys about the national objectives in education in Switzerland
Tasks of HTW Chur: IT support and data management
A tool for survey management
Where DDI helps
Where DDI does not help
Other metadata standards to be considered

3. Background: «EDK» and «HarmoS»
In Switzerland, the main responsibility for education and culture lies with the cantons. They coordinate their work at the national level.
The 26 cantonal ministers of education (called “directors of education”) together form a political body to carry out this work: the Swiss Conference of Cantonal Ministers of Education (EDK).
Legally binding, intercantonal agreements (known as concordats) form the foundation for the work of the EDK.
One of these agreements is the HarmoS concordat, the Intercantonal Agreement on the Harmonization of Compulsory Education.
It came into effect in 2009 and is aimed at harmonizing some cornerstones of the educational system nationwide.

4. The Swiss National Objectives in Education and «ÜGK»
The Swiss National Objectives in Education are some of these cornerstones of the HarmoS concordat.
They were developed until 2011 and describe which core skills students in all cantons should have obtained after 2, 6 and 9 years of school in languages, math and natural sciences.
They are now part of all curriculums in all cantons.
In 2013 it was decided how to assess these objectives and if the harmonization has been successful.
Therefor the Swiss National Core Skills Assessment Programme (in German: ÜGK – Überprüfung der Grundkompetenzen) was initiated.
Samples in each canton: all in all students
First survey in spring 2016: math, 9th school year
Second Survey in spring 2017: languages, 6th school year
The cantons are thinking about institutionalizing these assessments and coordinating them with the PISA assessments
15 Partners conducting theses surveys

5. Tasks of HTW Chur in «ÜGK»
Leading the work package IT support and data management
IT support:
Coordinating and organizing the hardware with the service provider
Coordinating the use and set up of software and servers with DIPF
Optimizing and developing the tools for the survey management
Data management:
Introducing and optimizing the data management
Introducing metadata standards for educational data
Consulting for FORS regarding the preservation of educational data and metadata

6. A Tool for Survey Management
Will be developed for PISA and ÜGK
Development starts probably in spring 2017
The tool should support the fieldwork and data management in an adequate manner:
Support the administration and field monitoring during the data collection process
Optimize the data documentation process by generating documentation und metadata as a byproduct without additional effort during the data collection process
Tool bases on a modified version of Rogatus Survey 2.0 (specifically the case management system)
Current version only supports CAPI – will be modified to use case “data collections in schools”
Should be able to use different metadata standards for preservation

7. Metadata-driven research process
Research project information stored in survey management system
Items enriched with metadata during design process and stored in item bank
Data collection using survey / case management system thus creating paradata / log files
Storing of all metadata, paradata and data in research data facility
Metadata and data are linked to existing publication (enhanced publication)
All items and results are available for re-use in research data facility
Research data facility ensures long-time preservation by rollover processes for format changes
Here are examples for all processes

8. Metadata-driven Survey Management
Software: IAB Metadata Portal (TBA21)
Metadata Standard: DDI Lifecycle 3.2

9. Metadata-driven Item Design & Item Banking
Software: TIPO Item Portal (TBA21/DIPF) / TAO (o.a.t. S.A.)
Metadata Standard: QTI v2.1

10. Metadata-driven Data Collection Process (Mobile)
Software: Rogatus Survey / Aitema Mobile Client (TBA21)
Metadata Standard: DDI Lifecycle 3.2 (not for paradata)

11. Metadata-driven Data Dissemination
Software: Rogatus Survey / Administrator Panel (TBA21)
Metadata Standard: DDI Lifecycle 3.2

12. Metadata-driven Publication Process
Software: Invenio 0.99 (CERN)
Metadata Standards: MARC21 / DataCite

13. Metadata-driven Secondary Data Analysis / Retrospective
Software: FORSbase (FORS)
Metadata Standard: DDI Lifecycle 3.2

14. Where DDI can help
Research process in the educational sciences is very similar to the social sciences
Most large-scale studies (e.g. PISA, PIAAC, PIRLS, TIMMS) use a mix of questionnaires and cognitive items
DDI Lifecycle v3.2 can be used in “ÜGK” to store survey information and questionnaire design including routing
It can be the standard to forward these metadata and data from the data collection towards the data archive
Currently it can be used for the questionnaires as well as very simple cognitive items (items with a simple stimulus like a picture plus a multiple choice battery)

15. Where DDI currently cannot help
DDI Lifecycle v3.2 is lacking the following features for computer-based assessment
Layout and screen size to avoid changes in item difficulty
Scoring rules
Response domains like “point-and-click”, “hotspot” or “highlighting”
Use of paradata / logfile data analysis e.g. for diagnostic assessment
Statistical parameters (e.g. Cronbach‘s Alpha) have to be stored with the cognitive item to support adaptive testing
Support for more complex item types like simulations
Very important are also the modifications due to the shift to DDI Lifecycle 4.0
Does DDI4 contain the same routing capabilities like DDI3.2 so a questionnaire or test can be rendered directly?
Otherwise DDI will be useless to survey organisations

16. PIAAC Problem Solving Item
DIPF PowerPoint-Präsentation

17. Other metadata standards in ÜGK
Some interactions from IMS Questionnaire and Test Interoperability (QTI)
ChoiceInteraction (multiple choice items with one or more correct answers)
OrderInteraction (sorting answers in a list)
AssociateInteraction (building pairs of answers)
MatchInteraction (assigning correct responses to a matrix)
GapMatchInteraction (matrix with open answers)
InlineChoiceInteraction (choice of different answers within a text)
TextEntryInteraction (open text boxes within different media)
ExtendedTextInteraction (longer text box for free text passages)
HotTextInteraction (choices of several text answers within a text)
HotSpotInteraction (clickable areas in a graphic)
SelectPointInteraction (marking an area in a graphic)
GraphicOrderInteraction (marking hotspots with numbers)
GraphicAssociateInteraction (building relations between hotspots in graphics)
GraphicGapMatchInteraction (graphics with text boxes to enter data)
PositionObjectInteraction (similar to HotSpot, but a predefined graphic can be used)
SliderInteraction (offers a slider bar for a numeric answer)
DrawingInteraction (offers a sketchboard for drawing)
UploadInteraction (data created by another tool can be attached as an answer)

18. Conclusion
DDI 3.2 is well suited for questionnaires, but is lacking some features to be the leading standard for educational sciences
More work on paradata and response domains is needed if this domain is supposed to be included as well
Nevertheless it is the standard of choice for many parts of the ÜGK survey

19. Thank you for your attention!