Metadata in the Educational Sciences Dr. Ingo Barkow, Associate Professor for Operational Data Management

Agenda Introduction Metadata in the educational sciences The Swiss National School Monitoring (ÜGK / COFO) An example process for metadata management in ÜGK Where DDI can help Where DDI (currently) cannot help Q&A

Obligatory Tourism Slide University of Applied Sciences HTW Chur 1.600 Students 3 Departments (Tourism, Management, Information Science) Swiss Institute for Information Science Master of Science (MSc) in Information Science and Data Management Project Team for Research Data Management

Metadata in the educational sciences Research process in the educational sciences is very similar to the social sciences Models like the Generic Longitudinal Business Process Model (GLBPM) can also be applied here Most large-scale studies (e.g. PISA, PIAAC, PIRLS, TIMMS) use a mix of questionnaires and cognitive items Shift from paper-based assessment towards computer-based assessment Metadata standards (e.g. QTI, SCORM) exist, but only for very specific parts of the research process

Metadata in the educational sciences Specific challenges in computer-based assessment Layout and screen size must be fixed to avoid changes in item difficulty Scoring rules have to be implemented (and documented) Very complex item types (e.g. simulations) Mode effects (e.g. paper to computer, computer to tablet) Use of logfile data analysis e.g. for diagnostic assessment Statistical parameters (e.g. Cronbach‘s Alpha) have to be stored to support adaptive testing Items are usually highly confidential (e.g. SAT, GMAT)

Other metadata standards in ÜGK Some interactions from IMS Questionnaire and Test Interoperability (QTI) ChoiceInteraction (multiple choice items with one or more correct answers) OrderInteraction (sorting answers in a list) AssociateInteraction (building pairs of answers) MatchInteraction (assigning correct responses to a matrix) GapMatchInteraction (matrix with open answers) InlineChoiceInteraction (choice of different answers within a text) TextEntryInteraction (open text boxes within different media) ExtendedTextInteraction (longer text box for free text passages) HotTextInteraction (choices of several text answers within a text) HotSpotInteraction (clickable areas in a graphic) SelectPointInteraction (marking an area in a graphic) GraphicOrderInteraction (marking hotspots with numbers) GraphicAssociateInteraction (building relations between hotspots in graphics) GraphicGapMatchInteraction (graphics with text boxes to enter data) PositionObjectInteraction (similar to HotSpot, but a predefined graphic can be used) SliderInteraction (offers a slider bar for a numeric answer) DrawingInteraction (offers a sketchboard for drawing) UploadInteraction (data created by another tool can be attached as an answer)

ÜGK Mathematics Item DIPF PowerPoint-Präsentation

PIAAC Problem Solving Item DIPF PowerPoint-Präsentation

CBA Itembuilder MicroFIN item DIPF PowerPoint-Präsentation

Background: «EDK» and «HarmoS» In Switzerland, the main responsibility for education and culture lies with the cantons. They coordinate their work at the national level. The 26 cantonal ministers of education (called “directors of education”) together form a political body to carry out this work: the Swiss Conference of Cantonal Ministers of Education (EDK). Legally binding, intercantonal agreements (known as concordats) form the foundation for the work of the EDK. One of these agreements is the HarmoS concordat, the Intercantonal Agreement on the Harmonization of Compulsory Education. It came into effect in 2009 and is aimed at harmonizing some cornerstones of the educational system nationwide.

The Swiss National Objectives in Education and «ÜGK» The Swiss National Objectives in Education are some of these cornerstones of the HarmoS concordat. They were developed until 2011 and describe which core skills students in all cantons should have obtained after 2, 6 and 9 years of school in languages, math and natural sciences. They are now part of all curriculums in all cantons. In 2013 it was decided how to assess these objectives ad if the harmonization has been successful. Therefor the Swiss National Core Skills Assessment Programme (in German: ÜGK – Überprüfung der Grundkompetenzen) was initiated. First survey in spring 2016: math, 9th school year Second Survey in spring 2017: languages, 6th school year The cantons are thinking about institutionalizing these assessments and coordinating them with the PISA assessments 15 Partners conducting theses surveys

Tasks of HTW Chur in «ÜGK» Leading the work package IT support and data management IT support: Coordinating and organizing the hardware with the service provider Coordinating the use and set up of software and servers with DIPF Optimizing and developing the tools for the survey management Data management: Introducing and optimizing the data management Introducing metadata standards for educational data Consulting for FORS regarding the preservation of educational data and metadata

A Tool for Survey Management Will be developed for PISA and ÜGK Development started in spring 2017 The tool should support the fieldwork and data management in an adequate manner: Support the administration and field monitoring during the data collection process Optimize the data documentation process by generating documentation und metadata as a byproduct without additional effort during the data collection process Tool bases on a modified version of Rogatus Survey 2.0 (specifically the case management system) Current version only supports CAPI – will be modified to use case “data collections in schools” Should be able to use different metadata standards for preservation

Metadata-driven research process Research project information stored in survey management system Items enriched with metadata during design process and stored in item bank Data collection using survey / case management system thus creating paradata / log files Storing of all metadata, paradata and data in research data facility Metadata and data are linked to existing publication (enhanced publication) All items and results are available for re-use in research data facility Research data facility ensures long-time preservation by rollover processes for format changes Here are examples for all processes with predecessors of the upcoming toolset

Metadata-driven Survey Management Software: IAB Metadata Portal (TBA21) Metadata Standard: DDI Lifecycle 3.2

Metadata-driven Item Design & Item Banking Software: TIPO Item Portal (TBA21/DIPF) / TAO (o.a.t. S.A.) Metadata Standard: QTI v2.1

Metadata-driven Data Collection Process (Mobile) Software: Rogatus Survey / Aitema Mobile Client (TBA21) Metadata Standard: DDI Lifecycle 3.2 (not for paradata)

Metadata-driven Data Dissemination Software: Rogatus Survey / Administrator Panel (TBA21) Metadata Standard: DDI Lifecycle 3.2

Metadata-driven Publication Process Software: Invenio 0.99 (CERN) Metadata Standards: MARC21 / DataCite

Metadata-driven Secondary Data Analysis / Retrospective Software: FORSbase (FORS) Metadata Standard: DDI Lifecycle 3.2

Where DDI can help DDI Lifecycle v3.2 can be used in “ÜGK” to store survey information and questionnaire design including routing It can be the standard to forward these metadata and data from the data collection towards the data archive Currently it can be used for the questionnaires as well as very simple cognitive items (items with a simple stimulus like a picture plus a multiple choice battery) Upcoming survey tool is DDI-based as well as the repository at FORS New project between HTW and FORS to create common DDI Profile based on former IAB metadata system (meaning it also contains common elements from previous versions of Questasy and Colectica)

Where DDI (currently) cannot help DDI Lifecycle v3.2 is lacking the following features for computer-based assessment Layout and screen size to avoid changes in item difficulty Scoring rules Response domains like “point-and-click”, “hotspot” or “highlighting” Use of paradata / logfile data analysis e.g. for diagnostic assessment Statistical parameters (e.g. Cronbach‘s Alpha) have to be stored with the cognitive item to support adaptive testing Support for more complex item types like simulations Very important are also the modifications due to the shift to DDI Lifecycle 4.0 Does DDI4 contain the same routing capabilities like DDI3.2 so a questionnaire or test can be rendered directly? Otherwise DDI will be useless to survey organisations

Thanks for your attention.