3rd International Digital Curation Conference Washington, DC, Dec 2007 Paper Presentations: Interoperability, Metadata & Standards Data Documentation Initiative: Toward a Standard for the Social Sciences Mary Vardigan, Pascal Heus, Wendy Thomas ICPSR/University of Michigan / Open Data Foundation / Minnesota Population Center / /

DDI Alliance – What is Metadata? Common definition: Data about Data Unlabeled stuffLabeled stuff The bean example is taken from: A Managers Introduction to Adobe eXtensible Metadata Platform,

DDI Alliance – Managing data and metadata is challenging! We are in charge of the data. We support our users but also need to protect our respondents! We want easy access to high quality and well documented data! We need to collect the information from the producers, preserve it, and provide access to our users! Producers Librarians Users General Public Policy Makers Sponsors Media/Press Academic Business Government We have an information management problem

DDI Alliance – Metadata issues Without producer / archive metadata –researchers cant work discover data or perform efficient analysis Without researcher metadata –Research process is not documented and cannot be reproduced (Gary King replication standard!) –Other researchers are not aware of what has been done (duplication / lack of visibility) –Producer dont know about data usage and quality issues Without standards –Such information cant be properly managed and exchanged between actors or with the public Without tools: –We cant capture, preserve or share knowledge

DDI Alliance – XML to the rescue! XML stands for eXtensible Markup Language Technology that is driving todays web service oriented architecture of the Internet and Intranets Using XML, we can capture, structure, transform, discover, exchange, query, edit and secure metadata and data XML is platform & language independent and can be used by everyone XML is both machine and human readable XML is non-proprietary, public domain and many open tools exist Domain specific standards are available!

DDI Alliance – Suggested XML metadata specifications for socio-economic data Statistical Data and Metadata Exchange (SDMX) –Macrodata, time series, indicators, registries – Data Documentation Initiative (DDI) –Microdata (surveys, studies) – ISO –Semantic modeling, concepts, registries – ISO –Geography – Dublin Core –Resources (documentation, images, multimedia) –

DDI Alliance – The Data Documentation Initiative (DDI) International XML based specification for the documentation of social and behavioral data –Started in 1995, now driven by DDI Alliance (30+ members) –Became XML specification in 2000 (v1.0) –Current version is 2.1 with focus on archiving (survey/codebook) New Version 3.0 (2008) –Focus on entire survey Life Cycle –Provide comprehensive metadata on the entire survey process and usage –Aligned on other metadata standards (DC, MARC, ISO 11179, SDMX, …) –Include machine actionable elements to facilitate processing, discovery and analysis DDI is being adopted by producers/archives but needs to extends to the researchers (who are using the data!)

DDI Alliance – DDI 3.0 and the Survey Life Cycle A survey is not a static process: It dynamically evolved across time and involves many agencies/individuals DDI 2.x is about archiving, DDI 3.0 across the entire life cycle 3.0 focus on metadata reuse (minimizes redundancies/discrepancies, support comparison) Also supports multilingual, grouping, geography, and others 3.0 is extensible

DDI Alliance – Metadata Components Producer metadata: –Codebook, questionnaires, reports, methodologies, processing, scripts, quality, admin, etc. Research metadata –Recodes, analysis, table, scripts, papers, logs, data quality, usage –Citations, references –Activities, discussions, knowledge base Outputs –Papers, presentations, tables, reports

DDI Alliance – When to capture metadata? Metadata must be captured at the time the event occurs! (not after the facts) Documenting after the facts leads to considerable loss of information This is true for producers and researchers

DDI Alliance – Solutions? Simple solutions: use good practices –File and variable naming conventions, sound statistical methods (metadata in names!) –Comment source code –Document your work Adopt DDI & other standard based metadata solutions: –DDI tools, citation database, source code level metadata capture, variable recodes, table disclosure, data quality feedback, comparability Take advantage of web based collaborative tools –Wiki, blogs, discussion groups, lists

DDI Alliance – Benefits Comprehensive data documentation –Through good metadata practices, comprehensive documentation captured by producers, librarians and users is available to ALL researchers Preservation, integration and sharing of knowledge –Research process is captured and preserved in standard formats –Research knowledge becomes integrant part of the survey and available to all –Reduce duplication of efforts and facilitates reuse –Producer gets feedback from the data users (usage, quality issues), which lead to better and more relevant data Research outputs and dissemination –Facilitate production of research outputs –Facilitate dissemination and fosters broader visibility of research results

DDI Alliance – Conclusions Metadata is a crucial component of social and behavioral science The Data Documentation Initiative (DDI) is a globally accepted specification for capturing microdata documentation and knowledge Latest version 3.0 extends into the entire survey Life Cycle Producers and data archives are rapidly adopting metadata standards. This adoption process should extend into the research community Best practices in data and metadata management benefit all users and have the potential to change the way we conduct research or