DDI 101 Presented to the : Ontario DLI training session Queens Kingston, Ontario Presented to the : Ontario DLI training session Queens Kingston, Ontario February 11, 2004 Carol Perry And Ernie Boyko April 2004

OutlineOutline uWhat is this all about? uWhy is it important? uMetadata and XML uWhat is DDI? u : A Metadata Framework uSTC Plans for DDI uWhere to from here? uWhat is this all about? uWhy is it important? uMetadata and XML uWhat is DDI? u : A Metadata Framework uSTC Plans for DDI uWhere to from here?

What is this all about? uData Documentation Initiative (DDI) uAnother flavour of information management uNot unlike cataloguing information uThink AACR2/MARC or Dublin Core uBut taking into the needs of data uAnd taking advantage of new technology uData Documentation Initiative (DDI) uAnother flavour of information management uNot unlike cataloguing information uThink AACR2/MARC or Dublin Core uBut taking into the needs of data uAnd taking advantage of new technology

Why Metadata (1) Unlabeled stuffLabeled stuff The bean example is taken from: A Managers Introduction to Adobe eXtensible Metadata Platform,

Why Metadata (2) Finding UnderstandingAssessing Sharing

Converting data to knowledge True data Liberation! Data Brainware + Knowledge = Software

Metadata and XML uA markup language for documents containing structured information uProvides a facility to define tags and the structural relationships between them uCreated so that richly structured documents could be used over the Web uHas become the de-facto exchange format on the Web uProvides the syntax to describe a metadata framework, like uProvides the syntax to describe a metadata framework, like uA markup language for documents containing structured information uProvides a facility to define tags and the structural relationships between them uCreated so that richly structured documents could be used over the Web uHas become the de-facto exchange format on the Web uProvides the syntax to describe a metadata framework, like uProvides the syntax to describe a metadata framework, like

What is ? uThe Data Documentation Initiative ( ) is an international effort to establish a standard for technical documentation describing social science data uIt is guided by a membership-based alliance that is developing/evolving the specification which is written in XML See See uThe Data Documentation Initiative ( ) is an international effort to establish a standard for technical documentation describing social science data uIt is guided by a membership-based alliance that is developing/evolving the specification which is written in XML See See

What is ? (contd) uAn XML structure for a codebook to be: vmanipulated vviewed vsearched, and vemployed by stat packages uInvolves diverse participants: vdata producers varchives/data centres vresearchers/users uAn XML structure for a codebook to be: vmanipulated vviewed vsearched, and vemployed by stat packages uInvolves diverse participants: vdata producers varchives/data centres vresearchers/users

Brief history of Brief history of u Established in 1995 to create a universally supported metadata standard for the social science community u Initiated and organised by the Inter- University Consortium for Political and Social Research (ICPSR), Michigan, USA u Members coming from social science data archives and libraries in USA, Canada and Europe and from major producers of statistical data u First version of the standard expressed as an SGML-DTD u Established in 1995 to create a universally supported metadata standard for the social science community u Initiated and organised by the Inter- University Consortium for Political and Social Research (ICPSR), Michigan, USA u Members coming from social science data archives and libraries in USA, Canada and Europe and from major producers of statistical data u First version of the standard expressed as an SGML-DTD

Brief history of (contd) u Translated to XML in 1997 u Extensive testing carried out Spring- Summer 1999 u DDI 1.0 published Spring 2000 u DDI 1.1 with minor revisions and some additions published Autumn 2001 u The DDI 2.0 published Summer 2003, including aggregate data, geographic elements, element formatting u Translated to XML in 1997 u Extensive testing carried out Spring- Summer 1999 u DDI 1.0 published Spring 2000 u DDI 1.1 with minor revisions and some additions published Autumn 2001 u The DDI 2.0 published Summer 2003, including aggregate data, geographic elements, element formatting

Importance to Data Producers uProvides guidelines for documenting research uIncreases usefulness of the collection due to standardization, increasing the potential for greater use by analysts uProvides consistent field mappings, facilitating import into statistical software uEnables reuse of survey components uProvides guidelines for documenting research uIncreases usefulness of the collection due to standardization, increasing the potential for greater use by analysts uProvides consistent field mappings, facilitating import into statistical software uEnables reuse of survey components

Importance of the DDI: To Archivists u Metadata supplied in complete form u Facilitates distribution of data collections: codebook already readily usable, and data definition statements can be generated easily u Facilitates online analysis and subsetting u Archival format u Metadata supplied in complete form u Facilitates distribution of data collections: codebook already readily usable, and data definition statements can be generated easily u Facilitates online analysis and subsetting u Archival format

Importance to Users uImproves searching by individual field and across collections uMakes available well-documented data collections more quickly uPotentially provides more information through extensive linking features uImproves searching by individual field and across collections uMakes available well-documented data collections more quickly uPotentially provides more information through extensive linking features

Projects Using DDI u NESSTAR u Health Canada -- DAIS u SDA, Berkeley u University of Alberta u University of Guelph u University of Toronto u ICPSRs metadata u University of Minnesota u US Census Bureau u Harvard Virtual Data Center?? u NESSTAR u Health Canada -- DAIS u SDA, Berkeley u University of Alberta u University of Guelph u University of Toronto u ICPSRs metadata u University of Minnesota u US Census Bureau u Harvard Virtual Data Center??

What are STCs Plans for DDI uSTC has purchased some NESSTAR licences uPlan to use NESSTAR Publisher to produce standardized metadata for master and public files uUse NESSTAR Server to provide access across master files to support Statistics Canada analysis uDisseminate compliant survey files/documentation to RDCs and DLI sites uSTC has purchased some NESSTAR licences uPlan to use NESSTAR Publisher to produce standardized metadata for master and public files uUse NESSTAR Server to provide access across master files to support Statistics Canada analysis uDisseminate compliant survey files/documentation to RDCs and DLI sites

uProvide controlled access to public use files uOnline tool for facilitating remote access using synthetic files uIntroduce students to microdata uArchival tool for master and public files uDevelop a two-way crosswalk other data extractors and metadata bases. uProvide controlled access to public use files uOnline tool for facilitating remote access using synthetic files uIntroduce students to microdata uArchival tool for master and public files uDevelop a two-way crosswalk other data extractors and metadata bases. What could be done with DDI/NESSTAR?

Whats next? uLets build on the work that is already starting in Canada uBut first, Carol will give you an overview of some of the how tos uLets build on the work that is already starting in Canada uBut first, Carol will give you an overview of some of the how tos