DDI Does it have a life beyond IASSIST? IASSIST/IFDO 2005 Edinburgh Edinburgh February 11, 2004 Ernie Boyko NESSTAR Americas Ottawa May, 2005

OutlineOutline u About DDI u Who is using it? u Producer issues u Some Metadata standards v DDI v ISO v SDMX u Analysis and conclusions u About DDI u Who is using it? u Producer issues u Some Metadata standards v DDI v ISO v SDMX u Analysis and conclusions

About DDI u Established in 1995 to create a universally supported metadata standard for the social science community (first meeting in Quebec city at IASSIST 1995) u Initiated and organised by the Inter- University Consortium for Political and Social Research (ICPSR), Michigan, USA u Guided by the DDI Alliance u Established in 1995 to create a universally supported metadata standard for the social science community (first meeting in Quebec city at IASSIST 1995) u Initiated and organised by the Inter- University Consortium for Political and Social Research (ICPSR), Michigan, USA u Guided by the DDI Alliance

Why ? uAn XML structure for a codebook to be: vmanipulated vviewed vsearched, and vemployed by stat packages uInvolves diverse participants: vdata producers varchives/data centres vresearchers/users uAn XML structure for a codebook to be: vmanipulated vviewed vsearched, and vemployed by stat packages uInvolves diverse participants: vdata producers varchives/data centres vresearchers/users

Who is using it? u Data libraries and archives u Some research projects u Some government projects u Some data producers u Builders of dataset collections The dream is to have data producers do the DDI markup at source……but! u Data libraries and archives u Some research projects u Some government projects u Some data producers u Builders of dataset collections The dream is to have data producers do the DDI markup at source……but!

Data Producer Issues u See it as extra work u Have existing systems and are reluctant to change u Complex surveys u Process management u Are often not interested in archiving But more importantly, they are concerned about which standard they should use! u See it as extra work u Have existing systems and are reluctant to change u Complex surveys u Process management u Are often not interested in archiving But more importantly, they are concerned about which standard they should use!

What Other Standards? u u ISO/IEC – Metadata registries u Statistical Data Metadata Exchange u SDMX - Statistical Data Metadata Exchange u u There are others but they do not come on the data producer radar screen u u ISO/IEC – Metadata registries u Statistical Data Metadata Exchange u SDMX - Statistical Data Metadata Exchange u u There are others but they do not come on the data producer radar screen

ISO/IEC – Metadata registries u u A standard for describing any kind of data and organizing the descriptions in a registry (previously, focus was data elements) Dan Gillman US Bureau Labor Statistics u u Being used by NSO’s to describe data as part of their metadata management strategy u u Enables comparison of data from different sources e.g., classification schemes u Freely available at u u A standard for describing any kind of data and organizing the descriptions in a registry (previously, focus was data elements) Dan Gillman US Bureau Labor Statistics u u Being used by NSO’s to describe data as part of their metadata management strategy u u Enables comparison of data from different sources e.g., classification schemes u Freely available at

ISO/IEC – Metadata registries u Not aware of any full implementation of an ISO model u Statistics Canada is close with their IMDB project u A good source for creating codebooks u See index.htm index.htm index.htm u Not aware of any full implementation of an ISO model u Statistics Canada is close with their IMDB project u A good source for creating codebooks u See index.htm index.htm index.htm

SDMXSDMX u To support efficient processes for exchanging data between and among partners u Common vocabularies u International code lists u National Accounts Frameworks u Balance of Payments Manuals u To support efficient processes for exchanging data between and among partners u Common vocabularies u International code lists u National Accounts Frameworks u Balance of Payments Manuals

SDMXSDMX u Bank for International Settlements u European Central Bank u Eurostat u International Monetary Fund u OECD u UN Statistical Office u World bank u Bank for International Settlements u European Central Bank u Eurostat u International Monetary Fund u OECD u UN Statistical Office u World bank

SDMXSDMX u Long history of standards and protocols u Numerous metadata repositories u Strong focus on consistency and data quality u See Deals only with aggregate data with a major emphasis on time series u Long history of standards and protocols u Numerous metadata repositories u Strong focus on consistency and data quality u See Deals only with aggregate data with a major emphasis on time series

DDI Version 3.0 uConceptual and representation uLooks beyond data archiving to the complete survey life cycle uInstrument documentation uTo facilitate more re-use of metadata uComparative analysis uTakes advantage of new XML technologies uConceptual and representation uLooks beyond data archiving to the complete survey life cycle uInstrument documentation uTo facilitate more re-use of metadata uComparative analysis uTakes advantage of new XML technologies

AnalysisAnalysis u Different players with different objectives u Data elements, data sets, aggregates and time series u Administrative components u Consistency and data quality u Different software and data processing considerations u Different implementation rates u Different players with different objectives u Data elements, data sets, aggregates and time series u Administrative components u Consistency and data quality u Different software and data processing considerations u Different implementation rates

ConclusionsConclusions To quote Karsten Boye Rasmussen “…..standards must be important, “…..standards must be important, ” To quote Karsten Boye Rasmussen “…..standards must be important, “…..standards must be important, ”

ConclusionsConclusions To quote Karsten Boye Rasmussen “…..standards must be important, “…..standards must be important, that is why there are so many to choose from!” that is why there are so many to choose from!” To quote Karsten Boye Rasmussen “…..standards must be important, “…..standards must be important, that is why there are so many to choose from!” that is why there are so many to choose from!”

ConclusionsConclusions u There is no gold standard for creating metadata and facilitating data and metadata exchange and preservation u Different objectives require different standards u Organizations will not invest in new tools and standards if they do not provide them with a direct benefit u There is no gold standard for creating metadata and facilitating data and metadata exchange and preservation u Different objectives require different standards u Organizations will not invest in new tools and standards if they do not provide them with a direct benefit

ConclusionsConclusions To answer the question that I posed at the beginning…….. Yes, I think that DDI will have a life well beyond the IASSIST community Yes, I think that DDI will have a life well beyond the IASSIST community To answer the question that I posed at the beginning…….. Yes, I think that DDI will have a life well beyond the IASSIST community Yes, I think that DDI will have a life well beyond the IASSIST community

Thank you for your attention Ernie Boyko, Nesstar Americas Ernie Boyko, Nesstar