1. DDI Does it have a life beyond IASSIST? IASSIST/IFDO 2005 Edinburgh Edinburgh February 11, 2004 Ernie Boyko NESSTAR Americas Ottawa May, 2005

2. OutlineOutline u About DDI u Who is using it? u Producer issues u Some Metadata standards v DDI v ISO 11179 v SDMX u Analysis and conclusions u About DDI u Who is using it? u Producer issues u Some Metadata standards v DDI v ISO 11179 v SDMX u Analysis and conclusions

3. About DDI u Established in 1995 to create a universally supported metadata standard for the social science community (first meeting in Quebec city at IASSIST 1995) u Initiated and organised by the Inter- University Consortium for Political and Social Research (ICPSR), Michigan, USA u Guided by the DDI Alliance u Established in 1995 to create a universally supported metadata standard for the social science community (first meeting in Quebec city at IASSIST 1995) u Initiated and organised by the Inter- University Consortium for Political and Social Research (ICPSR), Michigan, USA u Guided by the DDI Alliance

4. Why ? uAn XML structure for a codebook to be: vmanipulated vviewed vsearched, and vemployed by stat packages uInvolves diverse participants: vdata producers varchives/data centres vresearchers/users uAn XML structure for a codebook to be: vmanipulated vviewed vsearched, and vemployed by stat packages uInvolves diverse participants: vdata producers varchives/data centres vresearchers/users

5. Who is using it? u Data libraries and archives u Some research projects u Some government projects u Some data producers u Builders of dataset collections The dream is to have data producers do the DDI markup at source……but! u Data libraries and archives u Some research projects u Some government projects u Some data producers u Builders of dataset collections The dream is to have data producers do the DDI markup at source……but!

6. Data Producer Issues u See it as extra work u Have existing systems and are reluctant to change u Complex surveys u Process management u Are often not interested in archiving But more importantly, they are concerned about which standard they should use! u See it as extra work u Have existing systems and are reluctant to change u Complex surveys u Process management u Are often not interested in archiving But more importantly, they are concerned about which standard they should use!

7. What Other Standards? u u ISO/IEC 11179 – Metadata registries u Statistical Data Metadata Exchange u SDMX - Statistical Data Metadata Exchange u u There are others but they do not come on the data producer radar screen u u ISO/IEC 11179 – Metadata registries u Statistical Data Metadata Exchange u SDMX - Statistical Data Metadata Exchange u u There are others but they do not come on the data producer radar screen

8. ISO/IEC 11179 – Metadata registries u u A standard for describing any kind of data and organizing the descriptions in a registry (previously, focus was data elements) Dan Gillman US Bureau Labor Statistics u u Being used by NSO’s to describe data as part of their metadata management strategy u u Enables comparison of data from different sources e.g., classification schemes u Freely available at http://www.jtc1.org http://www.jtc1.org u u A standard for describing any kind of data and organizing the descriptions in a registry (previously, focus was data elements) Dan Gillman US Bureau Labor Statistics u u Being used by NSO’s to describe data as part of their metadata management strategy u u Enables comparison of data from different sources e.g., classification schemes u Freely available at http://www.jtc1.org http://www.jtc1.org

9. ISO/IEC 11179 – Metadata registries u Not aware of any full implementation of an ISO 11179 model u Statistics Canada is close with their IMDB project u A good source for creating codebooks u See http://www.statcan.ca/english/concepts/ index.htm http://www.statcan.ca/english/concepts/ index.htm http://www.statcan.ca/english/concepts/ index.htm u Not aware of any full implementation of an ISO 11179 model u Statistics Canada is close with their IMDB project u A good source for creating codebooks u See http://www.statcan.ca/english/concepts/ index.htm http://www.statcan.ca/english/concepts/ index.htm http://www.statcan.ca/english/concepts/ index.htm

11. SDMXSDMX u To support efficient processes for exchanging data between and among partners u Common vocabularies u International code lists u National Accounts Frameworks u Balance of Payments Manuals u To support efficient processes for exchanging data between and among partners u Common vocabularies u International code lists u National Accounts Frameworks u Balance of Payments Manuals

12. SDMXSDMX u Bank for International Settlements u European Central Bank u Eurostat u International Monetary Fund u OECD u UN Statistical Office u World bank u Bank for International Settlements u European Central Bank u Eurostat u International Monetary Fund u OECD u UN Statistical Office u World bank

13. SDMXSDMX u Long history of standards and protocols u Numerous metadata repositories u Strong focus on consistency and data quality u See http://www.sdmx.org http://www.sdmx.org Deals only with aggregate data with a major emphasis on time series u Long history of standards and protocols u Numerous metadata repositories u Strong focus on consistency and data quality u See http://www.sdmx.org http://www.sdmx.org Deals only with aggregate data with a major emphasis on time series

14. DDI Version 3.0 uConceptual and representation uLooks beyond data archiving to the complete survey life cycle uInstrument documentation uTo facilitate more re-use of metadata uComparative analysis uTakes advantage of new XML technologies uConceptual and representation uLooks beyond data archiving to the complete survey life cycle uInstrument documentation uTo facilitate more re-use of metadata uComparative analysis uTakes advantage of new XML technologies

15. AnalysisAnalysis u Different players with different objectives u Data elements, data sets, aggregates and time series u Administrative components u Consistency and data quality u Different software and data processing considerations u Different implementation rates u Different players with different objectives u Data elements, data sets, aggregates and time series u Administrative components u Consistency and data quality u Different software and data processing considerations u Different implementation rates

16. ConclusionsConclusions To quote Karsten Boye Rasmussen “…..standards must be important, “…..standards must be important, ” To quote Karsten Boye Rasmussen “…..standards must be important, “…..standards must be important, ”

17. ConclusionsConclusions To quote Karsten Boye Rasmussen “…..standards must be important, “…..standards must be important, that is why there are so many to choose from!” that is why there are so many to choose from!” To quote Karsten Boye Rasmussen “…..standards must be important, “…..standards must be important, that is why there are so many to choose from!” that is why there are so many to choose from!”

18. ConclusionsConclusions u There is no gold standard for creating metadata and facilitating data and metadata exchange and preservation u Different objectives require different standards u Organizations will not invest in new tools and standards if they do not provide them with a direct benefit u There is no gold standard for creating metadata and facilitating data and metadata exchange and preservation u Different objectives require different standards u Organizations will not invest in new tools and standards if they do not provide them with a direct benefit

19. ConclusionsConclusions To answer the question that I posed at the beginning…….. Yes, I think that DDI will have a life well beyond the IASSIST community Yes, I think that DDI will have a life well beyond the IASSIST community To answer the question that I posed at the beginning…….. Yes, I think that DDI will have a life well beyond the IASSIST community Yes, I think that DDI will have a life well beyond the IASSIST community

20. Thank you for your attention Ernie Boyko, Nesstar Americas Ernie Boyko, Nesstar Americasboykern@yahoo.com boykern@yahoo.com