An Overview of Data-PASS Shared Catalog Micah Altman, Harvard University

Contents Collaboration components What the Shared Catalog Does How it Works Syndicated Storage Integration

Collaboration Components Partnership Agreement Agreement to establish good practice Preservation copies of data collected Transfer Protocol: support in case of archival failure Operations Central database of leads for acquisition Development of shared procedures Review of acquisitions Documentation of procedure Identification & selection Metadata Security Confidentiality Catalog Discovery Layered Services An Overview of Data-PASS Shared Catalog

Search Across Entire Partners’ Catalogs Finding Data Search Across Entire Partners’ Catalogs Find Studies Collected for Datapass Simple and Fielded Search Browse by Subject, Date, Source An Overview of Data-PASS Shared Catalog

What does it look Like? An Overview of Data-PASS Shared Catalog

Viewing Study Information Study Information: Author, Title, Abstract, Citation,… Permanent Citation (optional) Provenance: Data Author, Producer, Distributor Chain of Responsibility for Metadata Files: Link to that study at partner site List of files (optional) Extended cataloging information Full Catalog Record DDI Record Variable-level information, descriptive stats (optional) An Overview of Data-PASS Shared Catalog

Delivering Data Through Partners’ Sites Through Shared Catalog Shared catalog results always give link to data at partners site If no file information supplied to catalog, this is the only option Through Shared Catalog Catalog server may cache a copy of data for performance Catalog can bundle requests for multiple files Through Analysis Services If partner site runs VDC (or data access proxy), analysis and extraction is available Download data in multiple formats Extract subsets, in multiple formats, with citations and UNF’s Run descriptive stats, crosstabs Advanced analysis -- dozens of statistical models An Overview of Data-PASS Shared Catalog

Current Participation Studies Listed Files Listed Files Available Analysis Available HMDC/ Murray Archives All From Catalog Yes ICPSR From Archive If hosting DataVerse software locally NARA Planned Planned: Selected files made available through The DataWeb ROPER ODUM An Overview of Data-PASS Shared Catalog

Advanced Analysis DataVerse network and Zelig make it easy to provide access to any statistical model available in the R statistical language Specification-driven architecture: describe the model to run the model Currently provides configuration for >25 models: regression, limited dependent variables, factor analysis, event counts, duration models An Overview of Data-PASS Shared Catalog

Supporting Technologies Metadata: OAI+DDI+XSL Citation, validation: Handle+UNF Workflow, repository, analysis: Dataverse Network (VDC) An Overview of Data-PASS Shared Catalog

Each partner catalog is exposed via Metadata Harvesting Each partner catalog is exposed via Dataverse Network (VDC) via OAI Other OAI Server, running on-site Proxy OAI Server, running at HMDC Harvested ad-hoc XSL Metadata to cross-walk applied Made available through OAI DDI-lite schema subset used for exchange Data Documentation Initiative (DDI) – international effort to establish specification schema for the content, presentation, transport, and preservation of documentation for datasets in the social and behavioral sciences Provenance, and structural metadata, including: document description (meta-meta data), study description, file description, variable description http://www.icpsr.org/DDI/ An Overview of Data-PASS Shared Catalog

UNF – Universal Numeric Fingerprints Same UNF regardless of hardware, operating system, statistical software, database, or spreadsheet software. UNF’s combine: generalized rounding (dessication), normalization (canonicalization), fingerprinting (cryptographic hash, e.g. SHA256) Available as: C++, R-stat language, Stata, SAS, S-Plus See: http://cran.r-project.org/src/contrib/Descriptions/UNF.html An Overview of Data-PASS Shared Catalog

Technologies Dataverse Network (<http://thedata.org>) Includes integrated developments in web application software, networking, data citation standards, and statistical methods designed to put some of the universe of data and data sharing practices on firmer ground. It facilitates the public preservation and distribution of persistent, authorized, and verifiable research data, with powerful but easy-to-use technology, whether or not the data are in the public domain. The project increases scholarly recognition (including formal scholarly citations to articles and to data sets) and distributed control for authors, journals, and others who make data available, facilitates data access and analysis by the scholarly community, and still enables professional archives to seemlessly provide extensive preservation and other services. Shared Catalog Support Provides ingest and dissemination framework and web-based GUI Provides OAI server Provides Data Services Broker, which supports identification, format conversion An Overview of Data-PASS Shared Catalog

An Overview of Data-PASS Shared Catalog http://dvn.iq.harvard.edu/dvn

Architectural Overview Data Mirror Metadata Catalog Harvester Online Catalog Online Analysis View Information on Data Through Catalog Link to Data at Partner Site Access Data With Extraction and Analysis, Through Catalog Direct to Partner Sites <XSL> Crosswalk proxy Search Shared Catalog OAI An Overview of Data-PASS Shared Catalog

Metadata described in detail on Data-PASS Site Metadata Standards Study level: title, author, abstract, id, usage info, … [Required] Files: What is it? -- Description, URI [Required for backups] Is it valid? UNF (universal numeric fingerprint), MD5, … [Improves backup reliability] Variables: Description, ID, Location, … [Enables on-line analysis] Metadata described in detail on Data-PASS Site An Overview of Data-PASS Shared Catalog

Distributed Preservation - Prototype Study Level Include usage metadata Include identifying tags Include Scanned Usage Agreement as File Files Provide URI’s in metadata Allow access by catalog harvester For more reliability, include MD5 or UNF’s Copies Current version of tagged studies will be mirrored at HMDC Resources may also be cached for speed An Overview of Data-PASS Shared Catalog

Distributed Backup Research – Potential Research Schemas to express inter-archival preservation commitments Asymmetric mirroring to match distribution of holdings across partners Preservation of versioned resources Syndicated Storage Technology Integration LOCKSS SRB/IRODS Distributed Data Manager An Overview of Data-PASS Shared Catalog

More Information Shared Catalog: http://vdc.hmdc.harvard.edu/dataverse/DATAPASS/ Dataverse Network Software: http://TheData.Org Data Citations, UNF’s: http://www.dlib.org/dlib/march07/altman/03altman.html Metadata and Other Partnership Documentation http://www.icpsr.umich.edu/DATAPASS/about.html An Overview of Data-PASS Shared Catalog