Hussein Suleman University of Cape Town Department of Computer Science Digital Libraries Laboratory February 2008 Data Curation Repositories:

Slides:

Advertisements

Similar presentations

IRRA DSpace April 2006 Claire Knowles University of Edinburgh.

Advertisements

Data Publishing Service Indiana University Stacy Kowalczyk April 9, 2010.

Computer Technology Timpview High School. A collection of local, regional, national, and international computer networks that are linked together to exchange.

Lawrence Webley, Hussein Suleman, Tatenda Chipeperekwa University of Cape Town Department of Computer.

National Digital Repository ® Preserving the imperfect: reflections from NDAD and elsewhere Kevin Ashley Head of Digital Archives Group ULCC.

Depositing e-material to The National Library of Sweden.

Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.

Introducing Symposia : “ The digital repository that thinks like a librarian”

Searching and Researching the World Wide: Emphasis on Christian Websites Developed from the book: Searching and Researching on the Internet and World Wide.

PAWN: A Novel Ingestion Workflow Technology for Digital Preservation Mike Smorul, Joseph JaJa, Yang Wang, and Fritz McCall.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Data-PASS Shared Catalog Micah Altman & Jonathan Crabtree 1 Micah Altman Harvard University Archival Director, Henry A. Murray Research Archive Associate.

Introduction to Digital Libraries hussein suleman uct cs honours 2004.

Introduction to the OAI Metadata Harvesting Protocol Hussein Suleman, Digital Library Research Laboratory Virginia Tech.

Ingest and Dissemination with DAITSS Presented by Randy Fischer, Programmer, Florida Center for Library Automation, University of Florida DigCCurr2007.

METS-Based Cataloging Toolkit for Digital Library Management System Dong, Li Tsinghua University Library

Wasim Rangoonwala ID# CS-460 Computer Security “Privacy is the claim of individuals, groups or institutions to determine for themselves when,

How to participate in the Union Catalogue Project Hussein Suleman Sivulile – Open Access South Africa Advanced Information Management.

Dspace 1 Introduction to DSpace Mukesh Pund Scientist NISCAIR, New Delhi.

Introducing the Internet Source: Learning to Use the Internet.

OCLC Online Computer Library Center CONTENTdm ® Digital Collection Management Software Ron Gardner, OCLC Digital Services Consultant ICOLC Meeting April.

Tutorial 1: Getting Started with Adobe Dreamweaver CS4.

Hussein Suleman University of Cape Town Department of Computer Science Advanced Information Management Laboratory High Performance.

XHTML Introductory1 Linking and Publishing Basic Web Pages Chapter 3.

ITIS 1210 Introduction to Web-Based Information Systems Chapter 23 How Web Host Servers Work.

Indo-US Workshop, June23-25, 2003 Building Digital Libraries for Communities using Kepler Framework M. Zubair Old Dominion University.

BEN Architecture Isovera Consulting Feb Internet consulting for non-profits 2 BEN Architecture Diagram.

UNESCO ICTLIP Module 1. Lesson 61 Introduction to Information and Communication Technologies Lesson 6. What is the Internet?

Data Management Practices for Early Career Scientists: Closing Robert Cook Environmental Sciences Division Oak Ridge National Laboratory Oak Ridge, TN.

1 Schema Registries Steven Hughes, Lou Reich, Dan Crichton NASA 21 October 2015.

A centre of expertise in digital information management RDN, e-Prints UK and NOF- Digitise: a (very) small sample of UK OAI activity Andy.

The Resource Discovery Network and OAI Andy Powell UKOLN, University of Bath UKOLN is funded by Resource: The Council.

MTA SZTAKI Department of Distributed Systems The problems of persistent identifiers in the context of the National Digital Data Archives of Hungary András.

Mirroring an OAI archive with an I2-DSI channel Ryan Richardson Edward A. Fox Digital Library Research Laboratory Virginia Tech May 7 th, 2002.

WDC-MARE – World Data Center for Marine Environmental Sciences Data portal based on Open Archives Initiative Protocols and Apache Lucene Uwe Schindler,

CBSOR,Indian Statistical Institute 30th March 07, ISI,Kokata 1 Digital Repository support for Consortium Dr. Devika P. Madalli Documentation Research &

Tsinghua University Library Yang Zhao & Airong Jiang Tsinghua University Library, Beijing China 4 June, 2004 Electronic Thesis and Dissertation System.

Uwe SchindlerGES 2007 – May 2-4, 2007 Data Information Service based on Open Archives Initiative Protocols and Apache Lucene Uwe Schindler 1, Benny Bräuer.

This presentation describes the development and implementation of WSU Research Exchange, a permanent digital repository system that is being, adding WSU.

1 GRID Based Federated Digital Library K. Maly, M. Zubair, V. Chilukamarri, and P. Kothari Department of Computer Science Old Dominion University February,

A radiologist analyzes an X-ray image, and writes his observations on papers  Image Tagging improves the quality, consistency.  Usefulness of the data.

OAI Overview DLESE OAI Workshop April 29-30, 2002 John Weatherley

The Danish National Research Database Which approach: Look at the environment first and the software afterwards or vice versa? Look at the environment.

Chapter 29 World Wide Web & Browsing World Wide Web (WWW) is a distributed hypermedia (hypertext & graphics) on-line repository of information that users.

Mercury – A Service Oriented Web-based system for finding and retrieving Biogeochemical, Ecological and other land- based data National Aeronautics and.

Enforcing Interoperability with the Open Archives Initiative Repository Explorer Hussein Suleman, Digital Library Research.

CONTENTS  Definition And History  Basic services of INTERNET  The World Wide Web (W.W.W.)  WWW browsers  INTERNET search engines  Uses of INTERNET.

Metadata and OAI DLESE OAI Workshop April 29-30, 2002 Katy Ginger Presentation available at:

WEB SERVER SOFTWARE FEATURE SETS

U.S. Environmental Protection Agency Central Data Exchange Pilot Project Promoting Geospatial Data Exchange Between EPA and State Partners. April 25, 2007.

DSpace System Architecture 11 July 2002 DSpace System Architecture.

Sharing Digital Scores: Will the Open Archives Initiative Protocol for Metadata Harvesting Provide the Key? Constance Mayer, Harvard University Peter Munstedt,

Digital Library Syllabus Uploader Will Cameron CSC 8530 Fall 2006 Presentation 1.

AHM04: Sep 2004 Nottingham CCLRC e-Science Centre eMinerals: Environment from the Molecular Level Managing simulation data Lisa Blanshard e- Science Data.

A Project of the University Libraries Ball State University Libraries A destination for research, learning, and friends.

Metadata-based Discovery: Experience in Crystallography UKOLN is supported by: Monica Duke UKOLN, University of Bath, UK A centre of.

Institutional Repositories and Licensing of Research Output advanced information management laboratory university of cape town department of computer science.

A s s i g n m e n t W e e k 7 : T h e I n t e r n e t B Y : P a t r i c k O b i s p o.

Discover ScholarSphere A repository service collaboration between the University Libraries and ITS.

Alessandro Yoshi Polliotti 1 / 13 TERENA Networking Conference 2005 Biblioteca d'Alessandria: A Peer-to-peer Network for Scholar Knowledge Exchange Terena.

Ingest and Dissemination with DAITSS

VI-SEEM Data Repository

Introduction to DSpace

SCALABLE OPEN ACCESS Hussein Suleman

eCulture Science Gateway – reloaded

Enforcing Interoperability with the Open Archives Initiative Repository Explorer Hussein Suleman, Digital Library Research Laboratory Virginia.

Institutional Repositories

Robin Dale RLG OAIS Functionality Robin Dale RLG

The Fedora Project April 28-29, 2003 CNI, Washington DC

The Internet and Electronic mail

Presentation transcript:

Hussein Suleman University of Cape Town Department of Computer Science Digital Libraries Laboratory February 2008 Data Curation Repositories: From past to present

The W3C-WCA Repository (~1999) ‏

What were we storing?  [10/Feb/2008:14:59: ] "GET /localdocs/openldap/rfc/rfc3712.txt HTTP/1.1" "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +  [10/Feb/2008:15:01: ] "GET /localdocs/Sablot/jsdom-ref/apidocs/api-Node.html HTTP/1.0" "-" "Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +  [10/Feb/2008:15:02: ] "GET /localdocs/openldap/rfc/rfc2294.txt HTTP/1.1" "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +  [10/Feb/2008:15:04: ] "GET / HTTP/1.1" "-" "Baiduspider+(+  [10/Feb/2008:15:04: ] "GET / HTTP/1.1" "-" "Baiduspider+(+  [10/Feb/2008:15:04: ] "GET /localdocs/openldap/rfc/rfc2255.txt HTTP/1.1" "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +  [10/Feb/2008:15:07: ] "GET /localdocs/openldap/rfc/rfc2253.txt HTTP/1.1" "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +  [10/Feb/2008:15:08: ] "GET /workshops/lesotho HTTP/1.1" "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +  [10/Feb/2008:15:10: ] "GET /localdocs/openldap/rfc/rfc3296.txt HTTP/1.1" "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +  [10/Feb/2008:15:14: ] "GET /localdocs/libwmf/html/classes.html HTTP/1.1" "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +

Open Access  All documents and data (mostly log files from servers) were open to the world.  This was the first OAI-compliant archive! The repository was used as a testbed when the OAI protocol was developed.  Data collections were contributed by: AOL, Altavista, Boeing, etc.

Features  Search and Browse interfaces.  Users could submit documents.  An editor always checked submissions.  Data was carefully validated!

Validation  Format of log files was defined using XML descriptions (long before XML schema).  Every dataset was tested against its formal definition using a special high-performance validation program. Data items failing validation were excluded from the repository! Datasets were certified (or not). Certification and error report were added to metadata.

Scalability  All datasets were stored on FTP servers. FTP was (is?) more efficient than HTTP for large data transfers.  Repository only handled metadata through Web UI.  Datasets were very large and often distributed. Magnetic tapes were sent by courier to exchange data among sites. A tape robot storage system was used on the central server. Subsets of data were kept online if complete datasets were too large.

Linking  Repository initially only contained datasets, but expanded to include papers and tools.  Metadata records for datasets and documents were stored in the same database.  Search/browse operated over all metadata.  Datasets were linked to documents in the metadata.

Legalities  Datasets were contributed by their owners.  All sensitive data was first removed (by the submitters or archive managers).  Tools were developed to anonymise datasets (e.g., removing IP addresses).  The tools were themselves added to the repository!

So what has been achieved …  Data repository  Quality control  Anonymity  Scalability  Hyperlinks

Where to now?  Roll out similar solutions using repository software?  Use institutional repositories where data repositories do not exist?  Get information specialists to talk to researchers and vice versa?