1. CWSpace Archiving Courseware Websites to DSpace Using a Content Packaging Profile & Web Services © Digital Library Federation Spring Forum, 2006 April 11, 2006 — Austin, Texas William Reilly, Rob Wolfe — MIT Libraries 2006

2. DLF Spring 2006 MIT's CWSpace 2 Project Overview Vision: Increase value in Institute publication: OCW –Also: Increase value in Open Source platform: DSpace * CLE = Collaborative Learning Environment Goal: Archive OCW to MIT’s DSpace –Also: Make content available to CLEs * CWSpace: MIT iCampus project, nearing completion Method: Interoperability of systems –Use Content Packaging standard –Use Web Services standard

3. DLF Spring 2006 MIT's CWSpace 3 Perceived Need: inter-operability ™ Obvious observation: Higher Ed. Technologies increasingly seeking integration with Library & Repository holdings Not so obvious(?): Libraries seeing their charter to include collection, preservation, dissemination of Teaching & Learning materials (?)

4. DLF Spring 2006 MIT's CWSpace 4 InterOp: What’s Available

5. DLF Spring 2006 MIT's CWSpace 5 InterOp: What’s Achievable 1 of 3

6. DLF Spring 2006 MIT's CWSpace 6 InterOp: Challenges 2 of 3

7. DLF Spring 2006 MIT's CWSpace 7 InterOp: Possible Resolution? 3 of 3

8. DLF Spring 2006 MIT's CWSpace 8 OCW Materials in DSpace = New Ground Educational MaterialsDigital Archive New kind of content: educational, teaching, learning POLICY - Cost/merit evaluation of archival treatment. Preservation and IP considerations New emphasis (ideally) on re-use, re-purposing: aggregation, disaggregation GRANULARITY - Challenge to digital archive; how flexible can it be? (cf. CMS) New kind of structure, composition: website, compound digital object, rich metadata COMPLEXITY - Need to package multi-file content, plus metadata Content Packaging New needs for system-to-system access: CLEs, Image Tools BACK-END / FRONT-END - Need for networked access Web Services

9. DLF Spring 2006 MIT's CWSpace 9 Content Packaging

10. DLF Spring 2006 MIT's CWSpace 10 Content Packaging: Steps Toward a Profile Model OCW Content Choose Packaging Specification Profile That Specification

11. DLF Spring 2006 MIT's CWSpace 11 Content Package Package Interchange File (PIF) Usually a.zip file Contains a manifest Manifest references files (some in, some out of the package)

12. DLF Spring 2006 MIT's CWSpace 12 Object Models Mapping one world to another Content Package design considerations Use, re-use, subsequent dissemination OCWDSpace SIP DIP … …

13. DLF Spring 2006 MIT's CWSpace 13 OCW to DSpace mapping

14. DLF Spring 2006 MIT's CWSpace 14 Which Packaging Spec.? Content packaging specifications are generic loose envelopes to carry content, metadata Choice of package is largely determined by user community Dspace to use METS internally Higher Education uses IMS Content Packaging METS IMS-CP MPEG-21 DIDL XFDU

15. DLF Spring 2006 MIT's CWSpace 15 METS and IMS-CP

16. DLF Spring 2006 MIT's CWSpace 16 IMS-CP & OCW Content Model

17. DLF Spring 2006 MIT's CWSpace 17 IMS-CP & DSpace Object Model

18. DLF Spring 2006 MIT's CWSpace 18 OCW to IMS-CP to DSpace

19. DLF Spring 2006 MIT's CWSpace 19 Profiling Package Standards Every Implementation of IMS-CP or METS will: Constrain the Standard Extend the Standard Interpret the Standard The sum of these adjustments is the implementation profile The Standards themselves encourage profiling in a systematic way: IMS Application Profile Guidelines Technical Manual http://imsglobal.org/ap/apv1p0/i msap_techv1p0.html METS Profile Schema and Guidelines http://www.loc.gov/standards/mets/mets- profiles.html

20. DLF Spring 2006 MIT's CWSpace 20 Application Profiles

21. DLF Spring 2006 MIT's CWSpace 21 Application Profiles

22. DLF Spring 2006 MIT's CWSpace 22 Application Profiles

23. DLF Spring 2006 MIT's CWSpace 23 Application Profiles

24. DLF Spring 2006 MIT's CWSpace 24 Application Profiles

25. DLF Spring 2006 MIT's CWSpace 25 Profiled Content Package… Now with Package defined… …need to move that across the network from producer to consumer… DSpace Content Provider

26. DLF Spring 2006 MIT's CWSpace 26 Web Services

27. DLF Spring 2006 MIT's CWSpace 27 DSpace dev: LNI & PKGers

28. DLF Spring 2006 MIT's CWSpace 28 Lightweight Network Interface 0. What Was The Problem? 1.What Is the LNI? 2.DSpace & WebDAV 3.Example of Use: Submit OCW 4.Future Development…

29. DLF Spring 2006 MIT's CWSpace 29 What was the Problem…? Content Providers wanted… Networked Access: Programmatically access the DSpace object model, but not running on the DSpace server Basic (Lightweight Network Interface (LNI)) [ Access, Deposit, Limited Mods. ] Submit Retrieve Item w. Metadata Build own U/I Collection Mgt, Map Items Advanced (possible future LNI dev.) [ Replace, Withdraw, Delete ] Specialist metadata Automated metadata Quality control checks Format migration … DSpace Content Provider

30. DLF Spring 2006 MIT's CWSpace 30 What Is the LNI? The "Lightweight Network Interface" is: A remote API to control DSpace It's essentially WebDAV: Extensions to HTTP –A SOAP (with WSDL) equivalent is provided It's a mapping: — –a "complete and comprehensive" view of the DSpace public object API… (the "business logic" layer) –…as mapped to WebDAV semantics

31. DLF Spring 2006 MIT's CWSpace 31 DSpace: Model DSpace digital repository Differs from file system Submit URI not known Workflow …

32. DLF Spring 2006 MIT's CWSpace 32 WebDAV: Model, Basics Each Resource named by a URL Properties are "name value" pairs Properties are extensible Property values can be found, and modified DAV = "Distributed Authoring and Versioning" URL

33. DLF Spring 2006 MIT's CWSpace 33 WebDAV: Model, Fuller Resources can be Collections or leaf nodes Collections can nest Everything can be discovered descending from the DAV server's root '/' resource

34. DLF Spring 2006 MIT's CWSpace 34 DSpace Mapped to WebDAV WebDAV

35. DLF Spring 2006 MIT's CWSpace 35 DSpace Implemented as WebDAV 1.Properties –WebDAV –Custom for DSpace 2.LNI URLs 3.WebDAV Methods

36. DLF Spring 2006 MIT's CWSpace 36 Properties: WebDAV DAV:creationdate DAV:displayname DAV:getcontentlength DAV:getcontenttype DAV:getlastmodified DAV:resourcetype … http://www.webdav.org/specs/rfc2518.html Akin to file system properties, with add'l. metadata

37. DLF Spring 2006 MIT's CWSpace 37 Properties: Custom DSpace (1 of 3) ALL DSpace Objects: –dspace:type "BITSTREAM", "ITEM", "COMMUNITY" etc. –DAV:resourcetype "COLLECTION" etc. –dspace:displayname –dspace:current-user-privilege-set

38. DLF Spring 2006 MIT's CWSpace 38 Properties: Custom DSpace (2 of 3) Community DSpace Objects –dspace:logo –dspace:short_description –dspace:side_bar_text –dspace:handle –…

39. DLF Spring 2006 MIT's CWSpace 39 Properties: Custom DSpace (3 of 3) Bitstream DSpace Objects –DAV:getcontentlength = getSize() –dspace:format = getFormat().getID() –dspace:sequence_id = getSequenceID() –… Many more DSpace Objects (some 10) –With many more Properties (some 87) http://wiki.dspace.org/LightweightNetworkInterface

40. DLF Spring 2006 MIT's CWSpace 40 URLs: DSpace LNI Community –http://uni.edu/dspace/dav/dso_1721.1$46 Collection –http://uni.edu/dspace/dav/dso_1721.1$349 Item –http://uni.edu/dspace/dav/dso_1721.1$5543 Bitstream –http://uni.edu/dspace/dav/dso_1721.1$5543/bitstream_13 The LNI URLs for DSpace objects are only for use with the LNI. They are therefore not to be used: as persistent URLs (e.g. bookmark, store for later use) as interactive URLs (e.g. browser)

41. DLF Spring 2006 MIT's CWSpace 41 WebDAV Methods in LNI PROPFIND - very powerful request with multiple functions: –list names of properties –return property values –recurse through 1 or more levels of a "collection" resource. PROPPATCH - changes value of or deletes properties on one resource. GET - retrieves contents of a resource PUT - replace or add a new resource to a collection COPY - map a resource ("Item") to a different Collection. As noted, equivalent SOAP methods are provided Not Implemented: Locking, Versioning.

42. DLF Spring 2006 MIT's CWSpace 42 Current DSpace (pre-LNI) Content Provider, currently Batch import (& export) End user Web U/I dspace-admin Web U/I SRW, OAI-PMH DSpace Batch Importer OCW Comm. "Biology (7)" "Physics (8)" DSpace Object API Web U/I Web, but Non Auto- mated Non Std. Non Web Workflow Adm Item

43. DLF Spring 2006 MIT's CWSpace 43 DSpace with LNI Content Provider using LNI: LNI SOAP end-point URL LNI WebDAV URLs dspace-admin (partial) SRW, OAI-PMH –(Not replicated in LNI) Still available: –Batch import (& export) –End user Web U/I DSpace DSpaceLNI.class OCW Comm. "Biology (7)" DSpace Object API DSpace LNI WebDAV - Web. - Stds. - Batch- able. "Physics (8)" Item

44. DLF Spring 2006 MIT's CWSpace 44 Submission: Overview OpenCourseWare Client Needs 3 Values: 1.LNI SOAP end-point URL http://userid:password@dspace.mit. edu/dspace-lni/lni/DSpaceLNI 2.Handle for OCW Community = hdl:1721.1/125 3.Dept. Name (MIT "Course" Number) for Course to be submitted = "Physics (8)" DSpace DSpaceLNI.class OCW Comm. 1721.1/125 "Biology (7)" DSpace Object API DSpace LNI WebDAV Lang. - Indep. Platform - Indep. "Physics (8)" Item

45. DLF Spring 2006 MIT's CWSpace 45 Submission: 1. Lookup Request: OCW Comm. HANDLE LNI lookup() = http://dspace.mit.edu/dspace- lni/dav/lookup/handle/1721.1/125 Response: OCW Comm. LNI URL http://dspace.mit.edu/dav/dso_172 1.1$125 DSpace DSpaceLNI.class OCW Comm. 1721.1/125 dso_1721.1$125 "Biology (7)" dso_1721.1$197 "Physics (8)" dso_1721.1$253 DSpace Object API DSpace LNI WebDAV Lookup

46. DLF Spring 2006 MIT's CWSpace 46 Submission: 2. Collections Request: PROPFIND on LNI URL for children DSpace Collections' "displaynames": Response: MULTISTATUS Biology (7) Physics (8)... DSpace DSpaceLNI.class OCW Comm. 1721.1/125 dso_1721.1$125 "Biology (7)" dso_1721.1$197 "Physics (8)" dso_1721.1$253 DSpace Object API DSpace LNI WebDAV Iterate…

47. DLF Spring 2006 MIT's CWSpace 47 Submission: 3. Submit (PUT) Request: PUT to DSpace Collection LNI URL, including specification of Packager Type (IMS- CP): PUT /dspace/dav/dso_1721.1$253?package=OCW- IMSCP....package contents in body... Response: LNI URL for DSpace Item in "Location" Header: HTTP/1.1 201 OK Location: http://dspace.mit.edu/dspace/dav/dso_1721.1$8 36....other headers.... DSpace DSpaceLNI.class OCW Comm. 1721.1/125 dso_1721.1$125 "Physics (8)" dso_1721.1$253 DSpace Object API DSpace LNI WebDAV Ingest Item dso_1721.1$836

48. DLF Spring 2006 MIT's CWSpace 48 Submission: 4. Item Handle Request: PROPFIND on DSpace Item LNI URL for the DSpace Handle: Response: Properties Requested === PROPERTIES Successfully returned: dspace:handle = " hdl:1721.1/836 " DSpace DSpaceLNI.class OCW Comm. 1721.1/125 dso_1721.1$125 "Physics (8)" dso_1721.1$253 DSpace Object API DSpace LNI WebDAV Reverse Lookup Item dso_1721.1$836 1721.1/836 OCW Content Mgt. System Stores DSpace Handle SQL Server Table: hdl:1721.1/836

49. DLF Spring 2006 MIT's CWSpace 49 Submission: Summary In this 4-step OpenCourseWare submission example, we've seen: SOAP endpoint URL Dspace Handles Dspace LNI URLs LNI Lookup() function WebDAV Properties WebDAV Methods PROPFIND (LNI request) MULTISTATUS (LNI response) LNI "Reverse Lookup" WebDAV Headers (Location; Success/Fail) DSpace DSpaceLNI.class OCW Comm. 1721.1/125 "Physics (8)" 1721.1/253 DSpace Object API DSpace LNI WebDAV Item 1721.1/836

50. DLF Spring 2006 MIT's CWSpace 50 Future Development Other Use Cases, Needs –Basic, Advanced … Other DSpace objects –Bitstream Format Registry –Metadata Registry … Other WebDAV methods –PUT (for Replacement) –DELETE –MKCOL …

51. CWSpace Archiving Courseware Websites to DSpace © Digital Library Federation Spring Forum, 2006 April 11, 2006 — Austin, Texas William Reilly, Rob Wolfe — MIT Libraries 2006 Questions, & Etc. http://cwspace.mit.edu/docs/ProjectMgt/Reports/MIT-CWSpace-DLF-Spring2006.ppt V. 20060410_1510