1 WebDAV and DeltaV: Collaborative Authoring, Versioning, and Configuration Management for the Web Jim Whitehead University of California, Santa Cruz

2 WebDAV and DeltaV  WebDAV and DeltaV are:  Application layer network protocols  Extensions to the Hypertext Transfer Protocol (HTTP 1.1)  Developed by the Internet Engineering Task Force (IETF)  WebDAV  Remote collaborative authoring of Web resources  Overwrite prevention, metadata management, namespace operations  DeltaV  Versioning and Configuration Management  Extends base WebDAV capabilities  Checkout/checkin, version history, logical change tracking, workspaces, configuration management

3 Major WebDAV Clients  Application Software:  Microsoft: Office 2000/XP (Word, Excel, PowerPoint, Publisher)  Adobe: Photoshop 6, Illustrator 10, Acrobat 5, In Design 2  Web Site Authoring  Adobe: Go Live 5  Macromedia: Dreamweaver 4  Remote File Access:  Apple: Mac OS X webdavfs  OS X also ships with Apache and mod_dav (have to configure mod_dav to make it work)  Microsoft: Windows Web Folders  Wind River Software: WebDrive  Goliath (Mac, open source)  WebDAV Explorer (UC Irvine, Feise/Kanomata, open source)  XML editors  Excosoft: Documentor  Altova: XML Spy  SoftQuad: XMetal

4 Major WebDAV Servers Microsoft: IIS 5/6, Exchange 2000, Sharepoint Apache: mod_dav (over 95,000 sites) Oracle: Internet File System Adobe: InScope Xythos: Web File Server Novell: Netware 5.1, Net Publisher W3C: Jigsaw Endeavors: Magi-DAV IBM: DAV4J (DeveloperWorks) FileNet: Panagon ECM Intraspect: 4i Merant: PVCS Dimensions, Content Manager Hyperwave: Information Server 5.5 4D: WebSTAR V

5 Collaborative Document Authoring  Three collaborators, in different cities, use Word 2000 to collaborate on a report they are producing together.

6 Collaborative Web Site Authoring  Two homes each develop their family Web site using WebDAV to interact with their ISP.

7 Remote Authoring, Part of Staged Production

8 Visions for WebDAV  Participants in WebDAV have many views on what it is:  A protocol for collaborative authoring of all document types  XML, HTML, word processing, spreadsheets,  A Web-based network file system, with nice high-latency behavior  Better performance than NFS and Samba over the Internet  A data integration technology for accessing a wide range of repositories  Document mgmt. systems, configuration mgmt. systems, filesystems, etc.  Remote software engineering infrastructure  A replacement protocol that can handle , calendaring, directory lookup and more  Could replace: POP, IMAP, CAP, LDAP…  All views are correct!

9 Filesystem View  Exemplars: Web Folders, WebDrive, WebIFS, TeamDrive, Mac OS X

10 Document Authoring  Exemplars: Office 2000/XP: Word, Excel, PowerPoint, as well as Photoshop, Documentor, and XML Spy Office: uses filesystem metaphor for WebDAV location

11 Photoshop  Workflow metaphor for WebDAV location

12 Web Site Authoring  Exemplars: Go Live 5/6, Dreamweaver  Site metaphor for WebDAV location

13 Remote Collaborative Annotation  Acrobat 5 views a WebDAV location as a storage location for document annotations  Annotations are stored in resources separate from the PDF document  One collection per document  One annotation resource per user (in collection)

14 Facets of WEBDAV  There are many ways to view the DAV work:  Collaboration infrastructure  Metadata repository infrastructure  Namespace management infrastructure  Access control infrastructure  Searching infrastructure – DASL

15 Collaboration Infrastructure  Whole resource locking supports:  remote collaborative authoring of any media type  Web pages, Word processing, Presentations, XML, …  Lock characteristics:  Long-duration locks  Not associated with network connection  Client receives a lock token identifying the lock  Can disconnect from network after receiving lock token  Locks automatically expire after a client specified timeout period  Lock single resources, or hierarchies of resources  Infrastructure for asynchronous, widely distributed, hypertext-aware, collaborative editing tools.

16 Metadata Recording Infrastructure  Metadata support  Properties are (name, value) pairs that can be created, modified, deleted, and read on Web resources.  Consistency of properties can be maintained by the server or the client  Property values are well-formed Extensible Markup Language (XML)  Can store RDF as well  Property name is a URI or URL  Extensible, global property namespace  Infrastructure for recording information about Web data  A general purpose metadata repository

17 Namespace Management Infrastructure  Remote name space management:  Copy and Move  individual resources  hierarchies of resources  to/from a locked hierarchy  Create and modify collections of resources  Retrieve listings of collection members  Useful for creating Save…As dialog boxes  Adds hierarchical navigation to the Web  Augments hypertext navigation and searching  Infrastructure for remotely organizing and viewing collections of Web resources

18 Access Control Infrastructure  Access Control:  The ability to remotely control who can read and write a resource  Key challenges:  Expose the access control capabilities of the repository…  …while ensuring the client-side user interface can be simple (I.e., avoid lots of feature discovery)  Provide some notion of principal, without duplicating LDAP  Handle large numbers of principals  Access control lists  Each entry grants/denies a privilege to a principal (or group)  Infrastructure for remotely creating collaboration groups

19 Searching Infrastructure  Searching a WebDAV repository - DASL:  Search for resources with a given property, or a given property value  Search for a substring inside a resource body  Search scope can be one resource, a collection of resources, a hierarchy of resources, or a whole server  Search syntax is extensible  Search specification specified as body submitted with SEARCH command  Could accommodate XML Query syntax  Infrastructure for remote searching

20 WebDAV Methods and Data Model

21 WebDAV Methods  Overwrite Prevention:  LOCK – prevents non-lock holders from writing to the resource  UNLOCK – removes a lock  Metadata Management:  PROPFIND – read properties from a resource  Allprop – all property names and values  Propname – only return property names  Prop – just return specified properties  PROPPATCH – write properties on a resource  Namespace Management  COPY – duplicate a resource  MOVE – move a resource (preserving identity)  MKCOL – create a new collection

22 Scope of WebDAV Methods Web Resource Body (primary state) Properties (name, value) pairs LOCK UNLOCK COPY MOVE † DELETE † MKCOL † (PUT † ) PROPFIND PROPPATCH † GET PUT † † - affected by LOCK

23 DeltaV: Versioning and Configuration Management

24 Remote Collaborative Document Authoring Scenarios  A task force of people from geographically dispersed business units need to develop a report together.  Team needs to solicit feedback, so they send out copies of the report.  As a result, the team needs a permanent copy of the exact report version they are having other people review.

25 What is DeltaV?  DeltaV is:  An application layer network protocol  Extends the WebDAV protocol  WebDAV itself extends HTTP 1.1  Thus DeltaV is also an extension to HTTP 1.1  HTTP is the core network protocol of the Web  An interoperability standard (RFC 3253)  Allows document authoring and software development applications to interoperate with versioning and configuration management systems using a common interface  A data model  DeltaV embodies a data model that is capable of modeling multiple versioning and configuration management repositories  The data model is independent of network protocol  A data integration infrastructure  A common place for cross-application data sharing

26 Version History Foo.htm initial Beta1 Test1 Beta2 URL path of Versioned Controlled Resource Version Name Label Successor Line of Descent Revision History Predecessor Branch Merge

27 Representation of a Version History

28 Autoversioning  There are many WebDAV-aware applications  They know nothing about versioning!  Would like to provide automatic versioning support  Two styles of autoversioning: 1.Every modification creates a new version  PUT/PROPPATCH  CHECKOUT  PUT/PROPPATCH  CHECKIN  Works best for authoring clients that replicate resources to a local disk, and only write to the server at the end of an editing session. 2.Every LOCK/UNLOCK pair creates a new version  LOCK  LOCK  CHECKOUT  UNLOCK  CHECKIN  UNLOCK  Works well for authoring clients that work directly on the WebDAV server and take out locks, like Office  Style is controlled by DAV:autocheckout and DAV:autocheckin properties on version controlled resource

29 Workspaces  A workspace is a location…  … where a person can work in isolation  … from ongoing changes made by other collaborators  to the same set of resources.  Workspace use scenarios:  Two people each modify a common source code file, and want to test their work before integrating changes  The directory structure of a large project is being changed  Don’t want to require everyone to stop working while this takes place  Keep namespace changes invisible to team until complete  A developer is trying a change that might not pan out  New files are in their workspace, but not under version control until the final viability of the change is known

30 Server Workspace Example  Version history of main.c: 1234 /repo/o522/v1/repo/o522/v2/repo/o522/v3/repo/o522/v4 version history resource /repo/o522/v1 /repo/o522/v2 /repo/o522/v3 /repo/o522/v4 version resources /his/o522 /projectX/main.c /users/chris/ projectX/main.c /users/geoff/ projectX/main.c version controlled resources 5 /repo/o522/v5

31 Server Workspace Example main.c 2 makefile defs.h 2 1 /users/chris/projectX/ main.c, 4 makefile, 3 defs.h, 3 /users/geoff/projectX/ main.c, 5 makefile, 3 defs.h, 2 3 5

32 Baselines  When a large collection of documents is released outside the developing organization, there is a need to record exactly which document versions were released.  Especially true for software project releases  A baseline can record the exact version of a large set of resources under version control  Baselines can also be used for recording intermediate project states  The state of the project as of last Friday  Quality assurance can then have a stable snapshot to work from  Baselines are used to perform configuration management

33 WebDAV Projects at UC Santa Cruz

34 Recontextualizing WebDAV  WebDAV and DeltaV gathered decontextualized requirements for collaboration and version control  Abstracted away characteristics of specific application domains  Used Web authoring & software development as driving problems  Led to a powerful, inexpensive, generic content management capability  Now, goal is to apply the general-purpose capabilities of WebDAV/DeltaV to solve specific problems:  Electronic records management  Digital libraries  … as well as ensuring they satisfy original goals:  Document management  Software development

35 Advanced Document Collaboration Infrastructure  Goal: Create advanced open-source WebDAV server with following characteristics:  Highly scalable  store and search millions of documents  Data integration  knit together multiple document and metadata repositories  Access Postdoc, legacy stores via DAV/DeltaV  Access Control  support the emerging ACL protocol standard  Versioning  document versioning and auto-versioning support

36 Advanced Document Collaboration Infrastructure (2)

37 Advanced Document Collaboration Infrastructure (3)  Research challenges:  Abstract repository interface  Develop API between mod_dav_repos and repository wrappers  Scalability  Ensuring the architecture supports large numbers of documents and storage  Supports efficient searching  Cross-repository searching  Support for efficient DASL searches  Accommodating server-defined and client-defined properties  Allowing searches that span the contents of multiple repositories  Access Control Implementation  Create an open reference implementation of the access control protocol  Develop an access control client that can adapt to multiple access privilege hierarchies

38 Advanced Document Collaboration Infrastructure (4)  Work performed so far:  Created mod_dav_dasl prototype  Single database, MySQL  Storage and searching of arbitrary dead properties  Implemented DASL  Some performance characterization  Client support for DASL in cadaver client  Work is underway to release the code  Work performed by Elias Sinderson (also working at ARC), Sung Kim, Kai Pan

39 Electronic Records Management  US Government agencies are required to support standard DoD  Describes metadata items useful for archiving electronic records  is not an interoperability standard  Approach  Develop an XML representation of metadata items  Store metadata in WebDAV properties  Develop server support for  automatic archiving/disposal of records  automatic setting of property values  Develop client support for entry/viewing/searching items  Work performed by Dorrit Gordon, Leila Naslavsky

40 Electronic Records Management (2)  Work performed so far:  Preliminary XML metadata schema for metadata items  Modular approach allows work to potentially be used for ERM in other countries (e.g., VERS standard in Australia)  Initial work on identifying items with controlled vocabularies  Containment data modeling of data model  Payoff  Wide deployment of interoperable, low-cost infrastructure for archiving electronic records  Infrastructure for personal records management  In the future, how will you manage that 10TB hard drive on your PC?

41 Personal Digital Library  Researchers interact with a number of institutional digital libraries  ACM and IEEE are significant for CS researchers  Limited existing support for users interacting with these institutional DLs  Either read files online, or  Store files in a filesystem  Institutional DLs have poor track record for reliability, QoS  Users want to have local copies of relevant documents  Disk sizes are large, and growing  It’s reasonable to keep a copy of every document ever read  How do you manage all these documents?

42 Personal Digital Library (2)  Most researchers know other researchers in their area  If another researcher in your area finds an interesting paper, you’ll want a copy  This should happen automatically, just by being a member of a research peer group  Our approach:  Use a WebDAV server to store a personal, or small workgroup’s digital library  Augment documents with bibliographic metadata  Special-purpose document browsing and searching client  Provides a document UI, not a file UI  Emphasizes bibliographic metadata in browsing UI  Integrate P2P capability for document sharing  User-selectable policies for IP enforcement  Current status: NSF proposal in-progress

43 Adaptive Configuration Management Client  DeltaV specification defines multiple packages (feature sets) of functionality  Core versioning  Basic server workspace  Basic client workspace  Advanced server workspace  Advanced client workspace  A server may support one or more of these  Research challenge:  Create a DeltaV client that can adapt to any server it encounters  Implies it can adapt to different workspace semantics, and different supported features  Existing CM clients are tailored to a single system  Status: looking for a sponsor

44 Software Engineering at UC Santa Cruz

45 Software UCSC  Currently in the process of building up an SE research group  Hiring for three SE faculty this year  In-process of developing a Masters degree program in Software Engineering  Aimed at working professionals in Silicon Valley  Goal is to eventually offer many courses at the UC Silicon Valley Center (on the Ames campus)  Coursework in Software Engineering plus a significant multi- quarter (perhaps including summer) project  Also developing a Masters program in Web and Internet Engineering  Courses in SE, databases, networking, storage systems  Also aimed at SV working professionals

46 Study Wish List  How are people actually using WebDAV technology?  Many stories of people using it for file sharing  Less evidence of collaborative authoring use  In particular, Go Live and Director are designed for group use  How are people using the collaboration facilities?  WebDAV has relatively poor awareness support –does this affect use?  How is the technology used differently in different organizational settings?  We finally have a remote collaboration technology that is starting to be broadly used  We can now test many long-held assumptions about this technology

47 Project Wish List  DAVCam  A digital camera that takes a picture…  Automatically writes the picture to a DAV store  Allows voice-to-text annotation of photo with metadata  Open Hypertext integration  Export hypertext links as Xlinks in DAV properties  Allow authoring of links via these properties  Mapping to OHP Storage interface  DAV as storage layer for spatial hypertext  Is DAV sufficient by itself? Or are additional capabilities or optimizations needed?  DAV-based Weblog (“blog”) system  Author content via DAV

48 WebDAV Resources

49 WebDAV  WebDAV Resources   A central collection of pages and links to all things WebDAV.  WebDAV Working Group   Contains links to active documents, and a complete list of WebDAV- supporting applications.  RFC 2518 – WebDAV Distributed Authoring Protocol   This is the WebDAV Distributed Authoring Protocol specification  WebDAV: A network protocol for remote collaborative authoring on the Web  Proc. of the Sixth European Conference on Computer-Supported Cooperative Work, Sept , 1999, Copenhagen, Denmark, pp   An academic paper giving an overview of the WebDAV Distributed Authoring Protocol.

50 DeltaV  Delta-V Working Group web page   The home page for the IETF Delta-V Working Group, with links off to the most recent specifications.  RFC 3253 – DeltaV Protocol   Extensions to the WebDAV protocol (RFC 2518) for versioning and configuration management  The Future of Distributed Software Development on the Internet  Web Techniques, Vol. 4, No. 10, October, 1999, pages   An introduction to WebDAV and DeltaV that describes the advantages of DeltaV over CVS for remote collaborative software development

51 DASL: DAV Searching & Locating  DAV Searching and Locating page   A web site containing links to the most recent WebDAV searching protocol specifications.  J. F. Reschke, S. Reddy, J. Davis, A. Babich, “WebDAV Search”  search-latest.html  The most recently edited DASL protocol specification.  Significantly increased momentum recently  Likely to see specification approved this year.

52 Access Control Protocol  Access Control page   A web site containing links to current access control protocol specifications.  WebDAV Access Control Protocol  draft-ietf-webdav-acl-07, November,  07.htm  The most recent revision of the access control protocol specification.

53 Advanced Collections  WebDAV Bindings  draft-ietf-webdav-binding-protocol-02, December 17,  webdav-binding-protocol-02.txt  The most recent revision of the WebDAV Bindings Protocol.  WebDAV Redirect Reference Resources  draft-ietf-webdav-redirectref-protocol-02, December 17,  webdav-redirectref-protocol-02.txt  The most recent revision of the Redirect Resources protocol.  WebDAV Ordered Collections Protocol  draft-ietf-webdav-ordering-protocol-02, December 20,  webdav-ordering-protocol-02.txt  The most recent revision of the Ordered Collections protocol.

54 Document Roadmap Distributed Authoring Locking, Properties, Copy/Move RFC 2518 complete Versioning & CM Protocol just sent to IESG RFC 3253 complete Advanced Collections Requirements and protocol for bindings, redirectors, ordered coll. Finish:June 2003 Access Control Protocol for remote access control Finish: Spring 2002 WebDAV Working Group: DASL: Searching Requirements and protocol for searching a WebDAV repository Finish: late 2002 Delta-V Working Group:

55

56 Overwrite Prevention: Locking

57 Write Lock  A write lock prevents a principal without the lock from successfully modifying the state of the resource  Specifically, it prevents execution of PUT, POST, DELETE, MKCOL, PROPPATCH, MOVE, LOCK, UNLOCK  GET, PROPFIND are unaffected by write lock

58 Write Lock Scope  The scope of a lock is an entire resource  It is not possible to specify a sub-resource lock. Why?  The ability to create a sub-resource lock requires knowledge about the content type being locked.  WebDAV wanted to support all content types equally, not give preferential support to a few.  Due to the large number of content types, support for sub- resource locks would have added a lot of extra semantics to the protocol. Some content types wouldn’t have been supported.  Content types change rapidly (two revisions of HTML and XML occurred during development of WebDAV protocol). Supporting one revision of a content type would make protocol brittle, quickly obsolete.  New content types would not be supported, yet new content types emerge constantly.

59 Write Lock Scope (3)  Implications of whole-resource locking:  Pro:  Supports existing applications which operate on entire files, providing easy migration path to add WebDAV support  Handles all content types  Con:  Reduced availability of resources during collaboration (but, shared locks can help…)

60 Write Lock (cont’d)  Live properties may still change even though a lock is active  Dead properties may only be changed by the lock owner.  A null resource may be locked to reserve its name. This puts the resource into a special state, lock-null, since it now has lock related properties defined on it.  In retrospect, a bad idea  Adds a lot of special case code  Not a compelling feature for clients  Feature may go away in next revision of RFC 2518

61 Lifecycle of a Lock unlocked locked LOCK UNLOCK LOCK (refresh) lock timeout administrator removal of lock

62 Exclusive Lock Zero or one possible lock holders lock holder access permission holders everyone on the Internet

63 Shared Lock Zero, one, or many possible lock holders lock holders access permission holders everyone on the Internet

64 Why Exclusive and Shared?  Exclusive locks are too rigid  people often forget to release locks  requires an administrator to release the lock  Shared locks allow people to use out-of-band communication to negotiate access to the resource  if someone forgets to release a lock, it doesn’t hold up the entire group  Collaborators work opportunistically

65 Lock Compatibility Current lock state Shared Lock Exclusive Lock None Shared Lock Exclusive Lock true false false* Legend: True = lock MAY be granted * = owner of lock MAY False = lock MUST NOT be granted have lock regranted Lock request

66 LOCK Required Support  A WEBDAV server is not required to support locking  Why? Systems differ markedly in the type of locking support they provide, and may not be able to support locks at all (i.e., some replicated stores)  … However, most WebDAV servers do support locking

67 LOCK Method  LOCK creates the lock specified by the XML element (in the request body) on the Request-URI.  Lock owner information is submitted with a lock request  A timeout value may also be requested  LOCK returns a lock token which uniquely identifies the lock to the server

68 Lock Owner Information  The owner XML element (inside ) provides a means to associate lock holder contact information with a lock.  If you want the lock holder to release the lock, perhaps you can contact them and ask them to relinquish it  Authentication information often does not contain contact information (e.g., a key)

69 LOCK, single resource LOCK /webdav.html HTTP/1.1 Host: sandbox.xerox.com Timeout: Second-500, Infinite Content-Type: text/xml Content-Length: 151

70 LOCK, single resource (2) HTTP/ OK Date: Tue, 09 Feb :25:21 GMT Server: PyDAV 1.1 filestore 1.1 Content-Type: text/xml Content-Length: 435

71 LOCK, single resource (3) Infinity Second-500 opaquelocktoken:

72 Hierarchy Locks  Using the Depth header set to Infinity, can lock a collection hierarchy  A single lock token is returned, identifying the lock on all resources.  An UNLOCK on this token removes the lock from all associated resources.  All or nothing semantics  All resources in hierarchy are locked, or none are  Hierarchy locks act to ensure:  All resources in the hierarchy are members of the lock  Resources removed from the hierarchy are removed from the lock  But…  If a locked hierarchy is copied/moved, the destination hierarchy is not locked.

73 UNLOCK  UNLOCK removes the lock identified by a lock token from the Request-URI, and all other resources included in the lock  If a lock affects an entire collection, UNLOCK removes the lock from all resources in the collection.