1. Andrew McNab - Grid HTTP/HTTPS extensions Grid HTTP/HTTPS extensions 18 November 2002 Andrew McNab, University of Manchester mcnab@hep.man.ac.uk

2. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 Overview u HTTPS as a grid protocol u HTTP as a data protcol u Multistream HTTP: curl-url-get u Grid HTTP/HTTPS usage u G-HTTPS u Trusted Caches u fileGridSite HTTPS server u Third Party Transfers u curlfs for SlashGrid u Summary

3. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 HTTPS as a Grid protocol u HTTPS is an interesting and important protocol for several reasons: n it is by far the most widely deployed secure protocol n has a large amount of high quality software that we could leverage n has excellent interaction with Firewalls, Network Address Translation and Application Proxies s has the potential to solve some of the problems sites have with private IP farms n along with HTTP, is the basis for Web and Grid Services u HTTPS consists of HTTP/1.1 over an SSL connection n security done by SSL layer, using X509 certificates (including GSI) u HTTP/1.1 (rfc2616) and extensions like WebDAV (rfc2518) have a rich set of methods (GET, PUT, DELETE, COPY etc) headers (“Expires:” etc) and Errors (“413 Request Entity Too Large”) n so a standard way exists for many of the transfer operations we need

4. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 HTTP as a data protocol u Same advantages as HTTPS: large amount of existing high quality software, and good operation with Firewalls, NAT etc. u If we build secure HTTPS information/control services, easy to provide HTTP data services: n Do GET during HTTPS session, but server responds with redirect to HTTP data server? n So GridFTP Control & Data channels --> HTTPS Negotiate and HTTP Data connections u Kernel-based “zero-copy” HTTP servers like tux are very efficient n need to do something like that to fully use a machine’s gigabit interface u HTTP connection and a GridFTP data channel are same at TCP layer n but may want a way to specify TCP parameters to be used by HTTP server responding with data

5. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 Multistream HTTP u HTTP can support application-level multiple streams and striping by using the standard Range: header from RFC 2616 (HTTP/1.1) to set up many partial fetches. u This mechanism is supported by almost all modern web servers n eg Apache and RedHat’s tux kernel httpd u Multiple streams implemented by client splitting into threads n Each thread requests a block of the file from the server n As each request completes, thread finds next unfetched block and requests it u Striping by doing the same mechanism, but with more than one server u curl-url-get demonstrates both of these n source is 300 lines of C, in EDG CVS

6. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 curl-url-get examples u Rough tests done, copying files from Manchester to CERN n elapsed times in seconds, average of 10 copies of each type, alternated Size curl-url-get globus-url-copy streams 292M64.6±6.162.1±4.920 292M96.0±9.274.8±3.8 5 29M 7.1±1.4 6.9±1.820 29M31.6±0.415.9±0.9 1 2.9M0.49±0.072.24±0.1020 2.9M3.30±0.162.61±0.18 1 2.9K2.15±0.0420 2.9K0.11±0.001.05±0.10 1

7. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 Extensions to HTTPS/HTTP u HTTPS/HTTP already have most of the functionality we need for Grid information/control/data transport n some of these come from several sources (eg the WebDAV RFC2518 not just HTTP/1.1 itself) and can be done different ways n so want to specify a sufficient subset for interoperability u However, can identify some extensions that are also needed: n delegation to HTTPS n some way of returning access control information along with data n other metadata too n may want to specify TCP parameters for bulk data tranfer

8. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 “G-HTTPS” u A proposal by Akos and me, for backwards compatible extensions to HTTPS n discussed on wp2-sec and wp7-security lists u Adds GSI proxy delegation to HTTPS using additional methods (eg PUT-PROXY) and headers (eg Delegation-ID) u Allows services to return generalised metadata in headers or by URL n initially this allows services to return the GACL ACL of a response for more efficient caching (ie sharing cached copies with other users.) n essential to include expiration and caching policy information too u Aim is to avoid breaking existing HTTPS systems and to achieve “pass through” compatibility: n even if HTTPS client or server software doesn’t understand extensions, they can make them available to the application which does

9. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 Example of delegation by HTTPS u Client issues GET-PROXY-REQ request, perhaps with a message body specifying any extensions required in the proxy cert u Server generates a key and a certificate request, returns this in the response message body. u Client signs this, and returns it in the body of a PUT-PROXY request u Need a Delegation-ID header in the above exchanges so can keep track of the delegation session n may want to maintain delegation sessions for the same user at one server, but with different amounts of delegation u Subsequent GET, PUT etc actions carry on using the Delegation-ID u Non G-HTTPS server will respond with “501 Method not implemented” to above methods

10. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 Application of delegation: Trusted Caches u Many information services are going to need delegation, but Trusted Caches are one purely file transfer application of this u Existing HTTPS isn’t cache-able: n connection from client to origin server for trust to mechanism work n So best you get is opaque proxying/tunneling of SSL u With delegation, can improve this: n identifies a caching server it trusts (in its VO maybe?) n delegates a credential to it n makes an HTTP proxy request via HTTPS: GET http://a.b.c/def n caching server fetches this using delegated credential, gives it to client n if can get an ACL for this file, may be able to return file from cache in subsequent requests n also means that only real HTTPS works, not other things hidden in SSL

11. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 fileGridSite u Read (GET) well supported by HTTPS servers. u However, write (PUT, DELETE, MOVE, COPY) usually left to CGI programs, servlets etc. n Access control also usually limited to client IP or HTTP passwords. u fileGridSite adds Grid authorisation and write operation support to Apache n a cut-down version of GridSite (used for https://marianne.in2p3.fr) n file rather than webpage orientated (no fancy headers on HTML etc) n uses GACL to handle the Access Control Lists n can work with mod_ssl-GSI so clients can authenticate with a GSI proxy u Turns an Apache webserver into a Grid HTTPS fileserver with the key functionality of a GridFTP server.

12. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 fileGridSite examples with curl u Curl is a standard HTTP/HTTPS command line client (cf wget) u Get a file using GSI proxy in /tmp/x509up_u100 n curl --capath /etc/grid-security/certificates/ --cert /tmp/x509up_u100 https://a.b.com/example1.txt u Copy a file to the fileGridSite server with HTTP PUT: n curl --capath /etc/grid-security/certificates/ --cert /tmp/x509up_u100 --upload-file /tmp/example2.txt https://a.b.com/example2.txt u Delete a file with HTTP DELETE: n curl --capath /etc/grid-security/certificates/ --cert /tmp/x509up_u100 --request DELETE https://a.b.com/example2.txt u Create a directory with PUT to …/ n curl --capath /etc/grid-security/certificates/ --cert /tmp/x509up_u100 --request PUT https://a.b.com/newdir/

13. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 Adding delegation to fileGridSite u Doing this as a demonstration of G-HTTPS extensions u Delegation needed for Third Party Transfers n Use COPY from WebDAV RFC2518 which allows source or destination to be absolute URL’s n Spec actually allows “fourth party” too, involving two remote URL’s and the transfer being tunneled through the server. u Delegation also useful for fileservers which need credentials to access local storage n to get token for local AFS cell (Lyon have had to work around this with GridFTP servers)

14. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 curlfs for SlashGrid u curl is built on top of a general library, libcurl n handles persistent HTTP and HTTPS connections, SSL setup etc u To add HTTP and HTTPS filesystems to SlashGrid, have made a libcurl filesystem plugin: curlfs u This maps parts of the URL space into the local filesystem: n https://a.b.com/newdir/ ---> /grid/https/a.b.com/newdir/ u Works with any standard HTTP or HTTPS server n rpm -i /grid/http/datagrid.in2p3.fr/distribution/globus/beta-21/RPMS/* u SlashGrid framework provides GSI proxy or full cert/key to curlfs so it can make authenticated requests. u Write with HTTP/1.1 PUT and DELETE being added to curlfs n Will complement fileGridSite support for these on server side

15. Andrew McNab - Grid HTTP/HTTPS extensions - 18 Nov 2002 Summary u HTTPS as a grid protocol n G-HTTPS extensions being worked out u HTTP as a data protocol n even a quick multistream HTTP hack seems very competitive u fileGridSite HTTP(S) server has been written n supports read/write with standard utilities like curl n third party transfers being added as demonstration of delegation u curlfs written for SlashGrid: maps URL’s into filesystem u Source code for curl-url-get, fileGridSite, curlfs is in EDG CVS u See http://www.gridpp.ac.uk/authz/ for more details