Web Server Design Week 15 Old Dominion University Department of Computer Science CS 495/595 Spring 2012 Michael L. Nelson <mln@cs.odu.edu> 04/17/12
“Soft 404s” foo.edu/lookup.php?key=123456 what http status code do you return if: script “lookup.php” exists and has no syntax errors but key 123456 is deleted or invalid Competing semantic events: script is there and running = 200 OK bad key for database = 404 Not Found reading: “Sic Transit Gloria Telae: Towards an Understanding of the Web’s Decay” http://www2004.org/proceedings/docs/1p328.pdf
Why? People Forget (Or Don't Even Think About It) % cat soft-404.cgi #!/usr/bin/perl print "Content-type: text/html\n\n"; $some_lookup = 0; if ($some_lookup) { # do something here } else { print "<h1>sorry, we can't find that key</h1>\n" } % curl -i http://www.cs.odu.edu/~mln/teaching/cs595-s12/cgi/soft-404.cgi HTTP/1.1 200 OK Date: Mon, 16 Apr 2012 23:06:34 GMT Server: Apache/2.2.17 (Unix) PHP/5.3.5 mod_ssl/2.2.17 OpenSSL/0.9.8q Transfer-Encoding: chunked Content-Type: text/html <h1>sorry, we can't find that key</h1>
Soft 404s in the Live Web % curl -I http://www.cs.odu.edu/show_alerts.shtml\?adfljalsdjflkajsdfljaldfja HTTP/1.1 200 OK Date: Mon, 16 Apr 2012 23:25:17 GMT Server: Apache/2.2.17 (Unix) PHP/5.3.5 mod_ssl/2.2.17 OpenSSL/0.9.8q Accept-Ranges: bytes Content-Type: text/html http://www.cs.odu.edu/show_alerts.shtml?adfljalsdjflkajsdfljaldfja
Sites Are Better Than They Used To Be… % curl -I -L http://cgi.ebay.com/ebaymotors/Ford-Galaxie-7-Litre-1966-Ford-Galaxie-7-Litre-RARE-SURVIVOR-43-000-ORIGINAL-MILES-/270870259402\?pt=US_Cars_Trucks\&hash=item3f112026ca HTTP/1.1 301 Moved Permanently Server: Apache-Coyote/1.1 Cache-Control: private [cookies] Pragma: no-cache Location: http://www.ebay.com/ebaymotors/sch/sis.html?_itemId=270870259402&_nkw=Ford Galaxie 7 Litre 1966 Ford Galaxie 7 Litre RARE SURVIVOR 43 000 ORIGINAL MILES RlogId: p4%60bo7%60jtb9%3Fv%7F.r%2B616d2tk-136bd7c1c4c Content-Length: 0 Date: Mon, 16 Apr 2012 23:28:08 GMT RlogId: p4kjkbsdab%60jtb9%3Fvo%7B2d71f%2B6%3Anpdyhw2vn-136bd7c1d5d Location: http://pages.ebay.com/ebaymotors/sch/sis.html?_itemId=270870259402&_nkw=Ford+Galaxie+7+Litre+1966+Ford+Galaxie+7+Litre+RARE+SURVIVOR+43+000+ORIGINAL+MILES HTTP/1.1 404 Not Found ETag: Last-Modified: Content-Type: text/html;charset=utf-8 Content-Length: 84418 Date: Mon, 16 Apr 2012 23:28:09 GMT
Embedded 404s % curl -i http://yfrog.com/h56kshgj HTTP/1.1 200 OK Server: nginx/1.0.4 Date: Tue, 17 Apr 2012 14:58:57 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive X-Powered-By: PHP/5.2.9 [html deleted] % curl -i http://desmond.yfrog.com/Himename=6kshg.jpg\&xsize=640\&ysize=640l HTTP/1.1 404 Not Found Server: nginx/1.0.4 Date: Tue, 17 Apr 2012 15:03:18 GMT Content-Type: text/html Connection: keep-alive X-Powered-By: PHP/5.2.9 Content-Length: 38 X-Varnish: 437098440 Age: 0 Via: 1.1 varnish X-Varnish-Hits: 0 X-Varnish-IP: 10.103.10.16 X-Varnish-Port: 17001 Unable to read source image dimensions see also: http://www.streetfire.net/video/Wicked-Fast-Ford_166416.htm
410 Gone Apparently a case of "infosuicide" % curl -i http://diveintomark.org/archives/2004/02/04/incompatible-rss HTTP/1.1 410 Gone Server: nginx Date: Tue, 17 Apr 2012 00:50:25 GMT Content-Type: text/html; charset=iso-8859-1 Connection: keep-alive Content-Length: 331 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>410 Gone</title> </head><body> <h1>Gone</h1> <p>The requested resource<br />/archives/2004/02/04/incompatible-rss<br /> is no longer available on this server and there is no forwarding address. Please remove all references to this resource.</p> </body></html> Apparently a case of "infosuicide" http://www.hanselman.com/blog/410GoneThoughtsOnMarkDiveintomarkPilgrimsAndWhysInfosuicides.aspx Ironically, http://en.wikipedia.org/wiki/Infosuicide returns a 404 instead of a 410…
How Can We Get to Prior Versions of the Resource? Memento… TimeMap http://mementoproxy.cs.odu.edu/aggr/timemap/link/http://diveintomark.org/archives/2004/02/04/incompatible-rss TimeGate http://mementoproxy.cs.odu.edu/aggr/timegate/http://diveintomark.org/archives/2004/02/04/incompatible-rss see: http://www.mementoweb.org/ http://bit.ly/memfox
"Soft 401s?" % curl -I http://www.techsideline.com/subscription/2012/04/16/spring-qa-with-raleigh-hokie/ HTTP/1.1 200 OK Date: Tue, 17 Apr 2012 00:58:29 GMT Server: Apache X-Powered-By: PHP/5.2.6-1+lenny16 Set-Cookie: PHPSESSID=089308a1d926c03026badaaecea2b9b4; expires=Fri, 20 Apr 2012 00:58:29 GMT; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Vary: Accept-Encoding Content-Type: text/html; charset=UTF-8 Clicking on: http://www.techsideline.com/subscription/2012/04/16/spring-qa-with-raleigh-hokie/ will produce a login page
Archiving Soft 401s… % curl http://mementoproxy.cs.odu.edu/aggr/timemap/link/http://subscription.techsideline.com/tslpass/2004/article287.htm <http://mementoproxy.cs.odu.edu/aggr/timebundle/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="timebundle", <http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="original", <http://http://mementoproxy.cs.odu.edu/aggr/timemap/link/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="timemap";type="application/link-format", <http://mementoproxy.cs.odu.edu/aggr/timegate/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="timegate", <http://api.wayback.archive.org/memento/20040901225310/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="first memento";datetime="Wed, 01 Sep 2004 22:53:10 GMT", <http://api.wayback.archive.org/memento/20041204130602/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="memento";datetime="Sat, 04 Dec 2004 13:06:02 GMT", <http://api.wayback.archive.org/memento/20050307143437/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="memento";datetime="Mon, 07 Mar 2005 14:34:37 GMT", <http://api.wayback.archive.org/memento/20070714193130/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="memento";datetime="Sat, 14 Jul 2007 19:31:30 GMT", <http://api.wayback.archive.org/memento/20090609165245/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="last memento";datetime="Tue, 09 Jun 2009 16:52:45 GMT"
What Have We Archived? % curl -I -L http://api.wayback.archive.org/memento/20090609165245/http://subscription.techsideline.com/tslpass/2004/article287.htm HTTP/1.1 302 Moved Temporarily Server: Apache-Coyote/1.1 Memento-Datetime: Tue, 09 Jun 2009 16:52:45 GMT Link: [deletia] [deletia] Location: http://api.wayback.archive.org/memento/20090609165245/http://www.techsideline.com/account/loginScript.php?prompt=You+must+be+a+subscriber+to+view+this+resource.&referer=http://www.techsideline.com%2Ftslpass%2F2004%2Farticle287.htm Content-Type: text/html;charset=UTF-8 Content-Length: 760 Date: Tue, 17 Apr 2012 01:40:05 GMT HTTP/1.1 200 OK [deletia Content-Type: text/html;charset=cp1252 Content-Length: 9296 X-Varnish: 844460327 Age: 0 Via: 1.1 varnish Connection: keep-alive X-Cache: MISS http://api.wayback.archive.org/memento/20090609165245/http://subscription.techsideline.com/tslpass/2004/article287.htm also, http://subscription.techsideline.com/tslpass/2004/article287.htm is nowunprotected
401 vs. 403 % curl -I -L http://api.wayback.archive.org/memento/20040901225310/http://subscription.techsideline.com/tslpass/2004/article287.htm HTTP/1.1 403 Forbidden Server: Apache-Coyote/1.1 Memento-Datetime: Wed, 01 Sep 2004 22:53:10 GMT Link: <http://api.wayback.archive.org/list/timebundle/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="timebundle", <http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="original", <http://api.wayback.archive.org/memento/20040901225310/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="first memento"; datetime="Wed, 01 Sep 2004 22:53:10 GMT", <http://api.wayback.archive.org/memento/20090609165245/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="last memento"; datetime="Tue, 09 Jun 2009 16:52:45 GMT", <http://api.wayback.archive.org/memento/20041204130602/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="next memento"; datetime="Sat, 04 Dec 2004 13:06:02 GMT" , <http://api.wayback.archive.org/list/timemap/link/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="timemap"; type="application/link-format",<http://api.wayback.archive.org/memento/timegate/http://subscription.techsideline.com/tslpass/2004/article287.htm>;rel="timegate" [some headers deleted] Content-Type: text/html;charset=UTF-8 Content-Length: 2999 Date: Tue, 17 Apr 2012 15:52:58 GMT X-Varnish: 703087026 Age: 0 Via: 1.1 varnish Connection: keep-alive X-Cache: MISS You can make a case this should be a 401, but most clients handle 401 with a popup, so perhaps 403 is the correct response. But 403 also means "Authorization will not help and the request SHOULD NOT be repeated." A more difficult question: what does it mean to have an archived authentication page?
503 Service Unavailable The server can optionally provide % curl -I http://1000Memories.com/egypt HTTP/1.1 503 Service Unavailable Server: nginx Date: Mon, 06 Feb 2012 22:06:47 GMT Connection: keep-alive Content-Length: 606 X-Varnish: 414798630 Age: 0 Via: 1.1 varnish The server can optionally provide the client with additional guidance: Retry-After: Fri, 31 Dec 1999 23:59:59 GMT Retry-After: 120
204 Used Incorrectly… % curl -I http://yfrog.com/h7ap5rdj:iphone HTTP/1.1 204 No Content Server: nginx/1.0.4 Date: Tue, 17 Apr 2012 02:31:24 GMT Connection: keep-alive should be 405 Method Not Allowed % curl -i http://yfrog.com/h7ap5rdj:iphone HTTP/1.1 301 Moved Permanently Server: nginx/1.0.4 Date: Tue, 17 Apr 2012 02:30:00 GMT Content-Type: text/html Transfer-Encoding: chunked Connection: keep-alive X-Powered-By: PHP/5.2.9 Location: http://desmond.yfrog.com/Himg619/scaled.php?tn=0&server=619&filename=ap5rd.jpg&xsize=480&ysize=480
100 Continue PUT /~mln/cars/1966-Fairlane.txt Host: coolfords.org Date: Tue, 14 Apr 2009 04:34:00 GMT Content-Type: text/plain Content-Length: 193 Connection: close ______________ // \\ ---------//--------------\\------- | __ __ | |--/ \--------------------/ \---| \__/ \__/ HTTP 1.1 201 Created Server: FordServer ETag: “XYZ” PUT /~mln/cars/1966-Fairlane.txt Host: coolfords.org Date: Tue, 14 Apr 2009 04:34:00 GMT Content-Type: text/plain Content-Length: 193 Connection: close Expect: 100-Continue HTTP 1.1 100 Continue ______________ // \\ ---------//--------------\\------- | __ __ | |--/ \--------------------/ \---| \__/ \__/ HTTP 1.1 201 Created Server: FordServer ETag: “XYZ” Possible response codes: 200, 201, 401, 403, 405, 411, 413, 414, 417 Applicable to any method in which the client sends an entity body to the server (i.e., POST, PUT) Ignore the header if not applicable to the method.
Status Codes I've Never Seen in the Wild (Perhaps they exist, but…) 203 Non-Authoritative Information 205 Reset Content 305 Use Proxy 307 Temporary Redirect 402 Payment Required 407 Proxy Authentication Required 417 Expectation Failed 502 Bad Gateway 504 Gateway Timeout
Partial GETs… Partial PUTs? % cat patch-1.txt A new method is necessary to improve interoperability and prevent errors. The PUT method is already defined to overwrite a resource with a complete new body, and cannot be reused to do partial changes. Otherwise, proxies and caches, and even clients and servers, may get confused as to the result of the operation. POST is already used but without broad interoperability (for one, there is no standard way to discover patch format support). PATCH was mentioned in earlier HTTP specifications, but not completely defined. % cat patch-2.txt Otherwise, proxies and caches, XXand even clients and servers, may get % diff patch-1.txt patch-2.txt 4c4 < Otherwise, proxies and caches, and even clients and servers, may get --- > Otherwise, proxies and caches, XXand even clients and servers, may get
PATCH Method PATCH /file.txt HTTP/1.1 Host: www.example.com Content-Type: text/x-patch If-Match: "e0023aa4e" Content-Length: 158 Connection: close 4c4 < Otherwise, proxies and caches, and even clients and servers, may get --- > Otherwise, proxies and caches, XXand even clients and servers, may get HTTP/1.1 204 No Content Content-Location: /file.txt ETag: "e0023aa4f" see RFC 5789 (text/x-patch & text/x-diff are not officially registered w/ IANA)
HTTP as a Filesystem WebDAV Methods: http://en.wikipedia.org/wiki/WebDAV RFC 2518 Methods: PROPFIND — used to retrieve properties (i.e., metadata) from a resource. PROPPATCH — used to change and delete multiple properties on a resource MKCOL — used to create collections (i.e., directories) COPY — used to copy a resource from one URI to another MOVE — used to move a resource from one URI to another LOCK — used to lock a resource UNLOCK — used to remove a lock from a resource