Web Server Design Week 1 Old Dominion University Department of Computer Science CS 495/595 Spring 2012 Michael L. Nelson 1/10/12

Goals We will write a web (http) server from scratch –we will not use Apache, IIS, or other existing web servers –the point is to learn http and have a working server at the end of the class your server won’t be as “good” as Apache -- and that’s ok… We will focus on the hypertext transfer protocol (http) –we will not focus on the all the neat things you can do that are outside / on top of the protocol modules, servlets, WebDAV/DASL, etc. –while not a class about REST, you will fundamentally “get” REST when we’re done…

What I’m Not Going To Teach HTML, SMTP, etc. –CS 312 Internet Concepts System administration –CS 454/554 Network Management Writing web applications (e.g. PHP/MySQL) –CS 418/518 Web Programming Network programming –CS 476/576 Systems Programming databases –CS 450/550 Database Concepts –CS 419/519 Internet Databases –and many others…. Java –CS 695 Java & XML

Administrivia This is a programming class! –I assume you know how to: do network (socket) programming write a daemon –you can develop in any environment you want to… a machine will be provided, deviate at your own risk –…but you will be graded only on the class machine real programmers use unix –your grade will be determined solely on your server’s performance on 5 different checkpoints through the semester Unlike other semesters, no teams this year.

Administrivia 2 Important URLs – – Class homepage: –Readings are listed under the day they are expected to be completed –assignments are listed under the day they will be demoed in class –each person will give a 3-4 minute status report the week before an assignment is due!

Grading 5 Assignments, 20 points each Days of in class demo are posted –demo + feedback in class –final grading done at midnight Assignments lose 3 points for every 24 hours they are late

Extra Credit Details TBD, but: –computed at of semester –extra points will be assigned based on the relative performance of your server fastest server in class gets full points slowest server in class gets 0 points separate competitions for grad & ugrad test suite, details, etc. TBD

No WWW History If you want to know more, read a book (irony intentional)

HTTP Developer’s Handbook Primary focus of this class will be reading & interpreting RFCs –RFCs are the technical documents that define how the web works But RFCs are not always the best resources to learn from –augment class slides + discussion with relevant sections from the class text book

Defining the Web / HTTP HTTP was originally defined by Request for Comments (RFCs)1945, 2068, 2616 –and several others for defining URLs, URIs, etc. – While RFC 2616 remains canonical for HTTP, we have a new document for the “Web” (URIs + protocols + formats), we have a slightly revisionist but ultimately useful unifying document: –The Architecture of the World Wide Web, Volume One.

Uniform Resource Identifiers URI URL URN RFC 2396 RFC 2141 RFC 1738 URI & URL: URL: ftp://ftp.isi.edu/pub/ftp://ftp.isi.edu/pub/ URI: info:pmid/ URN: urn:uuid:6e8bc430-9c3a-11d c9a66

From RFC 2396 “A URI can be further classified as a locator, a name, or both. The term "Uniform Resource Locator" (URL) refers to the subset of URI that identify resources via a representation of their primary access mechanism (e.g., their network "location"), rather than identifying the resource by name or by some other attribute(s) of that resource. The term "Uniform Resource Name" (URN) refers to the subset of URI that are required to remain globally unique and persistent even when the resource ceases to exist or becomes unavailable.”

URIs & URNs registered URI schemes – registered URN namespaces – namespaceshttp:// namespaces

URI Schemes \ / \________________/\_________/ \__/ \___/ \_/ \_________/ \_________/ \__/ | | | | | | | | | | userinfo hostname port | | parameter query fragment | \_______________________________/ \_____________|____|____________/ scheme | | | | | authority |path| | | | | path interpretable as filename | ___________|____________ | / \ / \ | urn:example:animal:ferret:nose interpretable as extension taken from:

How To Read RFCs (quoting from RFC 2119) The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC MUST This word, or the terms "REQUIRED" or "SHALL", mean that the definition is an absolute requirement of the specification. 2. MUST NOT This phrase, or the phrase "SHALL NOT", mean that the definition is an absolute prohibition of the specification. 3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. 4. SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label. 5. MAY This word, or the adjective "OPTIONAL", mean that an item is truly optional. One vendor may choose to include the item because a particular marketplace requires it or because the vendor feels that it enhances the product while another vendor may omit the same item. An implementation which does not include a particular option MUST be prepared to interoperate with another implementation which does include the option, though perhaps with reduced functionality. In the same vein an implementation which does include a particular option MUST be prepared to interoperate with another implementation which does not include the option (except, of course, for the feature the option provides.)

Important Web Architecture Concepts URIs Identify Resources Represent Home:: Michael L. Nelson, Old Dominion University … Representations* As defined by the Web Architecture *= we’ll see this use of “representations” conflates many terms from RFC-2616

W3C Web Architecture Resource URI Representation 2 Represents Representation 1 Represents Identifies Content Negotiation slide from Herbert Van de Sompel

Spending Some Quality Time With RFC 2616 note: HTTPbis but that is unlikely to impact our class

Terminology Highlights (section 1.3) connection A transport layer virtual circuit established between two programs for the purpose of communication. message The basic unit of HTTP communication, consisting of a structured sequence of octets matching the syntax defined in section 4 and transmitted via the connection. request An HTTP request message, as defined in section 5. response An HTTP response message, as defined in section 6. resource A network data object or service that can be identified by a URI, as defined in section 3.2. Resources may be available in multiple representations (e.g. multiple languages, data formats, size, and resolutions) or vary in other ways. entity The information transferred as the payload of a request or response. An entity consists of metainformation in the form of entity-header fields and content in the form of an entity-body, as described in section 7. representation An entity included with a response that is subject to content negotiation, as described in section 12. There may exist multiple representations associated with a particular response status.

Terminology Highlights (section 1.3) content negotiation The mechanism for selecting the appropriate representation when servicing a request, as described in section 12. The representation of entities in any response can be negotiated (including error responses). variant A resource may have one, or more than one, representation(s) associated with it at any given instant. Each of these representations is termed a `varriant'. Use of the term `variant' does not necessarily imply that the resource is subject to content negotiation. client A program that establishes connections for the purpose of sending requests. user agent The client which initiates a request. These are often browsers, editors, spiders (web-traversing robots), or other end user tools. server An application program that accepts connections in order to service requests by sending back responses. Any given program may be capable of being both a client and a server; our use of these terms refers only to the role being performed by the program for a particular connection, rather than to the program's capabilities in general. Likewise, any server may act as an origin server, proxy, gateway, or tunnel, switching behavior based on the nature of each request.

Terminology Highlights (section 1.3) origin server The server on which a given resource resides or is to be created. validator A protocol element (e.g., an entity tag or a Last-Modified time) that is used to find out whether a cache entry is an equivalent copy of an entity. upstream/downstream Upstream and downstream describe the flow of a message: all messages flow from upstream to downstream.

Intermediaries proxy –“a forwarding agent, receiving requests for a URI in its absolute form, rewriting all or part of the message, and forwarding the reformatted request toward the server identified by the URI” gateway –“a receiving agent, acting as a layer above some other server(s) and, if necessary, translating the requests to the underlying server's protocol” tunnel –“a relay point between two connections without changing the messages; tunnels are used when the communication needs to pass through an intermediary (such as a firewall) even when the intermediary cannot understand the contents of the messages.” definitions from section 1.4 of RFC 2616

No Intermediaries for Us For simplicity, we will ignore the possibility of intermediaries in our assignments No caching intermediaries –decreased importance of section 13 of RFC 2616 –unless stated otherwise, any caching activities will be on the part of the client

HTTP Operation Client Origin Server request = (method, URI, version, “MIME-like” message) response = (version, response code, “MIME-like” message)

General Template, CR/LF, and Magic Blank Lines Method URI HTTP/1.1 Some-Request-Header-1: value1 Some-Request-Header-2: value2 … (1st magic blank line) HTTP 1.1 Code String Some-Response-Header-1: value1 Some-Response-Header-2: value2 … (2nd magic blank line) entity-body Client: Server:

Talking to HTTP servers… % curl --head HTTP/ OK Date: Mon, 12 Jan :44:19 GMT Server: Apache/2.2.0 Last-Modified: Fri, 09 Jan :18:37 GMT ETag: " c71-f28dd540" Accept-Ranges: bytes Content-Length: 7281 Content-Type: text/html % curl --head HTTP/ OK Cache-Control: private, max-age=0 Date: Mon, 12 Jan :45:57 GMT Expires: -1 Content-Type: text/html; charset=ISO Set-Cookie: PREF=ID=9a80d3f602b685f3:TM= :LM= :S=imGxRyNsTD0Zczm5; expires=Wed, 12-Jan :45:57 GMT; path=/; domain=.google.com Server: gws Content-Length: 0 “curl” is convenient, but speaking raw HTTP is more fun…

GET AIHT:~/Desktop/cs595-s06 mln$ telnet 80 Trying Connected to xenon.cs.odu.edu. Escape character is '^]'. GET /~mln/index.html HTTP/1.1 Connection: close Host: HTTP/ OK Date: Mon, 09 Jan :07:04 GMT Server: Apache/ (Unix) ApacheJServ/1.1.2 PHP/4.3.4 Last-Modified: Sun, 29 May :46:53 GMT ETag: "1c52-14ed-42992d1d" Accept-Ranges: bytes Content-Length: 5357 Connection: close Content-Type: text/html Home Page for Michael L. Nelson <!-- [lots of html deleted] Connection closed by foreign host. Request (ends w/ CRLF) Response

HEAD AIHT:~/Desktop/cs595-s06 mln$ telnet 80 Trying Connected to xenon.cs.odu.edu. Escape character is '^]'. HEAD /~mln/index.html HTTP/1.1 Connection: close Host: HTTP/ OK Date: Mon, 09 Jan :14:39 GMT Server: Apache/ (Unix) ApacheJServ/1.1.2 PHP/4.3.4 Last-Modified: Sun, 29 May :46:53 GMT ETag: "1c52-14ed-42992d1d" Accept-Ranges: bytes Content-Length: 5357 Connection: close Content-Type: text/html Connection closed by foreign host.

OPTIONS (many methods) AIHT:~/Desktop/cs595-s06 mln$ telnet 80 Trying Connected to xenon.cs.odu.edu. Escape character is '^]'. OPTIONS /~mln/index.html HTTP/1.1 Connection: close Host: HTTP/ OK Date: Mon, 09 Jan :16:46 GMT Server: Apache/ (Unix) ApacheJServ/1.1.2 PHP/4.3.4 Content-Length: 0 Allow: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, PATCH, PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK, TRACE Connection: close Connection closed by foreign host.

OPTIONS (fewer methods) % telnet 80 Trying Connected to xenon.cs.odu.edu. Escape character is '^]'. OPTIONS /~mln/index.html HTTP/1.1 Connection: close Host: HTTP/ OK Date: Tue, 10 Jan :26:44 GMT Server: Apache/ (Unix) PHP/5.3.5 mod_ssl/ OpenSSL/0.9.8q Allow: GET,HEAD,POST,OPTIONS Content-Length: 0 Connection: close Content-Type: text/html Connection closed by foreign host.

Response Codes - 1xx: Informational - Request received, continuing process - 2xx: Success - The action was successfully received, understood, and accepted - 3xx: Redirection - Further action must be taken in order to complete the request - 4xx: Client Error - The request contains bad syntax or cannot be fulfilled - 5xx: Server Error - The server failed to fulfill an apparently valid request from section of RFC 2616 not “error” codes!

501 - Method Not Implemented AIHT:~/Desktop/cs595-s06 mln$ telnet 80 Trying Connected to xenon.cs.odu.edu. Escape character is '^]'. NOTAREALMETHOD /index.html HTTP/1.1 Connection: close Host: HTTP/ Method Not Implemented Date: Mon, 09 Jan :22:40 GMT Server: Apache/ (Unix) ApacheJServ/1.1.2 PHP/4.3.4 Allow: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, PATCH, PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK, TRACE Connection: close Transfer-Encoding: chunked Content-Type: text/html; charset=iso f 501 Method Not Implemented Method Not Implemented NOTAREALMETHOD to /index.html not supported. Invalid method in request NOTAREALMETHOD /index.html HTTP/1.1 Apache/ Server at Port 80 0 Connection closed by foreign host.

301 - Moved Permanently AIHT:~/Desktop/cs595-s06 mln$ telnet 80 Trying Connected to xenon.cs.odu.edu. Escape character is '^]'. GET /~mln HTTP/1.1 Connection: close Host: Connection closed by foreign host. HTTP/ Moved Permanently Date: Mon, 09 Jan :32:24 GMT Server: Apache/ (Unix) ApacheJServ/1.1.2 PHP/4.3.4 Location: Connection: close Transfer-Encoding: chunked Content-Type: text/html; charset=iso e 301 Moved Permanently Moved Permanently The document has moved here. Apache/ Server at Port 80 0

301- Moved Permanently % telnet bit.ly 80 Trying Connected to bit.ly. Escape character is '^]'. HEAD HTTP/1.1 Host: bit.ly Connection: close HTTP/ Moved Server: nginx Date: Tue, 10 Jan :34:29 GMT Content-Type: text/html; charset=utf-8 Connection: close Set-Cookie: _bit=4f0c76a5-002b9-048b1-331cf10a;domain=.bit.ly; expires=Sun Jul 8 17:34: ;path=/; HttpOnly Cache-control: private; max-age=90 Location: MIME-Version: 1.0 Content-Length: 125 the response code is REQUIRED; phrase is RECOMMENDED

302 - Found % telnet doi.acm.org 80 Trying Connected to doi.acm.org. Escape character is '^]'. HEAD HTTP/1.1 Host: doi.acm.org Connection: close HTTP/ Found Date: Tue, 10 Jan :53:36 GMT Server: Apache/2.2.3 (Red Hat) Location: Connection: close Content-Type: text/html; charset=iso

303 - See Other % telnet dx.doi.org 80 Trying Connected to dx.doi.org. Escape character is '^]'. HEAD HTTP/1.1 Host: dx.doi.org Connection: close HTTP/ See Other Server: Apache-Coyote/1.1 Location: Expires: Wed, 11 Jan :04:29 GMT Content-Type: text/html;charset=utf-8 Content-Length: 210 Date: Tue, 10 Jan :56:41 GMT Connection: close

404 - Not Found % telnet 80 Trying Connected to xenon.cs.odu.edu. Escape character is '^]'. HEAD /lasdkfjalsdkfjldaskfj HTTP/1.1 Host: Connection: close HTTP/ Not Found Date: Tue, 10 Jan :39:15 GMT Server: Apache/ (Unix) PHP/5.3.5 mod_ssl/ OpenSSL/0.9.8q Connection: close Content-Type: text/html; charset=iso Connection closed by foreign host.

401 - Unauthorized % telnet www4.cs.odu.edu 80 Trying Connected to www4.cs.odu.edu. Escape character is '^]'. HEAD HTTP/1.1 Host: www4.cs.odu.edu Connection: close HTTP/ Unauthorized Content-Length: 1656 Content-Type: text/html Server: Microsoft-IIS/6.0 WWW-Authenticate: Basic realm="www4.cs.odu.edu" MicrosoftOfficeWebServer: 5.0_Pub X-Powered-By: ASP.NET Date: Tue, 10 Jan :43:57 GMT Connection: close

400 - Bad Request % telnet 80 Trying Connected to xenon.cs.odu.edu. Escape character is '^]'. HEAD HTTP/1.1 Connection: close HTTP/ Bad Request Date: Tue, 10 Jan :24:17 GMT Server: Apache/ (Unix) PHP/5.3.5 mod_ssl/ OpenSSL/0.9.8q Connection: close Content-Type: text/html; charset=iso

505 - HTTP Version Not Supported % telnet 80 Trying Connected to xenon.cs.odu.edu. Escape character is '^]'. HEAD / HTTP/9.9 Host: Connection: close HTTP/ OK Date: Tue, 10 Jan :40:05 GMT Server: Apache/ (Unix) PHP/5.3.5 mod_ssl/ OpenSSL/0.9.8q Accept-Ranges: bytes Connection: close Content-Type: text/html Connection closed by foreign host. our servers will be more picky!

505 - HTTP Version Not Supported % telnet 80 Trying Connected to dolph.w3.org. Escape character is '^]'. HEAD / HTTP/9.9 Host: Connection: close HTTP/ Forbidden Cache-Control: no-cache Connection: close Content-Type: text/html 403 Forbidden Request forbidden by administrative rules. a curious response… 505 not defined in HTTP 1.0!

Many, Many Response Codes… 10.2 Successful 2xx OK Created Accepted Non-Authoritative Information No Content Reset Content Partial Content Redirection 3xx Multiple Choices Moved Permanently Found See Other Not Modified Use Proxy (Unused) Temporary Redirect Client Error 4xx Bad Request Unauthorized Payment Required Forbidden Not Found Method Not Allowed Not Acceptable Proxy Authentication Required Request Timeout Conflict Gone Length Required Precondition Failed Request Entity Too Large Request-URI Too Long Unsupported Media Type Requested Range Not Satisfiable Expectation Failed Server Error 5xx Internal Server Error Not Implemented Bad Gateway Service Unavailable Gateway Timeout HTTP Version Not Supported

Date/Time Format from RFC 2616 Sun, 06 Nov :49:37 GMT ; RFC 822, updated by RFC 1123 Sunday, 06-Nov-94 08:49:37 GMT ; RFC 850, obsoleted by RFC 1036 Sun Nov 6 08:49: ; ANSI C's asctime() format … HTTP/1.1 clients and servers that parse the date value MUST accept all three formats (for compatibility with HTTP/1.0), though they MUST only generate the RFC 1123 format for representing HTTP-date values in header fields. See section 19.3 for further information for simplicity, we’ll assume our clients will only generate RFC 1123 date/times

Things to Think About for Your Server Claim HTTP/1.1 –even though we’ll not fully satisfy all requirements Configuration files –should not have to recompile for trivial changes MIME types –most servers use a separate file (specified in your config file!) to map file extensions to MIME types Logging –real http servers log their events we’ll use “common log format” –you’ll need logging for debugging consider concurrent logs with varying verbosity

More Things To Think About… A resource is more than just a file in the file system… –content negotiation is in your future –sometimes we’ll give respond with only a “slice” of a file… –What does it mean to GET a directory? –eventually we’ll execute scripts…

In the future, some methods will allow a client to send an entity body to the server… Method URI HTTP/1.1 Some-Request-Header-1: value1 Some-Request-Header-2: value2 … (1st magic blank line) entity-body HTTP 1.1 Code String Some-Response-Header-1: value1 Some-Response-Header-2: value2 … (2nd magic blank line) entity-body Client: Server:

Revisiting What You Will Learn Fundamental knowledge about how http works –your future career is likely to involve web programming Working with others, explaining your results to colleagues –in real life, tasks are rarely performed in isolation How to read & interpret technical specifications and translate them into code –in real life, interesting problems are ambiguous & messy The importance of good, extensible design early in a software project –in real life, writing code from scratch is an uncommon luxury

Side Effect: You’ll Be Well Prepared for REST Programming REST == Representational State Transfer – –in contrast with RPC-style web applications: RPC: foo.com/bigApp.jsp?verb=showThing&id=123 REST: foo.com/thing/123 (w/ GET method) RPC: foo.com/bigApp.jsp?verb=editThing&id=123 REST: foo.com/thing/123 (w/ PUT method) RPC: foo.com/bigApp.jsp?verb=newThing REST: foo.com/thing/ (w/ POST method)

To Do for Next Time… Subscribe to the class list I’ll assign everyone a unique port: 70XX –your server will be accessible as: –Apache still available as: Submit preferred development language / environment –1 Unix development machine (mln-web.cs.odu.edu) will be made available; we’ll try to get whatever (Unix) languages you want –planning on: Perl, PHP, Python, Java, C/C++