Bringing It All Together Analyzing Web Server Log Files Eric Landrieu Lead Developer, PerfMan for Web Servers The Information Systems Manager, Inc.

Bringing It All Together Growth of Web Server Has become a vital part of the business model Internet web servers must be reliable, as they are truly an international 24x7x365 sales mechanism Content of site(s) can be just as damaging in user’s eyes as poor performance – we have a 2-edged sword

Bringing It All Together So how do we monitor the web server? OS-level tools –Performance Monitor (Windows NT) –SMF, RMF (OS/390) –Third-party offerings “Active” web site monitors (give a client-side view of the site) Database/Application monitoring Web server log files

Bringing It All Together So how do we monitor the web server? No one method can give you the whole picture on your web server’s health and performance OS Statistics Log File Analysis Active Site Monitoring Database Health

Bringing It All Together What’s in the Log Files? View of client-server “transactions” – client request, with the server response Multiple “transactions” can be required for a web page GET /parking/space.asp 404 File Not Found

Bringing It All Together What’s in the Log Files? Each “transaction” is totally separate in the log file Any “user-level” data must be manually grouped using criteria available in the particular log file

Bringing It All Together So what is in these log files?

Bringing It All Together Information in the log files Client IP - Usually the IP address, but can be resolved to DNS by the web server (not recommended) File requested by client (including directory) Method used in request (GET, POST, etc.)

Bringing It All Together Information in the log files Return Code - was it successful, and if not, why? Bytes Sent back to the client in the response Referring URL – where did the user find the link to this request? Browser String telling what browser is being used

Bringing It All Together Information in the log files Username - anonymous or authenticated access Cookie – The cookie relating to this “transaction”, if any Bytes Received by the server in the request Time Taken by the server to process the request

Bringing It All Together Standardized Log Formats Common Log Format (CLF) Extended Common Log Format W3C Standard Other formats may be product- specific, and many are extensions of the CLF or Extended CLF formats.

Bringing It All Together Common Log Format Advantages –Supported by just about every web server ever written Disadvantages –Inflexible –Contains very limited data: no Bytes Received, Time Taken, User agent (Browser), or Referer fields available.

Bringing It All Together Common Log Format [16/Feb/2001:06:59: ] "GET /cgi-bin/Count.cgi?df=gecbhome&dd=B HTTP/1.0" [16/Feb/2001:06:59: ] "GET /java/FixFontHeadline.class HTTP/1.0" [16/Feb/2001:06:59: ] "GET /graphics/trombone.gif HTTP/1.0" [16/Feb/2001:06:59: ] "GET /images/joinband.jpg HTTP/1.0" [16/Feb/2001:07:00: ] "GET /images/parade.jpg HTTP/1.0" [16/Feb/2001:10:20: ] "GET /schedule.shtml HTTP/1.0" [16/Feb/2001:10:26: ] "GET /index.shtml HTTP/1.0" [16/Feb/2001:10:21: ] "GET /about.shtml HTTP/1.0" [16/Feb/2001:10:26: ] "GET /communty.shtml HTTP/1.0" [16/Feb/2001:10:18: ] "GET /join.shtml HTTP/1.0" [16/Feb/2001:10:24: ] "GET /write.shtml HTTP/1.0" [16/Feb/2001:10:54: ] "GET /robots.txt HTTP/1.0"

Bringing It All Together Extended Common Log Format Adds User Agent (Browser) and Referrer to Common Log Format Advantages –Most web servers support it –More information available than CLF Disadvantages –Still no Time Taken or Bytes Received –Still inflexible

Bringing It All Together Extended Common Log Format [16/Feb/2001:06:59: ] "GET /cgi-bin/Count.cgi?df=gecbhome&dd=B HTTP/1.0" " "Mozilla/4.0 (compatible; MSIE 5.5; CS 2000; Windows 98)" [16/Feb/2001:06:59: ] "GET /java/FixFontHeadline.class HTTP/1.0" "-" "Java 1.1" [16/Feb/2001:06:59: ] "GET /graphics/trombone.gif HTTP/1.0" " "Mozilla/4.0 (compatible; MSIE 5.5; CS 2000; Windows 98)" [16/Feb/2001:06:59: ] "GET /images/joinband.jpg HTTP/1.0" " "Mozilla/4.0 (compatible; MSIE 5.5; CS 2000; Windows 98)" [16/Feb/2001:07:00: ] "GET /images/parade.jpg HTTP/1.0" " "Mozilla/4.0 (compatible; MSIE 5.5; CS 2000; Windows 98)" [16/Feb/2001:10:20: ] "GET /schedule.shtml HTTP/1.0" "-“ [16/Feb/2001:10:26: ] "GET /index.shtml HTTP/1.0" "-“ [16/Feb/2001:10:21: ] "GET /about.shtml HTTP/1.0" "-“ [16/Feb/2001:10:26: ] "GET /communty.shtml HTTP/1.0" "-“ [16/Feb/2001:10:18: ] "GET /join.shtml HTTP/1.0" "-“ [16/Feb/2001:10:24: ] "GET /write.shtml HTTP/1.0" "-“ [16/Feb/2001:10:54: ] "GET /robots.txt HTTP/1.0" “-” “-”

Bringing It All Together W3C Extended Log Format Advantages –Very Flexible –Extensible Disadvantages –Not as universally supported by web servers

Bringing It All Together W3C Extended Log Format #Software: Microsoft Internet Information Services 5.0 #Version: 1.0 #Date: :01:20 #Fields: date time c-ip cs-username s-ip cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-bytes time-taken cs-version cs-host cs(User-Agent) cs(Cookie) cs(Referer) :01: GET /Default.asp HTTP/1.1 entry.corp.com Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+95) SITESERVER=ID=547754cdab354b60fcd92cd e :01: GET /corporate.css HTTP/1.1 entry.corp.com Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+95) SITESERVER=ID=547754cdab354b60fcd92cd e;+ASPSESSIONIDGGQQGZEC=KEJNEBECDJLKONONHOOBBINF :01: GET /images/vDivider2.gif HTTP/1.1 entry.corp.com Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+95) SITESERVER=ID=547754cdab354b60fcd92cd e;+ASPSESSIONIDGGQQGZEC=KEJNEBECDJLKONONHOOBBINF :01: GET /images/toc_quicklink.gif HTTP/1.1 entry.corp.com Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+95) SITESERVER=ID=547754cdab354b60fcd92cd e;+ASPSESSIONIDGGQQGZEC=KEJNEBECDJLKONONHOOBBINF :01: GET /images/region_am.jpg HTTP/1.1 entry.corp.com Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+95) SITESERVER=ID=547754cdab354b60fcd92cd e;+ASPSESSIONIDGGQQGZEC=KEJNEBECDJLKONONHOOBBINF :01: GET /images/orange_square_bullet.gif HTTP/1.1 entry.corp.com Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+95) SITESERVER=ID=547754cdab354b60fcd92cd e;+ASPSESSIONIDGGQQGZEC=KEJNEBECDJLKONONHOOBBINF :01: GET /corpnews/images/org_pointer_2.gif HTTP/1.1 entry.corp.com Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+95) SITESERVER=ID=547754cdab354b60fcd92cd e;+ASPSESSIONIDGGQQGZEC=KEJNEBECDJLKONONHOOBBINF

Bringing It All Together Which Format(s) Does My Web Server Support Server Common Log FormatExtended CLF W3C Extended Log Format ApacheDefaultAvailable*No Microsoft IISAvailableNoDefault IBM HTTP Server (Websphere) (Based on Apache) DefaultAvailable*No iPlanet Web ServerDefaultAvailable*No Website Pro (Orielly) AvailableNoAvailable Lotus DominoDefaultAvailableNo

Bringing It All Together Which Format(s) Does My Web Server Support Server Common Log FormatExtended CLF W3C Extended Log Format AOLServerDefaultAvailableNo Zeus Web ServerDefaultAvailable*No XitamiAvailableDefaultNo I/Net Commerce Server/400 DefaultNo WebStar (Mac)AvailableNoAvailable Servertec Internet Server Available

Bringing It All Together Limitations -or- Why we can’t ignore other sources of information

Bringing It All Together Log File Limitations Not enough information to get the whole picture on the site’s performance and health –We need to correlate the log data with other sources. OS-level statistics (Performance Monitor, SMF, 3 rd party) “Active” web analysis (e.g. Keynote) Data on databases or other components of the site Web Server Clien t Internet Back End DB Web Site

Bringing It All Together Log File Limitations Not enough information to get the whole picture on the site’s performance and health –We need to correlate the log data with other sources. OS-level statistics (Performance Monitor, SMF, 3 rd party) “Active” web analysis (e.g. Keynote) Data on databases or other components of the site Web Server Clien t Internet Back End DB Web Site

Bringing It All Together Log File Limitations Not enough information to get the whole picture on the site’s performance and health –We need to correlate the log data with other sources. OS-level statistics (Performance Monitor, SMF, 3 rd party) “Active” web analysis (e.g. Keynote) Data on databases or other components of the site Web Server Clie nt Internet Back End DB Web Site

Bringing It All Together Log File Limitations Not enough information to get the whole picture on the site’s performance and health –We need to correlate the log data with other sources. OS-level statistics (Performance Monitor, SMF, 3 rd party) “Active” web analysis (e.g. Keynote) Data on databases or other components of the site Web Server Clien t Internet Back End DB Web Site

Bringing It All Together Log File Limitations Only when fit together with the other pieces do we get the complete picture of your total web site health. Web Server Clien t Internet Back End DB Web Site

Bringing It All Together Log File Limitations You may also have to deal with log file formats which don’t include all of the information that you would like. Bytes Received Time Taken Referrer User Agent Common Log Format

Bringing It All Together Issues With Log Files User or Session level statistics Caching Clustering What constitutes a “site”?

Bringing It All Together User or Session Level Statistics The server doesn’t give you statistics for the user (e.g. how long were they on the site?) You have to mine these yourself from the data available You will only be able to get approximations with this data, not exact figures

Bringing It All Together How do we group records for user-level statistics? Client’s IP address –Proxy Servers and firewalls with Network Address Translation (NAT) will make all users from behind the firewall look like one user –If the proxy or firewall has multiple IP addresses (or it is an array), multiple accesses of site from one user may look like multiple users

Bringing It All Together How do we group records for user-level statistics? Cookies –If the site assigns cookies to track users through the site, you can group the records based upon the cookie –Users who disable cookies on their browser mess this up –Not all log file formats include the cookie

Bringing It All Together How do we group records for user-level statistics? User name –Useful for intranet, but you must have the server disallow anonymous access –Impractical for most internet sites (except restricted access)

Bringing It All Together Caching Content from the web site may be cached outside of the web server The web server may not get notification of requests for content that are serviced by these caches The caches may be in Proxy Servers, Browsers, or elsewhere

Bringing It All Together Clustering Each server in a web cluster may maintain its own log file You have to combine the log files to get information relevant to the entire site One user accessing your site may get data from multiple servers You may still want information on each individual server, to verify that they are load-balancing properly

Bringing It All Together What constitutes a web site? You have to decide exactly what you want to call a site: –A load-balanced cluster –A single site running on a dedicated server –A single site on a server running multiple sites –A directory within a site on a server –Multiple servers which act as your web presence (home, support, e-commerce…)

Bringing It All Together What good is analyzing log files? OS-level analysis can’t: –Provide user (session)-level info –Break down by return code, file type or name, directory, etc.

Bringing It All Together What good is analyzing log files? “Active” monitoring: –Gives the client-side perspective –May not distinguish between a slow link/router and a slow response from server –Some are concerned only with response to the testing system, not server load –If a browser-based product, it may have troubles with browser incompatabilities

Bringing It All Together So what’s the key to analyzing log files? Grouping your log file records into useful statistics that will help you understand what is going on with your site

Bringing It All Together Example: 404 Errors When a user gets a 404 Error (File Not Found), they may perceive a lack of “professionalism” or “quality” with your site. You want to know not only what non- existent files are being requested, but why they are being requested (outdated link?)

Bringing It All Together Example: 404 Errors

Bringing It All Together Example: 404 Errors

Bringing It All Together Example: User Session Time You want to get as useful an approximation as is possible for how long users are staying at your site (at least, marketing will) Obviously, the longer they are browsing your site, the more interested they may be in what you have to offer You can use their first and last requests for files to get a rough approximation

Bringing It All Together Example: User Session Time Most sessions were very short (1-2 pages) This was an “Entry server” cluster, which passed off to other sites A few (<20% of total sessions) were very long

Bringing It All Together Example: Cluster Load- Balancing Ideally, your clustered servers for the site would be sharing the load equally If one server is carrying a larger load, it can lead to overall perceived slowdown of your site (most people going to a heavily loaded server while an idle server sits and does nothing)

Bringing It All Together Example: Cluster Load- Balancing

Bringing It All Together Example: Cluster Load- Balancing

Bringing It All Together So What Should I Take Out Of This? -or- Is there a point???

Bringing It All Together Summary Web server log file analysis is an important part of your monitoring of your web servers Log file analysis alone will not give you the complete picture of your web server, but you can’t get the complete picture without it Know what is useful in the log files, what limitations are inherent in them, and how to analyze them