1. SIUG Annual Meeting 2010 UNC Charlotte January 28, 2010 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Web Logs: Finally! Now What Do We Do With Them?

2. SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington What are web logs? What information do they gather? Where are the logs stored? How are the logs accessed and analyzed? What do the reports mean? What limitations exist?

3. Useful Web Sites CSDirect: http://csdirect.iii.com/documentation/weblogs.shtml http://csdirect.iii.com/documentation/weblogs.shtml WebPAC Wiki: http://csdirect.iii.com/lswiki/WebPAC/WebHome Web Access Logs topic: http://csdirect.iii.com/lswiki/WebPAC/AccessLogs IUG Listserv: http://innovativeusers.orghttp://innovativeusers.org Search term: web server logs SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

4. What is a web log file? SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

5. What’s in a log file line? 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_pa ge/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

6. What’s in a log file line? Port Number 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_pa ge/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

7. What’s in a log file line? User’s IP Address 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_pa ge/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

8. What’s in a log file line? Date/Time 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_pa ge/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

9. What’s in a log file line? Page requested 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_pa ge/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

10. What’s in a log file line? HTTP status code 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_pa ge/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

11. What’s in a log file line? Bytes transferred 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_pa ge/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

12. What’s in a log file line? Referring page 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_ page/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

13. What’s in a log file line? Browser info 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_pa ge/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

14. What’s in a log file line? Load time 80 152.20.226.60 - - [20/Jan/2010:19:15:01 - 0500] "GET /search~b001o001c001i001 HTTP/1.1" 200 4763 "http://library.uncw.edu/web/systems/start_pa ge/start_page2.htm" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727)" 5970 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

15. Here’s another. 443 152.21.32.14 - - [20/Jan/2010:19:01:37 - 0500] "GET /patroninfo~b01o01c01i01/ HTTP/1.1" 200 2043 "http://www.uncp.edu/library/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;.NET CLR 1.1.4322;.NET CLR 2.0.50727; InfoPath.2;.NET CLR 3.0.4506.2152;.NET CLR 3.5.30729)" 22934 SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

16. Web Logs are configurable (sort of): Specify where logs are stored. Specify storage schedule. Specify what information is collected. SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

17. Only Innovative can modify the Apache config. Rolling 35-day set of log files. Each log = 24 hours (not midnight to midnight). 2-day delay in log availability. More on Innovative’s implementation later. SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

18. So where are the Millennium server web logs and how do we get at them? SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

20. Web Server Logs – live/logs (livelogs) SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

25. Analysis Tools SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Analog: http://www.analog.cx/ http://www.analog.cx/ Webalyzer: ftp://ftp.mrunix.net/pub/webalizer/old/ Then select webalizer-2.01-10-win32-bin.zip ftp://ftp.mrunix.net/pub/webalizer/old/webalizer-2.01-10-win32-bin.zip WebLog Expert Lite: http://csdirect.iii.com/lswiki/WebPAC/Weblo gExpertLite http://csdirect.iii.com/lswiki/WebPAC/Weblo gExpertLite

26. Web Log Format SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Innovative server produces Apache Combined format From the WebPAC Wiki: %p %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" %D From Alan Dyck (April 2009). ‘%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D’ From my Analog configuration file. I substituted the parentheses for single quotes. (%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D)

27. Setting up analyzers SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Analog: Download/install software Readme file and manual Edit configuration file Run from a command line

28. Analog config file: SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

30. Analog Report SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington General Summary Daily Summary Hourly Summary Domain Report Organization Report Search Word Report Browser Summary Operating System Report Status Code Report File Size Report File Type Report Directory report Request Report Report written to specified output directory

31. Webalyzer Setup SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

32. Webalyzer Report SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Monthly Statistics Daily Statistics Hourly Statistics Top URLs, entry pages, exit pages, referrers,Search strings, User agents, Country codes More difficult than Analog to configure Runs a single log file, so cron job or combine logs Report written to same directory as logs

33. WebLog Expert Setup SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Download/install software Edit config files? Multiple config files Requires standard or professional edition Edit log files to remove port number.

34. SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Process web logs for WebLog Expert Uncompress log files into a new directory Edit logs to remove the port number from entries Can be done using WordPad’s Find & Replace? Maybe Use EditPlus’s SEARCH with Regular Expression ^[0-9]* Remove any.bak files from the directory.

35. Title SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

36. Title SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

37. WebLog Expert Lite Report SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Summary – hits, page views views, visitors, bandwidth Activity Access Visitors Referrers Browsers Errors Standard and Professional versions are much more robust and configurable. Both are reasonably priced. Report written to browser, must be saved to location of choice

38. What can we do with these numbers? SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington

39. Possible uses: SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Counts – justify existence, services, programs Activity Times – Help determine Global/Rapid Updates, MARC record loads, upgrade days/times, compiling lists, processing inventories, etc. Track effects of system/program changes Know popular entry pages. Where to place alerts. Troubleshoot error messages. Web development, testing pages, ensuring functionality. Search patterns.

40. Limitations SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Can only analyze the data collected. Not all browsers provide information. Time/ability to configure analyzers. Data and report storage. Knowing what you want/need to analyze

41. Summary SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington Millennium web access logs are now available for download and manipulation. Log files contain information such as browser type, Operating System, entry page, user IP address, etc. Programs such as Analog, Webalyzer, and WebLog Expert Lite are available to help analyze masses of data. Analysis can help in decision-making and reporting. Web log analysis has limitations.

42. Thank You! SIUG Annual Meeting 2010 Web Logs: Finally! Now What Do We Do With Them? Dan Pfohl, UNC Wilmington