Widgets – Usage statistics collection Task force for the strategic project on the development and use of common ESS tools and services for dissemination 24/05/2013 Nicolas Schwartz, Atos

Using Access logs for statistics Access logs of server One log line per served HTTP request Main information Client IP Request URI HTTP Response Code User Agent technical name Monthly report automatically generated to provide global figures about Eurostat portal statistics Open source stack for “live” analytics (30 days history) Logstash / ElasticSearch / Kibana

Access log in detail Tab separated values #Fields list: date time time-taken c-ip x-ClientIP sc-status bytes x-AuthUser cs-method cs-uri x-UserAgent x-Referer x-Scheme x-Protocol x-AcceptLanguage Example of one line of logs 2013-05-15 08:41:19 0.019 147.67.136.102 88.191.143.226 200 13731 - GET /dsws/rest/data/v1.1/jsonp/en/t2020_10?_=1222 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31 - http HTTP/1.1 en-US,en;q=0.8,fr;q=0.6

Structured format {    "c_ip":"147.67.136.102",    "x_ClientIP":"194.165.48.90",    "sc_status":200,    "bytes":8396,    "cs_method":"GET",    "cs_uri":"/wbs/widget_template/js/vendor/json-stat0.2.1uncompressed.js",    "x_UserAgent":"Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0",     "x_Scheme":"http",    "x_Protocol":"HTTP/1.1" }

Adding GeoIP Localisation From xClient_IP to Location "geoip":{       "ip":"194.165.48.90",       "country_code2":"PL",       "country_name":"Poland",       "continent_code":"EU",       "city_name":"Warsaw", }

Parsing User Agent information Raw: x_UserAgent":"Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0 Processed "ua":{       "name":"Firefox",       "os":"Windows XP",       "device":"Other",       "major":"19",       "minor":"0"    }

Live Demo

Live Demo

More information http://en.wikipedia.org/wiki/Common_Log_Format http://logstash.net/ http://www.elasticsearch.org/ http://kibana.org/ http://www.maxmind.com/en/geolocation_landing https://github.com/tobie/ua-parser/blob/master/regexes.yaml

Widgets – Usage statistics collection Task force for the strategic project on the development and use of common ESS tools and services for dissemination 24/05/2013 Nicolas Schwartz, Atos