1. Widgets – Usage statistics collection Task force for the strategic project on the development and use of common ESS tools and services for dissemination 24/05/2013
Nicolas Schwartz, Atos

2. Using Access logs for statistics
Access logs of server
One log line per served HTTP request
Main information
Client IP
Request URI
HTTP Response Code
User Agent technical name
Monthly report automatically generated to provide global figures about Eurostat portal statistics
Open source stack for “live” analytics (30 days history)
Logstash / ElasticSearch / Kibana

3. Access log in detail Tab separated values
#Fields list: date time time-taken c-ip x-ClientIP sc-status bytes x-AuthUser cs-method cs-uri x-UserAgent x-Referer x-Scheme x-Protocol x-AcceptLanguage
Example of one line of logs
:41: GET /dsws/rest/data/v1.1/jsonp/en/t2020_10?_=1222
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/ (KHTML, like Gecko) Chrome/ Safari/ http HTTP/ en-US,en;q=0.8,fr;q=0.6

4. Structured format
{    "c_ip":" ",    "x_ClientIP":" ",    "sc_status":200,    "bytes":8396,    "cs_method":"GET",    "cs_uri":"/wbs/widget_template/js/vendor/json-stat0.2.1uncompressed.js",    "x_UserAgent":"Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/  Firefox/19.0",     "x_Scheme":"http",    "x_Protocol":"HTTP/1.1" }

5. Adding GeoIP Localisation
From xClient_IP to Location
"geoip":{       "ip":" ",       "country_code2":"PL",       "country_name":"Poland",       "continent_code":"EU",       "city_name":"Warsaw", }

6. Parsing User Agent information
Raw: x_UserAgent":"Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/  Firefox/19.0
Processed
"ua":{       "name":"Firefox",       "os":"Windows XP",       "device":"Other",       "major":"19",       "minor":"0"    }

7. Live Demo

8. Live Demo

9. More information http://en.wikipedia.org/wiki/Common_Log_Format

10. Widgets – Usage statistics collection Task force for the strategic project on the development and use of common ESS tools and services for dissemination 24/05/2013
Nicolas Schwartz, Atos