DSpace Statistics Graham Triggs Head of Repository Systems, Symplectic

A Brief History

Statistics in DSpace 1.0

Statistics in DSpace 1.1 This slide is left intentionally blank

Statistics in DSpace 1.2 If I’m honest, this is just padding

Statistics in DSpace 1.3

Classic Statistics  Shows items archived, views, searches  Parses dspace.log  Renders flat HTML files  Uses two scripts which must be scheduled  Reports can be public, or admin only

Classic Statistics – Config  All configuration in [dspace]/config/dstat.cfg (overview and search exclusions)  Displays:  Overview  Archive breakdown (item types)  Items viewed  Actions (Deletion, Update, Create, etc.)  Logins  Searches (keywords)  Action names in [dspace]/config/dstat.map

Classic Statistics – Issues  dspace.log is primarily for debugging  May not log all information required  May log lots of unnecessary information  Size of log files  1 log line does not equal a single access  No filtering of spiders, robots, etc.  Log parsing may take some time  Slow to update stats

Fast Forward: DSpace 1.6

Solr Statistics  Available for JSP and XML Uis  Event logger writes to Apache Solr  Filters Spiders by IP address  Reports are searches of usage data  Reports can be public, or admin only

Solr Stats - What is Indexed  Time  Type (item, bitstream, etc), Id  Owning Community, Owning Collection, Owning Item  IP, Continent, Country, City, Longtitude / Latitude  Eperson Id, User Agent  Flag to indicate Robot / Spider

Solr Stats – Home  Top 10 items

Solr Stats – Community  Total visits  Visits last 7 months  Top 10 Countries  Top 10 Cities

Solr Stats – Collection  Total visits  Visits last 7 months  Top 10 Countries  Top 10 Cities

Solr Stats – Item  Total visits  Total file views  Visits last 7 months  Top 10 Countries  Top 10 Cities

Solr Stats – Config (v1.6)  [dspace]/config/dspace.cfg  solr.log.server  Location of Solr server / application  solr.dbfile  Location of Geo database  solr.spiderips.url  URLs to download IP addresses of search spiders  useProxies  Client identification when hosted behind proxy  solr.query.filter.spiderIp  Filter out spider IP addresses in query  solr.query.filter.isBot  Filter out ‘isBot’ field in query  statistics.item.authorization.admin  Set to ‘true’ to restrict to admins, false for public access

Solr Stats – Config (v1.7)  [dspace]/config/dspace.cfg  solr.log.server  solr.dbfile  solr.spiderips.url  useProxies  solr.query.filter.spiderIp  solr.query.filter.isBot  statistics.item.authorization.admin  solr.resolver.timeout  Timeout for the DNS resolver (lower for fewer connections)  solr.satatistics.logBots  Disable logging of events by spider IP addresses

Solr Stats – Config (v1.8)  [dspace]/config/modules/solr-statistics.cfg  server  spiderips.urls  dbfile  resolver.timeout  useProxies  logBots  query.filter.spiderIp  query.filter.isBot  authorization.admin  query.filter.bundles  Bundles for which to display file stats (requires 1.8 index)

Solr Stats – Improvements  Dspace v1.8  Displayed file bundle  Configurable - defaults to ORIGINAL bundle  [dspace]/bin/dspace stats-util –b –r  Dspace v1.7  Solr Optimization  [dspace]/bin/stats-util –o  Autocommit  Defaults to 15 minute intervals  Configurable in [dspace]/solr/statistics/colrconfig.xml  maxTime property

Solr Stats – Upgrade from Classic  Scripts parse dspace.log files to Solr entries  [dspace]/bin/dspace stats-log-converter  [dspace]/bin/dspace stats-log-importer  -I  Input file  -m  Adds a wildcard to the input (i.e. dspace.log*)  -s  Skip reverse DNS lookup (can be slow)  -v  Verbose output

Solr Stats –Custom Queries  You can expand the reports by querying the Solr index directly Example: Top downloads for a user – query on epersonid facet:

Solr Stats - Maintenance  [dspace]/bin/dspace stats-util –h usage: StatisticsClient -b,--reindex-bitstreams Reindex the bitstreams to ensure we have the bundle name -r,--remove-deleted-bitstreams While indexing the bundle names remove the statistics about deleted bitstreams -u,--update-spider-files Update Spider IP Files from internet into /dspace/config/spiders -f,--delete-spiders-by-flag Delete Spiders in Solr By isBot Flag -i,--delete-spiders-by-ip Delete Spiders in Solr By IP Address -m,--mark-spiders Update isBot Flag in Solr -h,--help help -o,--optimize Run maintenance on the SOLR index

Solr Stats - Issues  Privacy laws – IP addresses not anonymized  Performance issues / resource usage  Maintenance of Solr  Usage when Solr is unavailable  Usage tracking during periods of high usage

Summary  Classic Statistics  Possibly slow to analyse, fast to display  Delay in updating  Very imperfect  Solr Statistics  Updates ‘real time’  Can be slow to render as dataset grows  Improved in each release  Less imperfect