1. Alan Darnell & Sue McGillivray 1 Feb 2006
Measure Twice, Cut Once : What Can we Learn from Scholars Portal Statistics?
Alan Darnell & Sue McGillivray
1 Feb 2006

2. What’s available now? E-Journals SP Search SFX RefWorks RACER
Standard reports 
Custom reports
Database tables
Custom logs
Standard Web logs

3. Scholars Portal Search
Standard Reports
Scholars Portal Search
RefWorks
SFX / Get It!

4. SP Search Reports
Online reporting tool based on Clareos Crosscut
Consolidated usage data for CSA and OCUL databases accessed on any Illumina server
Consortia account provides aggregate reports of activity of all OCUL schools
Reports include logins, searches, queries, downloads, records examined and captured, and linking options per database
Aggregated by month
12 months online; 3 year archive

7. Multi-DB Search Trend

8. Refworks
Online reporting tool accessed through the Refworks “admin tool”
Consortia “admin tool” provides aggregate reports of all Refworks groupcodes
Reports include accounts created, accounts accessed (return users), number of sessions, number of citations added
Grouped by groupcode and by month
All data kept online -- represents 14 months worth of data

12. SFX - Get It!
Extensive set of reports provided as part of the SFX Administrator tool
None of the reports aggregate data for all instances -- i.e. no consortia reports
Usage data is captured in “online” tables and cannot be reported on until it is moved “offline”
Data can be moved from “offline” tables to an “archive”
Resource data is contained in the “knowledge base”

13. Definitions Request Clickthrough Target Service Object
occurs when a user invokes an SFX Menu by clicking on an OpenURL link
Clickthrough
occurs when a user clicks on a link in an SFX Menu (so one Request can have many Clickthroughs)
Target
place the user is sent to when he/she clicks on a link in an SFX Menu
Service
kind of information provided by a target (e.g. full-text, abstract, table of contents, document delivery)
Object
The resource associated with a Service (e.g. journal, book)

14. SFX Reports Transactions Number of requests and clickthroughs
Number of requests resulting in menus with certain services (e.g. full-text)
Number of clickthroughs to specific targets and to services offered by targets
Most often accessed objects (i.e. popular journals)
Objects never accessed (i.e. unpopular journals)
Requested objects that cannot be associated with a full-text service of any target (un-met demand)

15. SFX Reports

17. SFX Reports Collections
Objects that are associated with more than one target (e.g. journals offered by more than one vendor)
Comparing the overlap in objects between two particular targets

18. SFX Reports

20. What’s available now? E-Journals SP Search SFX RefWorks RACER
Standard reports 
Custom reports
Database tables
Custom logs
Standard Web logs

21. E-Journals (ScienceServer)
Custom Reports
E-Journals (ScienceServer)
RACER
List of reports available at:

22. Downloads

23. Download Geography

24. RACER Reports Reports Working Group Net-Lending Report Invoices
Performance Measures
Collection Reports

31. Sample Netting Out Report
Sample Netting Out Report. Contact Your ILL Department Manager for more information

32. Sample Invoice Report. Contact Your ILL Department Manager for more information

37. What’s available now? E-Journals SP Search SFX RefWorks RACER
Standard reports 
Custom reports
Database tables
Custom logs
Standard Web logs

38. Database Tables
An SQL database is comprised of “tables” (think Excel spreadsheets)
Tables are related through common “keys”
Keys allow programmers to connect data from different tables and construct particular views of the data (as reports or “virtual tables”)
This allows for “ad hoc” querying of tables -- exploring relationships between data not defined in a “standard report”
SQL Query Language defines syntax for querying tables

39. Different SQL Databases
SFX
mySQL
Each instance is represented in a separate database
Refworks
MS SQL Server
Each groupcode is represented in a separate database
RACER
Oracle

40. SQL Tables

41. What’s available now? E-Journals SP Search SFX RefWorks RACER
Standard reports 
Custom reports
Database tables
Custom logs
Standard Web logs

42. Custom Logs Web servers record requests and responses to log files
Search engines like ScienceServer and Illumina use the same approach to record higher-level transactions (e.g. searches, downloads)
This data is fed into reporting tools to aggregate the data
But there is often detail in these log files not represented in the aggregated data
The high level of detail makes it difficult to keep this data online for ad hoc querying

43. Illumina Usage Log Time stamp [31/Jan/2006:21:41:41+EST] IP Address
Username
uot
Password
uot046
Mode
Advanced,Quick,Expert
Function
QUERY | RESULTS | RECORD | TOC | CITERS | ABSTRACT | FULLTEXT | PERSONAL | THESAURUS | BIX | LINK | SCHOLAR | SIGNON
Extra Data
e.g. databases, query, hits

44. What’s available now? E-Journals SP Search SFX RefWorks RACER
Standard reports 
Custom reports
Database tables
Custom logs
Standard Web logs

45. Standard Web Logs
Web servers like Apache and IIS record a transaction record for every request to and response from the server
These logs represent the highest level of detail captured in our systems (file level detal)
Log files are rotated daily or weekly
The volume of transactions makes logs difficult to use for ad hoc queries
Invaluable resources for examining “user paths”

46. A Big Picture Anonymous Self Identified Authenticated Profiled
Articles
Journals
Books
Indexes
Subject Categories
Users
Resources
Services
Uses
Cite
Browse
Search
Request
Time
Duration
Quantity
Surveyed Reasons
for Use

47. Considerations Can we accurately map data elements between services?
What is our policy respecting the privacy of user data?
How much data should we keep online?
How long do we need to keep the data available?
How can we verify the accuracy of the data?
What data can we share with each other?
What data can we share with the world?
What data should we be capturing but are not?
How can we capture user intention and not just behaviour?

48. Creating Value
Collection Development
Identifying Predictors of Use
Understanding the User Experience

49. Collection Development
Using Scholars Portal statistics and data from vendors, circulation systems
Overlap reports - full-text and A&I databases
Subject clusters
Title utilization within a package
Backfile usage
Article utilization
Viewed articles, stored articles
Unfilled requests, requests without full text

50. Comparing Schools
Scholars Portal presents us comparative data from 20 different schools
Within those schools, there are significant variations in use of each service that can’t be explained in terms of FTE differences

51. Trent’s Share of Downloads

52. FTE Percentages

53. Predictors
By profiling each school in terms of instruction offered, reference hours, collection policies, teaching practices, etc.. we may find factors that explain and predict differences in use

54. User Experience User Paths Queries Within a service Across services
Syntax
Common Errors
Zero Hit Result Sets
Refinement Strategies
Selection Behaviours

55. Usage Data - 11/30 Hits 62,768,583 Searches 18,883
Ave. Records / Result Set
3324
Zero Hit Result Sets
5,010
Results Sets < 1000 Hits
16,940
Advanced
13,157
Quick
5,522
Tools
199