1. Why is it important? A first cut at a logging interface
VO Logging
Why is it important?
A first cut at a logging interface
Maria Nieto-Santisteban
Ani Thakar
Wil O’Mullane
Alex Szalay
JHU

2. Maria Nieto-Santisteban, JHU
What is logged
Record every single user request in separate log DB
For multiple sites, harvest logs to one location
Data model for logs
Table of log sources
Typically two types of log tables
Web hits
Service requests (e.g. SQL queries)
May be more than one of these
Logging distributed services quite a challenge
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

3. Maria Nieto-Santisteban, JHU
Benefits of Logging
SDSS/SkyServer experience
Usage/traffic profiles great management resource
Invaluable for funding proposals and reviews
Are we meeting user requirements?
Monitor impact of press articles
Use data for load balancing
Find crawlers and “inconsiderate” users
Track server performance
Guide schema and interface design
Help prospective mirror sites to budget hardware
VO logging
Analyze VO usage per institution/archive/country
Answer question “Who’s using the VO?”
Design archive services for future archives like LSST
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

4. Maria Nieto-Santisteban, JHU
Log Harvesting at JHU
Logs from multiple sources harvested at JHU
SDSS and VO services at FermiLab and JHU
Will be extended to worldwide SDSS mirrors in future
Log server copies local Web and SQL logs
Web logs copied with xcopy, bcp-ed into DB
Local SQL logs collected with bulk inserts
Remote log harvesting via Web services
Currently using direct link to deliver FNAL logs
Future – special purpose Web service
Log server runs bi-hourly update
Reads everything > timestamp of previous update
All logs harvested into Weblog DB
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

5. Log harvesting diagram
JHU Log Server
SDSS DR2
JHU
CopyWeblogs
CopySqllogs
SDSS DR3
Webservers
WebLog DB
Weblog
Sqllog
LogSource
CasJobs
WebService
DR3 FNAL
FetchLogs
WebClient
FNAL
ReadWebLog
WebService
Webserver
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

6. Log Harvesting Data Model
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

7. Maria Nieto-Santisteban, JHU
VO Logging Issues
VO should define minimum log schema for harvested logs
Archives may log more data locally
Optionally make these available to VO
VO should specify WS interface for log harvesting
Automatically harvest logs from VO archive sites
Need to meet privacy concerns
Archives can control visibility of log data
Some info (e.g. clientIP) cannot be legally publicized
VO should at least collect aggregates
Web hits, total bytes in queries, volume of results etc.
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

8. Maria Nieto-Santisteban, JHU
VO Logging Interface
Part of IVOA Support Interfaces working draft
Web and Grid Services working group
Version 0.22 is up on IVOA twiki
Specifies two interfaces
HarvestWebLog
Takes from and to dates (ISO 8601) and output format
Serializes set of WebLogEntry (WebLogEntrySet) to file
Returns URL to file location
HarvestServiceLog
Serializes set of ServiceLogEntry (ServiceLogEntrySet) to file
Example interface/WSDL for SDSS SkyServer logs
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

9. Maria Nieto-Santisteban, JHU
WebLogEntry
Date – Date request submitted
ClientIP – IP Address request came from
ServerUrl* - in case server handles multiple websites
Method* - post,get
Request* - what’s after ? in url – perhaps blank.
Browser* - browser which submitted request
Status* - http status code(500, )
Bytes - number of bytes returned
Elapsed* - time to complete transaction in seconds
* = Required field
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

10. Maria Nieto-Santisteban, JHU
ServiceLogEntry
ClientIp* - Address request came from
Server* Server this ran on
ClientAccessLevel - Level of access
one of public (anonymous), private (logged in), internal
UserId - Can be null
RunId* - Passed in, otherwise generated
Request* - What was asked for
Event - What that means internally
QUERY, HARVEST, UPDATE, etc.
TimeIn* - Time request was received
TimeOut* - Time request was serviced
TimeActual* - Processor time waits removed
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

11. ServiceLogEntry (contd.)
Volume - Size in bytes of the response
Status* - 0=pending, 1=done, 2=failed
ErrorCode*- Numerical error code
Response *- Error message if this failed
perhaps the actual response if it is a small response, otherwise some summary statement or blank. Mainly to have errors.
IsVisible* - 0 or 1 for pub/priv, def. 1
* = Required field
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU

12. Maria Nieto-Santisteban, JHU
It’s a start, but ….
Interface will have to be modified
Return set is typically very large
Takes a long time to retrieve logs
Client needs to be notified when output is ready
Interface/service needs to be batch/asynchronous
Harvested log can be sent/saved to named VOStore
A realistic VO log harvester will need a dedicated logging VOStore
So logs can be harvested asynchronously
So archives can write their log data to it
Using put or importData
Need account delegation for authentication to work
IVOA Kyoto, June 12, 2019
Maria Nieto-Santisteban, JHU