Why is it important? A first cut at a logging interface VO Logging Why is it important? A first cut at a logging interface Maria Nieto-Santisteban Ani Thakar Wil O’Mullane Alex Szalay JHU

Maria Nieto-Santisteban, JHU What is logged Record every single user request in separate log DB For multiple sites, harvest logs to one location Data model for logs Table of log sources Typically two types of log tables Web hits Service requests (e.g. SQL queries) May be more than one of these Logging distributed services quite a challenge IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

Maria Nieto-Santisteban, JHU Benefits of Logging SDSS/SkyServer experience Usage/traffic profiles great management resource Invaluable for funding proposals and reviews Are we meeting user requirements? Monitor impact of press articles Use data for load balancing Find crawlers and “inconsiderate” users Track server performance Guide schema and interface design Help prospective mirror sites to budget hardware VO logging Analyze VO usage per institution/archive/country Answer question “Who’s using the VO?” Design archive services for future archives like LSST IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

Maria Nieto-Santisteban, JHU Log Harvesting at JHU Logs from multiple sources harvested at JHU SDSS and VO services at FermiLab and JHU Will be extended to worldwide SDSS mirrors in future Log server copies local Web and SQL logs Web logs copied with xcopy, bcp-ed into DB Local SQL logs collected with bulk inserts Remote log harvesting via Web services Currently using direct link to deliver FNAL logs Future – special purpose Web service Log server runs bi-hourly update Reads everything > timestamp of previous update All logs harvested into Weblog DB IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

Log harvesting diagram JHU Log Server SDSS DR2 JHU CopyWeblogs CopySqllogs SDSS DR3 Webservers WebLog DB Weblog Sqllog LogSource CasJobs WebService DR3 FNAL FetchLogs WebClient FNAL ReadWebLog WebService Webserver IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

Log Harvesting Data Model IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

Maria Nieto-Santisteban, JHU VO Logging Issues VO should define minimum log schema for harvested logs Archives may log more data locally Optionally make these available to VO VO should specify WS interface for log harvesting Automatically harvest logs from VO archive sites Need to meet privacy concerns Archives can control visibility of log data Some info (e.g. clientIP) cannot be legally publicized VO should at least collect aggregates Web hits, total bytes in queries, volume of results etc. IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

Maria Nieto-Santisteban, JHU VO Logging Interface Part of IVOA Support Interfaces working draft Web and Grid Services working group Version 0.22 is up on IVOA twiki Specifies two interfaces HarvestWebLog Takes from and to dates (ISO 8601) and output format Serializes set of WebLogEntry (WebLogEntrySet) to file Returns URL to file location HarvestServiceLog Serializes set of ServiceLogEntry (ServiceLogEntrySet) to file Example interface/WSDL for SDSS SkyServer logs http://skydev.pha.jhu.edu/SkyServerLog/SkyServerLog.asmx IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

Maria Nieto-Santisteban, JHU WebLogEntry Date – Date request submitted ClientIP – IP Address request came from ServerUrl* - in case server handles multiple websites Method* - post,get Request* - what’s after ? in url – perhaps blank. Browser* - browser which submitted request Status* - http status code(500, 404 ..) Bytes - number of bytes returned Elapsed* - time to complete transaction in seconds * = Required field IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

Maria Nieto-Santisteban, JHU ServiceLogEntry ClientIp* - Address request came from Server* - Server this ran on ClientAccessLevel - Level of access one of public (anonymous), private (logged in), internal UserId - Can be null RunId* - Passed in, otherwise generated Request* - What was asked for Event - What that means internally QUERY, HARVEST, UPDATE, etc. TimeIn* - Time request was received TimeOut* - Time request was serviced TimeActual* - Processor time waits removed IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

ServiceLogEntry (contd.) Volume - Size in bytes of the response Status* - 0=pending, 1=done, 2=failed ErrorCode*- Numerical error code Response *- Error message if this failed perhaps the actual response if it is a small response, otherwise some summary statement or blank. Mainly to have errors. IsVisible* - 0 or 1 for pub/priv, def. 1 * = Required field IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU

Maria Nieto-Santisteban, JHU It’s a start, but …. Interface will have to be modified Return set is typically very large Takes a long time to retrieve logs Client needs to be notified when output is ready Interface/service needs to be batch/asynchronous Harvested log can be sent/saved to named VOStore A realistic VO log harvester will need a dedicated logging VOStore So logs can be harvested asynchronously So archives can write their log data to it Using put or importData Need account delegation for authentication to work IVOA Kyoto, June 12, 2019 Maria Nieto-Santisteban, JHU