IVOA Interop, Beijing, China, May IVOA Data Access Layer Working Group Sessions Doug Tody (NRAO/NVO ) Markus Dolensky (ESO/EuroVO) Data Access Layer Working Group I NTERNATIONAL V IRTUAL O BSERVATORY A LLIANCE

IVOA Interop, Beijing, China, May Advanced Service Operations stageData

IVOA Interop, Beijing, China, May DAL Service Profile Standard Profile –Intended as starting point for any 2ndGen DAL service –Certain operations require GWS technology, or were motivated by efforts such as VOSI Standard Operations –queryData find available datasets (may be virtual) –(getData) get a single dataset (synchronous) –stageData start async job to generate/stage datasets –getCapabilities get service metadata (capabilities) –getAvailability get service availability, status

IVOA Interop, Beijing, China, May DAL Service Profile QueryData –Describes multiple datasets which can be retrieved In the future, a more complex query (e.g., multi-region) may describe many datasets –Data may of course be virtual data, requiring potentially significant computation to produce Hence a "dataset" represents a computational task A synchronous GET can be used to access data Large computations or batch mode requires stageData –Can include cost estimate information –Can iterate query to refine computational parameters

IVOA Interop, Beijing, China, May StageData Operation Concept –StageData is the "job submission" mechanism –Primary input is a list of datasets to be generated can be referred to by PubDID or acref job can be any size (thousands of concurrent tasks possible) –Includes information on data disposition (e.g., for VOSpace) How Might it Work? –Implemented as a POST –Response is status info and a JobID –All further communication based on service baseURL and JobID –Can use standard UWS job mechanism once job initiated –Could also be used to send commands to job once it is running

IVOA Interop, Beijing, China, May StageData Operation Monitoring Job Status –Two approaches Polling (a GET to some status URL based on baseURL, JobID) Event stream (a streaming GET used to carry event messages) –Summary Both are worth having Polling is simplest, should be mandatory Asynchronous messaging not difficult, much more powerful

IVOA Interop, Beijing, China, May Monitoring Job Status Polling –Simple, can support multiple clients –High latency, does not scale well, not terribly efficient –Can only see current status (no events, state transitions) Event Stream –Implemented as a streaming GET Current status transmitted at connect time, updates thereafter Status packet (e.g., %done) transmitted periodically –Still simple; can support multiple clients –Can disconnect and later reconnect –Any event can be reported to the client immediately –Low latency, efficient –Can add interactivity to any service

IVOA Interop, Beijing, China, May Monitoring Job Status Sample Events –Status (%done/keep-alive; cadenced) –Information (verbose/debug messages) Typical Job Events –jobStarted, jobCompleted, jobError –taskCompleted (data object finished and ready) –dataDelivered (can include acref)