Load-Workflow in WS-VLAM Third Provenance Challenge Amsterdam, The Netherlands June

PC3 workflow The Workflow Background – PC3-workflow is part of the Pan-STARRS project which is building/operating thenext generation sky survey with the ability to continuously scan the visible skyonce a week and build a time series of data – This helps detect objects that may impact with earth besides building a massivecatalog of the solar system and 99% of visible stars in the northern hemisphere. – The load workflow used in PC3 appears at the handoff between the image pipeline and the ODM, and uses the Trident workbench to ingest incoming CSVfiles into SQL Server DBs load workflow in ws-vlam “ wsvlam-load-workflow” – Two version of PC3 workflow have been created in ws-vlam, with two differentgranularities. The proposed load workflow is decomposed into a number ofworkflow components exchanging data sets through the input/output ports – The wsvlam-load-workflow is grid-enabled in the sense that each workflowcomponents can be running on geographically distributed computing resources

PC3 workflow components (wsvlam modules) M1: ReadCSVFileReady – Parameter: JobID [String] – Outputs: Stream objects [CSV_File_ENTRY] M2: MatchCSVFileTable – Inputs – Outputs: DB_ENTRY M3:ReadCVSFileColumnName – Inputs: Stream objects [CSV_File_ENTRY] – Outputs: Stream objects [CVS_FILE_ENTRY_ColumnName] M4-M8: – Inputs: DB_ENTRY, Stream objects[CVS_FILE_ColumnName] – Outputs: Stream [true, false] M9: compact DB – Inputs: DB_ENTRY, Stream Strings [true/false] – Outputs:

M1: ReadCSVFileReady M2: MatchCSVFileTable M3:ReadCVSFileCol umnName M4-M8 M9:CompoactD B Stream- CSVFileEntry DB-ENTRY Stream of True, false DB-ENTRY PC3 workflow (Streaming mode) M1 –M9 are running concurrently The workflow components becomes active when data is available at their inputport(s) The workflow terminates when M1 has finished processing all the data

CP3 workflow farming mode (1) M1: ReadCSVFileReady M2: MatchCSVFileTable M3:ReadCVSFileCol umnName M4-M8 M9:CompoactD B List- CSVFileEntry DB-ENTRY CSVFileEntry- columnnames Boolean FileWriter FileReader Pre-Load workflow Load workflow (Farmed) Post-Load workflow The 3 phases of the workflow are executed sequentially/independently intermediate data is saved into files

CP3 workflow farming mode (2) Pre-Load workflow Load workflow Post-Load workflow Continue (run-workflow) Continue (run-workflow) Workflow description Continue (run-workflow) Continue (run-workflow) Workflow description Data locations The 3 phases of the workflow are executed independently CSV_FILE_ENTRY are processed in parallel

PC3 workflow sequential Mode Pre-Load workflow Load workflow Post-Load workflow Continue (run-workflow) Continue (run-workflow) Workflow description Continue (run-workflow) Continue (run-workflow) Workflow description Data locations Next - CSVFileEntry CSV_FILE_ENTRY 1/0 The 3 phases of the workflow are executed sequentially CSV_FILE_ENTRY are processed sequentially

Computing Nodes Workflow components Grid Middleware  GT4 WSRF Services - WS-VLAM engine - workflow component repository VLe Studio WS-VLAM composer VBrowser Semantic tools SAW: Semantic Annotation for Workflow CLAMP: Connecting LAnguage for Modules & Programs HAMMER: Hybrid-bAsed MatchMaker for e-Science Resources DAS-3 Matrix-Sara pc-vlab19.uva PoC environment RP environment Testbed SRB

Testbed User start VL-e Studio SSearch for services for the workflow UUpload in the workflow composer CCompose the workflow SSearch for input data set in the SRB SSet the workflow parameters EExecute the workflow MMonitor the execution DAS-3 Matrix-Sara pc-vlab19.uva SRB PoC environment RP environment