SDM Center A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, Terence Critchlow, Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas Mladen Vouk, Zengang Cheng, Lots of Others…. UCRL-PRES

SDM Center Web page is up and running  Example workflows available for download  Documentation exists and is improving  List of (somewhat dated) publications

SDM Center TSI-1 Workflow Web interface for Simulation configuration submission Error notification Identify config files Are files consistent Move files to correct locations Submit batch request at NERSC Check on job statusDelay Generate directory listing Are there new files? Identify complete files Job still running? Count files Empty list Transfer files to HPSS Transfer completed correctly Transfer files to SB Transfer completed correctly Delete file Extract Get Variables Remap coordinates Create Chem vars Create neutrino vars Derive other varsWrite diagnostic file Generate plots Tool-1 Tool-2 Tool-3 Tool-4 Generate thumbnails Generate movie Update notebook Update web page Update notebook Update web page Delay Update notebook Update web page Update notebook Update web page

SDM Center Delay TSI-1 Workflow Web interface for Simulation configuration submission Error notification Identify config files Are files consistent Move files to correct locations Submit batch request at NERSC Check on job statusDelay Generate directory listing Are there new files? Identify complete files Job still running? Count files Empty list Transfer files to HPSS Transfer completed correctly Transfer files to SB Transfer completed correctly Delete file Extract Get Variables Remap coordinates Create Chem vars Create neutrino vars Derive other varsWrite diagnostic file Generate plots Tool-1 Tool-2 Tool-3 Tool-4 Generate thumbnails Generate movie Update notebook Update web page Update notebook Update web page Update notebook Update web page Update notebook Update web page Remote job submission - done

SDM Center TSI-1 Workflow Web interface for Simulation configuration submission Error notification Identify config files Are files consistent Move files to correct locations Submit batch request at NERSC Check on job statusDelay Generate directory listing Are there new files? Identify complete files Job still running? Count files Empty list Transfer files to HPSS Transfer completed correctly Transfer files to SB Transfer completed correctly Delete file Extract Get Variables Remap coordinates Create Chem vars Create neutrino vars Derive other varsWrite diagnostic file Generate plots Tool-1 Tool-2 Tool-3 Tool-4 Generate thumbnails Generate movie Update notebook Update web page Update notebook Update web page Delay Update notebook Update web page Update notebook Update web page Real-time file transfer - done

SDM Center TSI-1 Workflow Web interface for Simulation configuration submission Error notification Identify config files Are files consistent Move files to correct locations Submit batch request at NERSC Check on job statusDelay Generate directory listing Are there new files? Identify complete files Job still running? Count files Empty list Transfer files to HPSS Transfer completed correctly Transfer files to SB Transfer completed correctly Delete file Extract Get Variables Remap coordinates Create Chem vars Create neutrino vars Derive other varsWrite diagnostic file Generate plots Tool-1 Tool-2 Tool-3 Tool-4 Generate thumbnails Generate movie Update notebook Update web page Update notebook Update web page Delay Update notebook Update web page Update notebook Update web page Data analysis – not yet started

SDM Center TSI-1 Workflow  Developed modular, configurable, actors that can be used elsewhere  File filtering Use regular expressions to filter file list  File transfer User specified command transfers files between two remote locations  SSH Executes remote command, catches errors  Retry Repeat an action until successful or until user defined limit is reached

SDM Center Top level view of TSI workflow

SDM Center Submit actor Submit job using llsubmit Return the job number Debug actor lets you test without actually submitting job

SDM Center PIW Workflow  Added logging  Modularized workflow a bit more  Added debugging and testing code  New visualization actor

SDM Center Next Steps  Working through deadlock / race condition problems in Ptolemy PN director  Discovered because of test suite / debug actor  Need to be able to “merge” multiple streams  Data analysis section of TSI-1 workflow  More documentation

SDM Center Next talks  Steve Parker  SCIRun and SPA integration status  Mladen Vouk  On Large Data Flow Scientific Workflows: An Astropysics Case Study  Bertram Ludaescher  Kepler / SPA Extensions for Scientific Workflows – Now and Upcoming