1. SEGL HPC Workflow System
SEGL©
Speaker: Yuriy Yudin
Natalia Currle-Linde
Yevgen Dorozhko
Tatyana Krasikova
Yuriy Yudin

2. How to start a job? HPC resource input data transfer shell script
scp, sftp, rsync,...
transfer
shell script
PBS queue
qsub
submit
Waiting… qstat … qstat … … YES!

3. How to start a complex experiment?
Specific HPC knowledge +
Number of datasets

4. Description of the experiment
Is this workflow?
This is an example of using GriCoL
Control flow level
Data flow level

5. input, output and intermediate data
Data flow level
Dataspace stores
input, output and intermediate data
Dataspace “X”
Dataset 1
Dataset 2 …
Dataset n
file 1, file 2
file 1
file1,…, file 100
file1, file2, file3
file

6. Data flow level. Component wrapping
computational module
Implementation
for Resource 1
Implementation
for Resource n
Implementation
for Resource 2
SEGL database
Module library
All experiment
models

7. HPC Resources How does it work? HPC Organization SEGL Server Database
Agent
https
ssh
jxta
SEGL Server
ssh
https
HPC Resources
ssh
jxta
Agent
https
ssh
jxta
ssh
Database
Agent

8. SEGL Middleware

9. N > M Details PBS queue Limitation: max M jobs N jobs job 1 job 2…
job 1
job 2
job n
common job
Internal
System queue

10. It is very simple. No changes
SUDO command
Simple user account for
SEGL Agent
ROOT privileges
Special groups

11. Thank you for your attention.
SEGL©
Speaker: Yuriy Yudin
Natalia Currle-Linde
Yevgen Dorozhko
Tatyana Krasikova
Yuriy Yudin