1. SC’07 Demo Draft
VGrADS Team
June 2007

2. Two vgES mechanisms to support FTR
vgLaunch
vgLaunch
LooseBagOf(cluster)
LooseBagOf(cluster)
ClusterOf
ClusterOf
Broadcast / overprovision
Restart / migration

3. FTR Mode - Step 1: Find Workflow Execution Manager FTR vgrid vgES
vgdl = LooseBagOf (cluster) [5] {
cluster = ClusterOf(node) [16] { node = [WRF == true] } }
vgrid
LooseBagOf(cluster)
vgES
ClusterOf

4. Find: Input Interfaces to FTR
vgrid annotated with following for VC’s
BQP
NWS
MDS
Reliability (if available)
Mapping from virtual to real cluster
for performance model
vgrid
LooseBagOf(cluster)
LB0
VC0
VC1
VC2
VC3
VC4
ClusterOf
VC: virtual cluster

5. Step 2: FTR decision + Bind
vgrid
LooseBagOf(cluster)
LB0
+ Annotations
VC0
VC1
VC2
VC3
VC4
ClusterOf
FTR
Bind on VC1, VC2 and VC3

6. Step 3: Execution in Broadcast Mode
vgLaunch
FTR
vgrid
LooseBagOf(cluster)
LB0
vgLaunch on vgrid
[broadcast on VC1, VC2 and VC3]
VC0
VC1
VC2
VC3
VC4
vgES
ClusterOf
run, status, cancel
Broadcast / overprovision

7. Step 3: Execution in Restart Mode
vgLaunch
FTR
vgrid
LooseBagOf(cluster)
LB0
vgLaunch on vgrid
[run on VC1; restart on VC2 on failure]
VC0
VC1
VC2
VC3
VC4
vgES
ClusterOf
Restart

8. Step 3: Execution in Migration Mode (Future)
vgLaunch
FTR
vgrid
LooseBagOf(cluster)
LB0
vgLaunch on vgrid
[migration path - VC1 to VC2 to VC3]
VC0
VC1
VC2
VC3
VC4
vgES
ClusterOf
Migration

9. Step 4: Monitor Execution
Broadcast case
run and monitor status for each copy of application
cancel remaining copies on success
application fails if all copies fail
Restart case
run and monitor status for one copy
call-back FTR with new vgrid (or pruned vgrid)
FTR decides another target vc and calls vgLaunch
repeat

10. Constantly collecting data over time
SC’07 Demo Flow
Resource
Broker
Constantly collecting data over time
Performance
Model
Batch
Queue
Prediction
If not reserved resource, ask - Is it time to submit?
(Reserved)
Query the performance model for task’s resource requirements
Execution System
Virtual Grid
DAG +
Constraint
Here is the workflow and constraints + pointer to performance model. Give me a mapping
Find me two slots
(vgFind)
GT4 GRAM
If reserved submit PBS-glidin at slot start time else submit when BQP suggests
(Reserved)
Annotated
DAG
Scheduler
Mapper
PBS
Return slots above threshold
Return mapping
Bind Resources
(vgBind)
Use performance model and map the tasks to the slots. If deadline can’t be met, return.
(Reserved)
Planning
Execution
Normal Mode
vgLaunch
Run Job
Slot
PBS
Globus
Gateway
FTR
(vgFind)
Run job
vg + annotations
FTR Mode
Run Job**
(vgBind)
vgLaunch**

11. Current Issues Key decision point Requirement Comment
Ryan’s scheduler plans for entire workflow before execution of any step of the workflow
FTR operates dynamically as every workflow step is executed
Requirement
Ability to execute on subset of VCs in a vgrid
Comment
Can’t determine “redundancy” without knowing the available vgrid and annotations

12. Demo and Beyond
Current demo scenario is “above the line” implementation
vgES provides the mechanisms and FTR provides the smarts
Longer term
Pushing FTR “below the line” (inside vgES)
Integrating reliability aspects during workflow planning
High-reliability bag

13. Milestones Testbed Nail down interfaces between vgES and FTR
List of machines – accounts, keys, certs
vgES and LEAD application installation
Test old vgES + scheduler + RB software stack
Nail down interfaces between vgES and FTR
Implementation of vgES mechanisms
Developer’s workshop
New vgES (with multiple submissions) release
vgES and FTR component test

14. Milestones Dummy FTR + new vgES test Test FTR working with vgrid
test calls from FTR to vgES
Test FTR working with vgrid
extracting vgrid and annotations via new interfaces
FTR and vgES code freeze
Test demo scenario for October workshop
scenario resulting in multiple submissions
October all-hands workshop
“glue” code freeze
SC’07 demo

15. Milestones
Dates, responsibilities and details TBD