1. VGrADS Execution System or “the Virtual Grid”
Andrew Chien, Henri Casanova, Yang-suk Kee, Richard Huang, Fran Berman
with input from the entire VGrADS Team
Department of Computer Science and Engineering
San Diego Supercomputing Center
Center for Networked Systems
UC San Diego
September 13-14, 2004
VGrADS Workshop
University of Tennessee, Knoxville

2. Meta-Goals for VGrADS Execution System Architecture
Leverage GrADS Learning
Provide Organizing Framework for VGrADS Execution System Efforts
Enable Integration of All of the Efforts
Provide Working Framework which
Enables Myriad Research Efforts and Experiments
Enables Leverage Across Different Teams
Leverages Globus Infrastructure
Support Application and PPS Research and Progress
We’re Leading and Organizing… not doing all of it!
VGrADS Workshop 9/13/2004

3. Focused Execution System Goals
Provide Simple Resource Abstraction for PPS
Enable Simpler and Better Optimization and Scheduling
Enable Application-driven Adaptation
Notify on Significant Changes in Resource Behavior
Scale Good Resource Selection and Binding to
Large and Dynamic Resource Pools
Dedicated Sharing, Competitive Sharing, Malicious Behavior
Present Dynamic Resource Information
Better Presentation/Organization
More Up-to-Date Information at Large Scale
VGrADS Workshop 9/13/2004

4. Virtual Grids as a Resource Abstraction
Application / PPS
Virtual Grid
Resource Classification/Characterization
Resource Properties
Resource Mgmt
Application: Resource Requirements, Comp and Scheduling Opt, Adaptation
Virtual Grid: Resource Abstraction, Information, Monitoring, Adaptation
Underlying Resource:
Characterization and Classification
Properties and Management
VGrADS Workshop 9/13/2004

5. Basic Virtual Grid Operations
Application Interfaces
VGManage()
Create a virtual grid, add resources, delete resources
VGReplaceResources()
Swap a specified set of resource for some currently in the system
Application reconfiguration must be dealt with by application
VG System Management
VGCreateBroker()
VGTerminateBroker()
Terminate Agent
VGrADS Workshop 9/13/2004

6. VG System Elements Client: Application
Broker: Forms Virtual Grids, Modifies them, Negotiates with Resource Managers
Monitor: Collects Dynamic VG Information, Notifies Application
Broker and Monitor are active parts of the Virtual Grid
There may be more in the future…
Brokers compete with each other on behalf of their application
VGrADS Workshop 9/13/2004

7. VG Evolution I: Acquire and Use
VGrADS Workshop 9/13/2004

8. VG Evolution II: Acquire, Adapt, App Reconfigure
VGrADS Workshop 9/13/2004

9. VG Evolution III: Acquire, Transparent Adaptation, FT?
VGrADS Workshop 9/13/2004

10. Refining Virtual Grid Concepts
Good start, but what do they mean?
How do they compose?
How to describe them, precisely…
VGrADS Workshop 9/13/2004

11. Virtual Grid Description Language
Vgrid ::= Identifier = Rdl-expression | Rdl-expression at time/event
Rdl-expression ::= Rdl-subexpression | ( Rdl-subexpression ) | Rdl-subexpression op Rdl-subexpression
Rdl-subexpression ::= Associator-expression | Node-expression
Associator-expression ::= Bag-of-expression | Cluster-of-expression
Bag-of-expression ::= LooseBagof "<" Identifier ">" "[" MinNode ":" MaxNode "]" [ "[" Number [ “su” | “sec” ] "]" ] ";" Node-expression |
TightBagof "<"Identifier ">" "[" MinNode ":" MaxNode "]" [ "[" Number [ “su” | “sec” ] "]" ] ";" Node-expression
Identifier ::= String
Min ::= Integer
Max ::= Integer
Node-expression ::= Identifier "=" Node-constraint
Node-constraint ::= "{" Attribute-constraint | Rdl-expression "}" | Rdl-expression
Attribute-constraint ::= (see Redline)
Cluster-of-expression ::= Clusterof "<" identifier ">" "[" MinNode ":" MaxNode [ “,” MaxTime “:” “MinTime”] "]" ";" Node-expression
op := close | far | highBW | lowBW
VGrADS Workshop 9/13/2004

12. Virtual Grid Description Concepts
Hierarchical Aggregators
ClusterOf (homogeneous and close)
TightBagOf (heterogeneous and close)
LooseBagOf (heterogeneous and far)
Composers
Close, Far, LowBW, HighBW
Resource Attributes (extensible)
Performance, Security, Owner, Pricing Policy, etc.
Advanced and Extent Reservation (start time, stop time)
Quantity of Resources (service units)
VGrADS Workshop 9/13/2004

13. Application Examples: GrADS
Fasta = LooseBagOf<Node3>[8:32] {
Node3 = [Software=DB] }
Scalapack =
ClusterOf<Node2>[8:32][3600 SEC] {
Node2 = [Processor=Pentium4]
VGrADS Workshop 9/13/2004

14. Application Examples: VGrADS
EOL1 = {
Node=[Software=DB,Memory=[1024:INF], Rank=ClockSpeed] }
EOL2=
{SNode=[Software=DB] }
far
LooseBagOf<CNode>[32:4096] {
CNode = [Memory=[1024:INF]]
… and more complex versions …
VGrADS Workshop 9/13/2004

15. Application Examples: VGrADS (cont.)
EMAN = LooseBagOf<Node4>[1:64] {
Node4 = ClusterOf<Node5>[8:32] {
Node5 = [Processor=Opteron, Rank=ClockSpeed] }
VGrADS Workshop 9/13/2004

16. Virtual Grid Interface = Annotated VG Resource Request
Rname: foo.ucsd.edu
Load: 0.6
Pred Load: 2.0 …
Request
Response &
Active Representation
Virtual Grid is an Explicit, Active Entity
Bound Resources and their relation to the Request
Information: Static and Dynamic
Expression Attributes (AST with Attributes)
Resources, Static Info (proc type, speed, location, etc.)
Dynamic Information (load, prediction, security state, failure state, NWS info, etc.)
Characterization / Classification Information
VGrADS Workshop 9/13/2004

17. Discussion
Virtual Grid as Explicit and Active entity enables explicit query, use, adaptation
Annotated VG Description provides convenient, structured presentation of attribute information
New types of Resource Information easily integrated in a pluggable attribute framework
Internal VG Expression Nodes are natural places for some aggregate properties
VGrADS Workshop 9/13/2004

18. Challenges in Realizing the Virtual Grid
Selection and Binding in Complex Resource Environments
Scaling, Quality of Results, Complex Application Requirements, Rich Resource Descriptions, Competitive Sharing
Scalable and Convenient Presentation of Resource Information
Interfaces and Ease of Use, Integrating Group/Interrelation Information, Scaling and Accuracy of Dynamic Information
Monitoring and Reacting
Customization to Resource Description, Practically Good Monitoring, Application-Aided and Automatic Adaptation Approaches
and Offline activities
Resource Classification
Resource Characterization
VGrADS Workshop 9/13/2004

19. An Abstract Virtual Grid Execution System Architecture
VG Creation, Management
Brokers
Applications
VG Monitoring
Trigger Adaptation
Monitors
Resource
Managers
Information
Services
Brokers Select, Bind, Implement Operations on VGrid
Monitors Watch Resources, Trigger on Changes, Notify Apps
Application Needs ==== Execution System Intelligence
(Resource Descriptions)
VGrADS Workshop 9/13/2004

20. An Initial Approach to Selection and Binding
Resource Broker
Simplify/Reduce Resource Description
Assemble a Surplus of Candidates
Bind Candidates to Form Virtual Grid
If Succeed
Return Virtual Grid
Discard excess Candidates
If Fail, many Strategies
Collect more Candidates Next Time (Aggression)
Application Change Resource Spec (less picky)
Return Partial Virtual Grid (application adapt) …
VGrADS Workshop 9/13/2004

21. VGrADS Execution System Prototype: Local Broker Architecture
Capture Resource Information from MDS/Generator
Local Resource Broker
DB Queries
DVC Launcher
Globus/GRAM Backend
Monitoring Not Implemented
VGrADS Workshop 9/13/2004

22. Execution System Prototype: Local Broker
PPS/Application
Resource Broker
VGrADS APIs
VG RB APIs
Specification
Parser/SQL Generator
(javacc)
SQL phrase
DVC JNI Interface
JDBC
(MySQL Connector/J)
Launch
DVC/Globus DB
(MySQL)
Resource
Information
(Generator)
VGrADS Workshop 9/13/2004

23. Resource Selection Procedure
Parse
Description
Parse
Description
Simplify
Parse tree
Optimize
Parse tree
Generate
SQL phrase
Generate
SQL phrase
Annotate
Description
Annotate
Description VG
ClusterOf
node
1, 100
Processor
Opteron VG
ClusterOf
cluster
1, 10
CluterOf
node
Processor
Opteron
SELECT ID FROM Cluster
WHERE CPUs BETWEEN 1 AND 100;
SELECT * FROM HOST
WHERE Cluster = ID AND Processor = “Opteron”;
VG=
ClusterOf[1:10]<cluster> {
Cluster[1:10]<node> {
node = [Processor=Opteron] }
VG:
Cluster0:
Node0: Opteron, 2.4Ghz, 1024 KB, 4096 MB, 200 GB,
Node1: Opteron, 2.4Ghz, 1024 KB, 4096 MB, 200 GB,
Node2: Opteron, 2.4Ghz, 1024 KB, 4096 MB, 200 GB,
Node3: Opteron, 2.4Ghz, 1024 KB, 4096 MB, 200 GB,
VGrADS Workshop 9/13/2004

24. Resource Description Simplification
Simplify Redundant Aggregators and Expensive Composers
Aggregator Simplification
ClusterOf<>[n1, n2] { ClusterOf<> [m1,m2] }
-> ClusterOf<>[n1*m1,n2*m2]
TightBagOf<>[n1,n2] { ClusterOf/TightBagOf<>[m1,m2] }
-> TightBagOf<>[n1*m1,n2*m2]
LooseBagOf<>[n1,n2] { ClusterOf/TightBagOf/ LooseBagOf<>[m1,m2]}
-> n1 instance of ClusterOf/TightBagOf/LooseBagOf<>[m1,m2]
Composer Simplification
Aggregator<>[n1,n2] close Aggregator<>[m1,m2]
->TightBagOf<>[n1+m1,n2+m2]
VGrADS Workshop 9/13/2004

25. Database Table Structure (Schema)
TightBag
TightBag
Cluster
Cluster
Cluster
Cluster
Cluster
Host
Host
Host
Host
Host
Host
Host
Host
Host
Host
Host
VGrADS Workshop 9/13/2004

26. Demo I: Description & Selection in a Synthetic Grid
VGrADS
PPS
VGrADS
Broker
EMAN, EOL, Iter
2. Resources
15,000 tight bags
50,000 clusters
1,000,000 hosts
Synthetic Grid
Resource Generator
VGrADS Workshop 9/13/2004

27. Demo II: Select, Bind, and Use in a Real Grid Testbed
VGrADS
PPS
4. Script output
2. Script launch
1. Description/
Script
3. Resources
DVC
VGrADS
Broker
Globus
UCSD Cluster
VGrADS Workshop 9/13/2004

28. Summary and Prototype Status
Designed Virtual Grid Description Language
Developed Concepts and Integration Architecture
Incorporates input from VGrADS team, supports breadth of research
Built and Demonstrated 1st Working Prototype
Resource Description language
Accepts full language, not all semantics implemented
Local Resource Broker
Many parameters to be explored; Dynamic Data, Resource Availability
Resource Binder
Utilizes DVC (Distributed Virtual Computer)
Utilizes Globus toolkit (GSI,GRAM) to bind resources
Prototype Functions End-to-End: Describe resources, request, bind, execute application
Change Modules, Evolve to Support Research
VGrADS Workshop 9/13/2004

29. More Virtual Grid Issues
Quantitative Access and Control
Coupling, Matching
Detailed Control
Complex Reasoning
Resource Properties
Confidence and Quality
Interaction w/ Resource Management/sharing
Selection vs. Selection&Binding
Advance and Extent Reservation
Quantity of Resource (speed*number)
Batch Queues and Variable Latency
Adaptation
Explicit vs. Transparent
Application Initiated
VG Monitoring Initiated
<Any other way> Initiated
Attributes
Security Properties
Statistical Properties
Fault Tolerance
Classification
Characterization
Aggregate Properties
VGrADS Workshop 9/13/2004

30. VGrADS Execution System Collaboration/Coordination
API’s/Services for presentation to PPS/Applications
Selection and Binding
Adaptation
API’s/Services for integration to information services
API’s/Services for integration/ties to Fault-Tolerance Work
Characterization, Prediction – Connect under Attribute System
VGrADS Workshop 9/13/2004

31. Core Virtual Grid Execution System Plans I
Resource Brokers, Algorithms and Implementations
Local Approach: Challenges
Scaling and Quality of Results
Evaluation on Synthetic Environments, Real Grid Environments (VGrADS)
Experiments with Kernels and Full-scale Applications
Local Approach: Schedule
First Prototype, UCSD Team (9/2004)
VGrADS Team Usable Prototype (1Q2005)
Distributed Approach: Challenges
Distributed Information and Accuracy
Distributed Decision Making and Quality of Results
Distributed Approach: Schedule
First Prototype, UCSD Team (??)
VGrADS Team Usable Prototype (??)
Experiments with Kernels and Full-scale Applications (??)
VGrADS Workshop 9/13/2004

32. Core Virtual Grid System Plans II
Virtual Grid Monitoring: Challenges
Per VGrid Description Generated Monitor
Implicit Specification of Performance Requirements
Experiments with Detection, Notification, and Robustness
Virtual Grid Monitoring: Schedule
First Prototype System (2Q2005)
VGrADS Team Usable Prototype (??)
VGrADS Workshop 9/13/2004

33. Additional Virtual Grid Activity
MicroGrid Modeling Tools (Andrew talk)
Large-scale Experiments
Increasingly Robust Operation
Enable Application Experiments with Virtual Grid
Scheduling (Henri C. and Richard H. talks)
Volatility and Information Quality
Scheduling for Virtual Grids
Application Driven Evaluation
EMAN, LEAD, EOL Kernels
Fault Tolerance (Dongarra), Reasoning Systems (Reed), Characterization (Wolski)
VGrADS Workshop 9/13/2004