“ A Distributed Computational Economy and the Nimrod-G Grid Resource Broker ”

A Typical Grid Computing Environment Grid Resource Broker Resource Broker Application Grid Information Service Grid Resource Broker database R2R2 R3R3 RNRN R1R1 R4R4 R5R5 R6R6 Grid Information Service

Need Grid tools for managing Security Resource Allocation & Scheduling Data locality Network Management System Management Resource Discovery Uniform Access Computational Economy Application Development Tools

mix-and-match Object-oriented Internet/partial-P2P Network enabled Solvers Market/Computational Economy

Building Grid Economy (Next Generation Grid Computing!) To enable the creation and promotion of: Grid Marketplace (competitive) ASP Service Oriented Computing... And let users focus on their own work (science, engineering, or commerce)!

Grid Node N GRACE: A Reference Grid Architecture for Computational Economy Grid Consumer Programming Environments Grid Resource Broker Grid Service Providers Grid Explorer Schedule Advisor Trade Manager Job Control Agent Deployment Agent Trade Server Resource Allocation Resource Reservation R1R1 Misc. services Information Service R2R2 RmRm … Pricing Algorithms Accounting Grid Node1 … Grid Middleware Services … … Health Monitor Grid Market Services JobExec Info ? Secure Trading QoS Storage Sign-on Grid Bank Applications

Grid Fabric Grid Apps. Grid Middleware Grid Tools Networked Resources across Organisations Computers Clusters Data Sources Scientific Instruments Storage Systems Local Resource Managers Operating Systems Queuing Systems TCP/IP & UDP … Libraries & App Kernels … Distributed Resources Coupling Services SecurityInformation … QoS Process Development Environments and Tools Languages Libraries Debuggers … Web tools Resource BrokersMonitoring Applications and Portals Prob. Solving Env. Scientific … Collaboration Engineering Web enabled Apps Resource Trading Grid Components Market Info

Economy Grid = Globus + GRACE Applications GRAM Globus Security Interface (GSI) Local Services LSF Condor GRDQBank PBS TCP SolarisIrixLinux UDP High-level Services and Tools CactusMPI-G Nimrod-G Broker CC++ GASSGTSGARA Grid Fabric Grid Apps. Grid Middleware Grid Tools GBank GMD eCash JVM DUROC Core Services ScienceEngineeringCommercePortalsActiveSheet … … … … MDS Higher Level Resource Aggregators Nimrod Parametric Language

Economic Models Price-based: Supply,demand,value, wealth of economic system Commodity Market Model Posted Price Model Bargaining Model Tendering (Contract Net) Model Auction Model English, first-price sealed-bid, second-price sealed-bid (Vickrey), and Dutch (consumer:low,high,rate; producer:high, low, rate) Proportional Resource Sharing Model Monopoly (one provider) and Oligopoly (few players) consumers may not have any influence on prices. Bartering Shareholder Model Partnership Model

Grid Open Trading Protocols Get Connected Call for Bid(DT) Reply to Bid (DT) Negotiate Deal(DT) Confirm Deal(DT, Y/N) …. Cancel Deal(DT) Change Deal(DT) Get Disconnected Trade Manager Trade Server Pricing Rules DT - Deal Template: - resource requirements (TM) - resource profile (TS) - price (any one can set) - status - change the above values - negotiation can continue - accept/decline - validity period API

Cost Model Without cost model any shared system becomes un-managable Charge users more for remote facilities than their own Choose cheaper resources before more expensive ones Cost units (G$) may be Dollars Shares in global facility Stored in bank

Cost Grid site X Non-uniform costing Encourages use of local resources first Real accounting system can control machine usage User 5 Machine 1 User 1 Machine 5 Resource Cost = Function (cpu, memory, disk, network, software, QoS, current demand, etc.) Simple: price based on peaktime, offpeak, discount when less demand,..

Nimrod/G : A Grid Resource Broker A resource broker for managing, steering, and executing task farming (parameter sweep/SPMD model) applications on Grid based on deadline and computational economy. Based on users ’ QoS requirements, our Broker dynamically leases services at runtime depending on their quality, cost, and availability. Key Features A single window to manage & control experiment Persistent and Programmable Task Farming Engine Resource Discovery Resource Trading Scheduling & Predications Generic Dispatcher & Grid Agents Transportation of data & results Steering & data management Accounting

Parametric Computing (What Users think of Nimrod Power) Multiple Runs Same Program Multiple Data Killer Application for the Grid! Parameters Courtesy: Anand Natrajan, University of Virginia Magic Engine

Sample P-Sweep/Task Farming Applications Bioinformatics: Drug Design / Protein Modelling Bioinformatics: Drug Design / Protein Modelling Sensitivity experiments on smog formation Combinatorial Optimization: Meta-heuristic parameter estimation Ecological Modelling: Control Strategies for Cattle Tick Electronic CAD: Field Programmable Gate Arrays Computer Graphics: Ray Tracing High Energy Physics: Searching for Rare Events Finance: Investment Risk Analysis VLSI Design: SPICE Simulations Aerospace: Wing Design Network Simulation Automobile: Crash Simulation Data Mining Civil Engineering: Building Design astrophysics

Build Distributed Application & Scheduler Build App case by case basis Complicated Construction E.g., AppLeS/MPI basedTotal Cost:$???

Nimrod-G Broker Automating Distributed Processing Compose, Submit, & Play!

Nimrod & Associated Family of Tools P-sweep App. Composition: Nimrod/Enfusion Resource Management and Scheduling: Nimrod-G Broker Design Optimisations: Nimrod-O App. Composition and Online Visualization: Active Sheets Grid Simulation in Java: GridSim Drug Design on Grid: Virtual Lab Remote Execution Server (on demand Nimrod Agent) File Transfer Server

A Glance at Nimrod-G Broker Grid Middleware Nimrod/G Client Grid Information Server(s) Schedule Advisor Trading Manager Nimrod/G Engine Grid Store Grid Explorer GE GIS TM TS RM & TS Grid Dispatcher RM: Local Resource Manager, TS: Trade Server Globus, Legion, Condor, etc. G G C L Globus enabled node. Legion enabled node. G L Condor enabled node. RM & TS CL See HPCAsia 2000 paper!

GlobusLegion Fabric Nimrod-G Broker Nimrod-G Clients P-Tools (GUI/Scripting) (parameter_modeling) Legacy Applications P2PGTS Farming Engine Dispatcher & Actuators Schedule Advisor Trading Manager Grid Explorer Customised Apps (Active Sheet) Monitoring and Steering Portals Algorithm1 AlgorithmN Middleware... ComputersStorageNetworksInstrumentsLocal Schedulers G-Bank... Agents Resources Programmable Entities Management JobsTasks... AgentSchedulerJobServer PC/WS/ClustersRadio TelescopeCondor/LL/NQS... Database Meta-Scheduler Nimrod/G Grid Broker Architecture Channels... Database CondorGMD IP hourglass! Condor-AGlobus-ALegion-AP2P-A

Nimrod/G Interactions Grid Info Server Process Server User Process File access File Server Grid Node Nimrod Agent Compute Node User Node Grid Dispatcher Grid Trade Server Grid Scheduler Local Resource Manager Nimrod-G Grid Broker Task Farming Engine Grid Tools And Applications Do this in 30 min. for $10?