Sphinx Server Sphinx Client Data Warehouse Submitter Generic Grid Site Monitoring Service Resource Message Interface Current Sphinx Client/Server Multi-threaded Control Process Planner DAG Reducer Information Gatherer Message Interface Prediction Engine

Sphinx Server Sphinx Client Data Warehouse Submitter Generic Grid Site Monitoring Service Resource Message Interface Sphinx Client/Server with V2 Components Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Message Interface

Sphinx Server Data Warehouse Stand alone Sphinx Server with light Client Sphinx Client Generic Grid Site Monitoring Service Resource Message Interface Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface

Sphinx Server Data Warehouse Message Tables Policy Information Tables Entity Accounting Tables Work (DAG, Job, etc) Tables State Unreduced Unpredicted Unaccepted (not admitted) Unplanned (not yet feasible) Unsent (assigned) Unfinished (executing) Remove (accounted or rejected) Entity Resource Property Requirements Dependency Requirements Do not start before “event” I/O QoS Requests Resource Assignment for next k steps Application Tables (in general, distributed…) Transformation Catalogue Profiles (CPU, Disk, Bandwidth requirements) History (CPU, Disk, Bandwidth, actual use) Data Tables (in general, distributed…) Replica Catalogue (from RLS) Profiles (Size, Bandwidth requirements) History (Access rate, etc) Resource Property Tables CPU, Disk, Network Connection, middleware configuration, etc Grid Weather Tables CPU View Storage View Bandwidth View Data Warehouse Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface What about including Virtual Data Tables?

Message Interface Component Input (incoming) DAG Requirements QoS requests Status requests Output (incoming) DAG Table (“unreduced”) Job Table (“unreduced”) application Requirements QoS Input (outgoing) Message Output (outgoing) Status updates Sphinx Server Data Warehouse Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface

DAG Reducer (Decomposer?) Input DAG (“unreduced”) Available (existing) data Action Remove DAG nodes for which input data exists Add Jobs to Job table Output Pruned DAG (“unpredicted”) Jobs (“unpredicted”) Sphinx Server Data Warehouse Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface

Sphinx Server Prediction Engine Component Input: “unpredicted” Job Requirements (if available) CPU hours Storage hours Bandwidth Profile/History Output: Estimated Resource Usage “unaccepted” job Data Warehouse Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface

Sphinx Server Admission Control Component Input Entity Account Information “unaccepted” DAG Request Requirements QoS Predictions of Jobs (decomposed DAG) Resource Usages Policy Constraints Current schedule for next k steps Output Reject DAG Request “Remove” DAG “Remove” Jobs Accept DAG with Estimated QoS “Unplanned” DAG “Unplanned” Jobs Data Warehouse Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface

Sphinx Server Planning Component Job Planner (independent “work” that can be fully resolved onto resources at time t) Input “unplanned” Job Information Requirements Dependencies (I/O) Job QoS Requests Entity Account Information Policy Constraints Available Data Available Applications Available Resource Properties Prediction of Job Resource Usage Output QoS Deliverable “unsent” Job(s) schedule for the next k steps DAG Planner (intra-dependent “work” that can not all be fully resolved onto resources at time t) Input “unplanned” DAG Information Requirements DAG QoS Requests Current Job Schedule for next k steps Output QoS Deliverable Modified Job Schedule for next k steps Data Warehouse Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface

Sphinx Server Job Submission Component (also part of Sphinx Client) Input Fully planned Job: Job state “unsent” Assigned gatekeeper and jobmanager I/O data movement locations Action Construct Job DAG Set up environment Import input data (if necessary) Publish to DMC (if necessary) Run application Export output data (if necessary) Publish to DMC (if necessary) Cleanup environment Local I/O data (if necessary) Submit to Condor-G/DAGMan Job state “unfinished” Monitor Job statistics Output Exit status (if available) Final Job statistics (resource usage) CPU Storage Bandwidth Entity account update Job state “remove” Data Warehouse Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface

Sphinx Server Data Management Components “User Demanded Movement” Module Input Source(s) Destination(s) Output DMC update “Pre-Planned Movement” Module (most important to Sphinx) Input Available Data Available Resources Prediction of Data Access Patterns DAG (“reduced”) Jobs (“unsent”) Job(s) schedule for next k steps Output Data Movement DMC update Prediction Module Input: Historical Data Access Patterns Output : Future Data Access Patterns Data Warehouse Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface

Sphinx Server Information Gathering Module Resource Property Module Input Grid Information Systems (such as MDS using GLUE Schema, etc) Output Resource Property views Job Tracking Module Input Job Monitoring information (such as Condor-G, etc) Output Work Views Grid Weather Module Input Grid Monitoring systems (such as GEMS, MonALISA, etc) Output Grid Weather Views CPU Storage Bandwidth Grid Weather Prediction Module Input: Grid Weather History Output: Grid Weather Forcast for next k steps Data Warehouse Multi-threaded Control Process Planner DAG Reducer Information Gatherer Prediction Engine Data Management Admission Control Submitter Message Interface