GriPhyN Status and Project Plan Mike Wilde Mathematics and Computer Science Division Argonne National Laboratory

Mike Wilde Planning Goals l Clarify our vision and direction u Know how we can make a difference in science and computing! l Map that vision to each experiment u Design concrete instances of our vision l Coordinate our research programs l Shape toolkit to challenge-problem needs l Coordinate overlapping technologies l Organize as coordinated subteams with specific missions and defined points of interaction

Argonne National LaboratoryMike Wilde Project Approach CS Research VDT Development Application Analysis Infrastructure Development and Deployment Challenge Problem Identification Challenge Problem Solution Development Challenge Problem Solution Integration VDT Development VDT Development Infrastructure Deployment IS Deployment time Process

Argonne National LaboratoryMike Wilde Project Activities l Research u Experiment Analysis l Use cases, statistics, distributions, data flow patterns, tools, data types, HIPO u Vision Refinement u Attacking the “hard problems” l Virtual data identification and manipulation l Advanced resource allocation and execution planning l Scaling this up to Petascale u Architectural Refinement l Toolkit Development l Integration u Identify and Address Challenge Problems u Testbed construction l Support l Evaluation

Argonne National LaboratoryMike Wilde Research Milestone Highlights Y1:Execution framework Virtual data prototypes Y2:Virtual data catalog w/glue language Integ w/ scalable replica catalog service Initial resource usage policy language Y3:Advanced planning, fault recovery Intelligent catalog Advanced policy languages Y4:Knowledge management and location Y5: Transparency and usability Scalability and manageability

Argonne National LaboratoryMike Wilde Research Leadership Centers l Virtual Data: u Chicago (VDC, VDL, KR), ISI (Schema) u Wisconsin (NeST), SDSC (MCAT,SRB) l Request Planning u ISI (algorithms), Chicago (policy), Berkeley (query optimization) l Request Execution u Wisconsin l Fault Tolerance u SDSC l Monitoring u Northwestern l User Interface u Indiana

Argonne National LaboratoryMike Wilde Project Status Overview l Year 1 research fruitful virtual data, planning, execution, integration– demonstrated at SC2001 l Research efforts launched 80% focused – 20% exploratory l VDT effort staffed and launched Yearly major release; VDT1 close; VDT2 planned; VDT3-5 envisioned l Year 2 experiment integrations high level plans done; detailed planning underway l Long term vision refined and unified

Argonne National LaboratoryMike Wilde Milestones: Architecture l Early 2002: u Specify interfaces for new GriPhyN functional modules l Request Planner l Virtual Data Catalog service l Monitoring service u Define how we will connect and integrate our solutions, e.g.: l Virtual data language l Multiple-catalog integration l DAGman graphs l Policy langauge l CAS interaction for policy lookup and enforcement l Year-end 2002: phased migration to a web- services based architecture

Argonne National LaboratoryMike Wilde Status: Virtual Data l Virtual Data u First version of a catalog structure built u Integration language “VDL” developed u Detailed transformation model designed l Replica location service at Chicago & ISI u highly scalable and fault tolerant u soft-state distributed architecture l NeSt at UW u Storage appliance for Condor u treats data transfer as a job step

Argonne National LaboratoryMike Wilde Milestones: Virtual Data l Year 2: u Local Virtual Data Catalog Structures (relational) u Catalog manipulation language (VDL) u Linkage to application metadata l Year 3: Handling multi-modal virtual data u Distributed virtual data catalogs (based on RLS) u advanced transformation signatures u Flat, objects, OODBs, relational u Cross-modal depdendency tracking l Year 4: Knowledge representation u ontologies; data generation paradigms u Fuzzy dependencies and data equivalence l Year 5: Finalize Scalability and Manageability

Argonne National LaboratoryMike Wilde Status: Planning and Execution l Planning and Execution u Major strides in execution environment made with Condor, CondorG, and DAGman u DAGs evolving as pervasive job specification model with the virtual data grid u Large-scale CMS production demonstarted on international wide-area multi-org grid u LIGO demonstrated full GriPhyN integration u Sophisticated policy language for grid-wide resource sharing under design at Chicago u Knowledge representation research underway at Chicago u Research in ClassAds explored in Globus context l Master/worker fault tolerance at UCSD u Design proposed to extend fault tolerance of Condor masters

Argonne National LaboratoryMike Wilde Milestones: Request Planning l Year 2: u Protype planner as a grid service module u Intial CAS and Policy Language Integration u Refinement of DAG language with data flow info l Year 3: u Policy enhancements: dynamic replanning (based on Grid monitoring), cost alternatives and optimizations l Year 4: u Global planning with policy constraints l Year 5: u Incremental global planning u Algorithms evaluated, tuned w/ large-scale simulations

Argonne National LaboratoryMike Wilde Milestones: Request Execution l Year 2: u Request Planning and Execution l Striving for increasingly greater resource leverage with increasing both power AND transparency l Fault tolerance – keeping it all running! u Intial CAS and Policy Language Integration u Refinement of DAG language with data flow info u Resource utiization monitoring to drive planner l Year 3: u Resource co-allocation with recovery u Fault tolerant execution engines l Year 4: u Execution adapts to grid resource availability changes l Year 5: u Simulation-based algorithm eval and tuning

Argonne National LaboratoryMike Wilde Status: Supporting Research l Joint PPDG-GriPhyN Monitoring group u Meeting regularly u Use-case development underway l Research into monitoring, measurement, profiling, and performance predication u Underway at NU and ANL l GRIPE facility for Grid-wide user and host certificate and login management l GRAPPA portal for end-user science access

Argonne National LaboratoryMike Wilde Status – Experiments l ATLAS u 8-site testgrid in place u Year-2 plan well refined l CMS u Working prototypes of production and distributed analysis, both with virtual data u Year-2 plan – simulation production – underway l LIGO u Working prototypes of full VDG demonstrated u Year-2 plan well refined and development underway l SDSS u Year-2 plan well refined u Challenge problem development underway; close collaboration with Chicago on VDC

Argonne National LaboratoryMike Wilde Year 2 Plan: ATLAS

Argonne National LaboratoryMike Wilde Year 2 Plan: CMS Plans

Argonne National LaboratoryMike Wilde Year 2 Plan: LIGO Plans

Argonne National LaboratoryMike Wilde Year 2 Plan: SDSS Plans