Claudio Grandi INFN Bologna CSN1 - Perugia 11/11/2002 Gli esperimenti LHC hanno qualcosa in comune? (HEPCAL RTAG di LCG) C. Grandi INFN - Bologna

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 2 What is HEPCAL OS & Net services Bag of Services (GLOBUS, Codor-G,…) DataGRID middleware PPDG, GriPhyn, EU-DataGRID ALICEATLASCMSLHCbOtherHEPOther Apps HEP Common Application Layer VO common application layer … ALICEATLASCMSLHCb Specific application layer OtherHEPOther Apps

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 3 How to proceed CMSATLAS ALICELHCb CMS ATLAS ALICELHCb Core common Use Cases

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 4 Physicist domain Computer Scientist domain Domains interface Why Use Cases? OS & Net services Bag of Services (GLOBUS, Codor-G,…) DataGRID middleware PPDG, GriPhyn, EU-DataGRID HEP Common Application Layer… ALICEATLASCMSLHCbOtherHEPOther Apps

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 5 Domains interface Definition of the applications that analyze the data is in the physicist domain Definition of the tools for accessing the data and the resources in a transparent way is in the computing scientist domain An interface is needed to make physicists and computing scientists to collaborate! Use a syntax taken from the CS domain: Use cases

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 6 Use cases What Use Cases are: –a standard technique for gathering requirements in software development methodologies –narrative documents that describe the sequence of events of an actor using a system [...] to complete processes (*) What Use Cases are NOT: –the description of an architecture –the representation of an implementation (*) Jacobson, I., et al. Object-Oriented Software Engineering Addison Wesley. Reading, MA.

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 7 LCG HEPCAL RTAG Mandate Identify use cases for distributed (Grid) computing common to the LHC experiments Focus on goals Try to be implementation independent Two months time: First meeting on April 3 rd, delivered on May 24 th About 10 full days of meetings

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 8 HEPCAL RTAG membership Chair: Federico Carminati Members: Piergiorgio Cerello (ALICE) Oxana Smirnova (ATLAS) Claudio Grandi (CMS) Eric VanErvijnen (LHCb) Jeff Templon (DataGrid WP8)

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 9 RTAG Report

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 10 Datasets Collection of files treated as a whole Read-only once uploaded to the grid Identified by a unique Logical Dataset Name May be replicated in many physical locations May be a Virtual Dataset the algorithm to produce it is registered to the Grid along with the input data and/or parameters May contain references to objects in other datasets May be associated with a default remote access protocol

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 11 Catalogues The only read/write entities on the grid May be defined by applications but: Implementation not under application control Replication not under application control Read/write datasets not discussed! Examples of grid-defined catalogues Dataset Metadata Catalogue Associates to each Logical Dataset Name a list of attributes ( key=value pairs) of the dataset Job Catalogue Associates to each Job Identifier a list of attributes

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 12 Jobs A single invocation of job submission May be composite (e.g. Direct Acyclic Graph) May be split Official productions are a special case of job submission Interactive jobs not discussed! Actually we discussed a lot on interactivity but we decided to leave it out because of time…

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 13 Job splitting Automatic splitting: Based on location of replicas of input datasets Using applications plug-in for splitting and for joining of the results. If interactivity is supported, splitting may be done by the running application (spawning processes on the grid, a-la PROOF) Job splitting not discussed in detail!

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 14 Persistency layer Grid tools will be used to navigate from one dataset to objects in other datasets The application persistency layer provides a mapping between the target object identifier and the Logical Dataset Name The Grid provides the mapping between the LDN and the physical copy of the dataset files

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 15 Identifying datasets The Dataset Metadata Catalogue maps attributes to Logical Dataset Names The input dataset of a job may be specified as a query that acts on the dataset attributes, e.g.: “give me all the datasets corresponding to events acquired during the period 22/11/2007 through 18/07/2008 using the XYZ trigger configuration” Applications may add attributes to datasets Special fields of the Dataset Metadata Catalogue may be used for virtual dataset materialization: Executable=…; StdIn=…; etc…

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 16 Identifying jobs The Job Catalogue maps attributes to Job Identifiers A query to the job scheduler may be specified as a query that acts on the job attributes, e.g.: “give me the status of all the jobs analyzing dataset XYZ using the application program version 1.2.3” Applications may add attributes to jobs Information used both for monitoring and for book- keeping (a-la BOSS)

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 17 Use cases Obtain Grid authorisation Ask for revocation of Grid authorisation Grid login Browse Grid resources DS metadata update DS metadata access Dataset registration to the Grid Virtual dataset declaration Virtual dataset materialization Dataset upload User-defined catalogue creation Data set access Dataset transfer to non-Grid storage Dataset replica upload to the Grid Data set access cost evaluation Data set replication Physical data set instance deletion Data set deletion (complete) User defined catalogue deletion (complete) Data retrieval from remote Datasets Data set verification Data set browsing Browse condition database Job catalogue update Job catalogue query Job submission Job Output Access or Retrieval Error Recovery for Aborted or Failing Production Jobs Job Control Steer job submission Job resource estimation Job environment modification Job splitting Production job Analysis 1 Data set transformation Job monitoring Simulation Job Experiment software development for the Grid VO wide resource reservation VO wide resource allocation to users Condition publishing Software publishing

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 18 Example A user needs to submit a job that analyzes a dataset and produces a file to be saved on the grid for further analysis Input dataset Input dataset Output file Output file Job Input dataset Input dataset Input dataset Input dataset

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 19 User operations sequence User logs into the grid User writes the job description –executable and arguments –logical name of input dataset or complex query –name and metadata of output file to be saved –... User submits the job Job output is made available to the user, including the logical file name of the output User uses the logical file name to access it

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 20 Running job operations sequence Information is made available to the job: –The physical file names for the input datasets The physical file names are either the names of a local files to be POSIX-opened (e.g. if they have been copied locally) or names that are accessible remotely via some protocol (e.g. AMS) The job is run The output file is uploaded to the grid –The output file is registered to the grid catalogues The user-defined metadata of the output are stored in the dataset metadata catalogue

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 21 Formal representation

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 22 Joint EU-US HEPCAL response Draft available on nov 1 st. Requested: quantitative performance metrics more details on VO management more details on devious flows and error handling use cases for site administrators more details on software publishing and versioning Among other suggestions: mantain ongoing collaboration among LCG, EDG, US grid projects and RTAG groups! …this means having a permanent body with the same competence of the HEPCAL RTAG…

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 23 Use CaseEDG 1.2 VDT EDG 2 planned VDT Spring 2003 Other US gridware Compatibilit y between VDT and EDG After EDG 2 VDT later plans Comments AAA and VO Obtain Grid Authorizatio n Yes Work with DOESG Revoke Grid Authorizatio n Yes Grid LoginPartial Roles, expiration issues Yes Neither Extensions nor Additional Requirements supported by both Browse Grid Resources Basic Yes Meta-Data, Data Mgmt. and Access DS Metadata Update Yes?Basic Yes - [1]Basic EDG: User must do all the work DS Metadata Access Yes?Basic YesYes – [1]Basic EDG: User must do all the work! EDG: Primitive (J.T.) Dataset (DS) Registration Yes?Basic Yes?Yes – [1]Basic Joint response: use cases tables

Claudio Grandi INFN Bologna 11/11/2002 CSN1 - Perugia 24 Conclusions Use cases drawn after about one year of experience with grid tools Some items need more thinking: Read/write datasets Interactive jobs Job splitting First feed-back from grid projects available: The use cases are being used by computing scientists working in the grid projects to build tools useful for the physicists. Some use cases already implemented and used by the experiments (see Mario’s talk)