INFSO-RI Enabling Grids for E-sciencE ATLAS Distributed Analysis A. Zalite / PNPI

Enabling Grids for E-sciencE INFSO-RI Overview Why? Goal ADA Model First steps Demo example More examples Conclusion

Enabling Grids for E-sciencE INFSO-RI Why? Huge amount of data –Atlas experiment is expected to record several petabytes of data per year –Atlas offline system will produce similar amount of data (ESD, AOD, …) Globally-distributed members of Atlas collaboration –Over 1000 physicists from all over the world will take part in data analysis The data have to be available to all members of the collaboration

Enabling Grids for E-sciencE INFSO-RI Goal Provide to globally distributed users –Access to globally distributed data –Tools to perform globally distributed processing on this data Easy to use and access from analysis environment –Flexible to adopt to environment Enable effective use of all ATLAS computing resources Trace information about processing with any data –Where did this data (event or analysis) come from?

Enabling Grids for E-sciencE INFSO-RI ADA Model Components: Data described by a Datasets (collection of data) –Location of the data (e.g. files) –Content (e.g. list of event ID’s and the type of the data for each event) Transformation describes an operation that can act on a dataset to produce a new dataset –Application scripts used to run job to build task or process data –Task carries user parameters or code (E.g. atlas release, job options, and/or algorithm code) Job is an instance of the transformation acting on a dataset

Enabling Grids for E-sciencE INFSO-RI ADA Model Many ATLAS-specific transformations have been defined –Atlasopt: user provides ATLAS release and job options –Aodhisto: atlasopt plus code to build in UserAnalysis package –Atlasdev: atlasopt plus local development directory –Atlasdev-src: same as atlasdev except development area is tarred up and will be rebuilt if platform changes All these transformations run Athena

Enabling Grids for E-sciencE INFSO-RI ADA Model Transformation

Enabling Grids for E-sciencE INFSO-RI ADA Model This view enables distributed processing: Split input dataset –Along event, file, or sub-dataset boundaries Create separate sub-job for each sub-dataset Implies post-processing stage to merge results (output datasets) Users carry out processing by Defining a job –Application, task and dataset Submitting this definition to a scheduler –Typically an analysis service Polling for status –Job state (and sub-job states) –Result dataset

Enabling Grids for E-sciencE INFSO-RI ADA Model On receiving a job request, the scheduler Builds the task (or locates an existing build) Split the dataset into sub-datasets Create and submits a sub-job for each sub-dataset Merge the results (output datasets) from each sub-job into overall result

Enabling Grids for E-sciencE INFSO-RI ADA Architecture

Enabling Grids for E-sciencE INFSO-RI ADA ADA uses DIAL framework. Release 1.20 of DIAL is the basis for the current ADA system. To use ADA it is necessary To have Grid certificate –Certificate from Russian CA is OK To be member of Atlas VO –Takes some time

Enabling Grids for E-sciencE INFSO-RI First Steps Working node - LXPLUS at CERN Setup grid environment:. /afs/cern.ch/project/gd/LCG-share/sl3/etc/profile.d/grid_env.sh Certification proxy initialization: grid-proxy-init DIAL setup (setup script that defines a few environmental variables and aliases) at CERN: DIALSETDIR=/afs/cern.ch/user/d/dial/apps/dial/setup Verify user certificate and check the status of the unique ID service by issuing the command "uidtest" after setting up dial.

Enabling Grids for E-sciencE INFSO-RI First Steps

Enabling Grids for E-sciencE INFSO-RI First Steps The best way to start with DIAL is to run the demos inside ROOT These demos define a job –application (papp) –task (ptsk) –dataset (pdst) and submit it to the current scheduler (msch) Start: dialroot –i flag –i means that any missing DIAL configuration, example or demo files will be copied into the local directory (necessary only 1 st time)

Enabling Grids for E-sciencE INFSO-RI First Steps

Enabling Grids for E-sciencE INFSO-RI First Steps

Enabling Grids for E-sciencE INFSO-RI Demo Example Distributed analysis is an iterative process where a physicist defines a job, submits it to a processing system, examines the result and then repeats the sequence. Demo selects an application, task and dataset which are then submitted to a scheduler to define a job. root [0].x demos/demo4.C This defines papp, ptsk and pdst root [1] submit() Submit a job based on papp, ptsk and pdst root [2] get_results() Get job status and partial result root [3] TBrowser br Check ouput ntuples and histgrams

Enabling Grids for E-sciencE INFSO-RI Demo Example A job is specified by defining a transformation and selecting a dataset to process with this transformation. The transformation is specified by an application and a task. The application carries the scripts that do the processing and the task carries user configuration data. Demo4 uses aodhisto to create histograms and ntuples from user source code The demo identifies objects by name, extract the corresponding ID from a selection catalog and use this ID to extract the object from a repository.

Enabling Grids for E-sciencE INFSO-RI Demo Example void demo4() { string aname = "aodhisto"; string tname = "aodhisto_zll_aod"; string dname = "hma.dc digit.A1_z_ee.aod files"; aid = asc.id(aname); tid = tsc.id(tname); did = dsc.id(dname); papp = ar.extract(aid); ptsk = tr.extract(tid); pdst = dr.extract(did); }

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI Demo Example Objects: papp - pointer to the current application ptsk - pointer to the current task pdst - pointer to the current dataset Can be displayed root [4] pprint(papp) Display the application root [5] pprint(ptsk) Display the task root [6] pprint(pdst) Display the dataset

Enabling Grids for E-sciencE INFSO-RI Demo example

Enabling Grids for E-sciencE INFSO-RI Demo Example

Enabling Grids for E-sciencE INFSO-RI More Examples There are more examples: Demo5 uses esd2aod to create AOD from ESD using the prodsys transformation Demo6 uses atlasopt to run a job with provided job options Demo7 uses atlasdev to run a job based on a users atlas development area Demo8 uses atlasdev-src to run a job based on a tarball of a user development area

Enabling Grids for E-sciencE INFSO-RI More Examples Displaying the status of all catalogs to verify connection and see the size of each: root [4] show_catalogs()

Enabling Grids for E-sciencE INFSO-RI More Examples A list of available datasets may be obtained by querying the DSC (dataset selection catalog, object dsc). The DSC is the primary user interface to datasets and it plays a role of what is often called a metadata catalog. Limit the query to 100 results (received 12). The query resticts the selection to TOP level datasets, i.e. complete samples intended for user access and then uses the name to select Rome samples with v10 reconstruction, SUSY data using all AOD data avaialble at BNL. AOD-bnl replaced with AOD to get samples available at both CERN and BNL. Counting datasets matching a query with the query_count method

Enabling Grids for E-sciencE INFSO-RI More Examples

Enabling Grids for E-sciencE INFSO-RI More Examples DCS supports list of parameters which can be used in selection of Datasets

Enabling Grids for E-sciencE INFSO-RI More Examples List attributes for given Dataset Record ID and fetch the Dataset from repository

Enabling Grids for E-sciencE INFSO-RI More Examples

Enabling Grids for E-sciencE INFSO-RI More Examples Select an application in a similar way

Enabling Grids for E-sciencE INFSO-RI More Examples Select a task in a similar way

Enabling Grids for E-sciencE INFSO-RI More Examples The application usually is not modified, but necessity of task modification is very likely Extract the files from the task

Enabling Grids for E-sciencE INFSO-RI More Examples The list of jobOptions can be found in CVS repository at atlas/PhysicsAnalysis/AnalysisCommon/ AnalysisExamples/share/

Enabling Grids for E-sciencE INFSO-RI More Examples Now it is possibly to build a new task from the modified files: ptsk = new dial::Task("atlas_release jo.py output_content", "mytask"); The list of files used to construct the task may be replaced with "*" if you want all the files from the directory Now papp, ptsk and pdst are defined, and job can be submited

Enabling Grids for E-sciencE INFSO-RI More Examples

Enabling Grids for E-sciencE INFSO-RI More Examples

Enabling Grids for E-sciencE INFSO-RI More on More Examples It is not necessary to do a lot of typing (as we did before) to perform previous analysis There is simple way to avoid this – job definition script that defines the application, task and dataset (variables papp, ptsk and pdst). Sample script can be found here: The sample script is copied into the local directory when the dialroot files are installed (dialroot -i). Edit the top part of this script to specify the application, task and dataset of interest. Run: root [0].x jobdef.C root [1] submit()...

Enabling Grids for E-sciencE INFSO-RI More on More Examples void jobdef() { // Specify names for the application, task and dataset. // Typical job definition is created by changing these values. // Depending on the following code, a name may be intepreted as // one or more of the following. // 1. ID: Object identifier. // 2. name: Object name in the default selection catalog. // 3. directory: Name of a directory holding files to be used // construct the object. // 4. xml: Name of a file holding the XML description of the object. // Application: directory, name, or ID. string aname = "atlasopt"; // Task: directory, xml, name, or ID. string tname = "atlasopt_example_zll "; // Dataset: ID or name. string dname = "hma.dc digit.A1_z_ee.aod files"; …….

Enabling Grids for E-sciencE INFSO-RI More on More Examples There is web-interface “DIAL CATALOG QUERY PAGE”

Enabling Grids for E-sciencE INFSO-RI More on More Examples This interface permit to switch to: –Dataset Selection Catalog (DSC) –Task Selection Catalog (TSC) –Application Selection Catalog (ASC) A list of available datasets may be obtained from DSC query page Some useful applications and example tasks are cataloged as well. The application and task catalogs may also be examined using the ASC query page and TSC query page

Enabling Grids for E-sciencE INFSO-RI More on More Examples

Enabling Grids for E-sciencE INFSO-RI More on More Examples

Enabling Grids for E-sciencE INFSO-RI More on More Examples

Enabling Grids for E-sciencE INFSO-RI More on More Examples

Enabling Grids for E-sciencE INFSO-RI More on More Examples

Enabling Grids for E-sciencE INFSO-RI More on More Examples

Enabling Grids for E-sciencE INFSO-RI More on Transformations Transformations are applied to datasets to produce new datasets. A transformation includes: –an application which carries out the processing –a task used to configure the application An application provides two entry points: one to build (e.g compile) a task and one to process a dataset A task is a collection of named text files It is not sensible to arbitrarily combine any task with any application.

Enabling Grids for E-sciencE INFSO-RI More on Transformations There is a task interface that specifies which files must or may be present in a task and how these files are to be used. Tasks are labeled with the interface they provide and applications with the task interface they expect. Task interfaces: –atlas_release –atlas_job_options –atlas_simple_analysis –atlas_user_analysis –atlas_developer_directory –atlas_developer –atlas_xform

Enabling Grids for E-sciencE INFSO-RI More on Transformations The task interface atlas_release specifies an atlas release List of files for atlas_release: –atlas_release - ATLAS release version, e.g The task interface atlas_job_options specifies an atlas release, job options and output content List of files for atlas_job_options: –atlas_release - ATLAS release version, e.g –jo.py - User job options. –output_content - describes the output to be saved (content label and name - HIST hist.root )

Enabling Grids for E-sciencE INFSO-RI More on Transformations The ADA task interface atlas_user_analysis specifies an atlas release and files to replace those in the UserAnalysis package List of files for atlas_user_analysis: –atlas_release - ATLAS release version, e.g –*.h - header files. –*.cxx - C++ source files –requirements - CMT requirements file –AnalysisSkeleton_jobOptions.py - job options file –output_content - describes the output to be saved

Enabling Grids for E-sciencE INFSO-RI Documentation ADA system described on ADA home page – This page also has a link to DIAL 1.20 release page

Enabling Grids for E-sciencE INFSO-RI Conclusion ADA permits now to perform distributed analysis for Atlas experiment Available documentation permits to newcomers to start using of ADA Further development (especially user-oriented) will allows more wider distribution of ADA among physicists