ATLAS DIAL: Distributed Interactive Analysis of Large Datasets David Adams Brookhaven National Laboratory February 13, 2006 CHEP06 Distributed Data Analysis session

ATLAS D. Adams CHEP06 DIALFebruary 13, Contents Goals Interactive analysis Model Service paradigm Datasets Transformations Jobs Schedulers Catalogs ATLAS deployment Performance Conclusions Documentation DIAL contributors Updated: February 13, 2006

ATLAS D. Adams CHEP06 DIALFebruary 13, Goals DIAL was initiated in 2002 and first presented at CHEP03 Goals remain the same: Demonstrate the feasibility of interactive analysis of large datasets –How much data can we study interactively? Set requirements for grid tools and services –As much as possible, try to use existing and evolving products Provide ATLAS with a useful environment for distributed analysis –Enable physicists to easily examine the AOD (analysis oriented data) and other samples from Monte Carlo production –Be ready for “real” data when it appears in a year or two

ATLAS D. Adams CHEP06 DIALFebruary 13, Interactive analysis Interactive is taken to mean a system which responds to user requests on the time scale of a few minutes (or less) So user may submit many requests of the course of a few hours to try out different ideas without losing his or her train of thought Does not require that user literally “interacts” directly with running jobs Result should be available in a few minutes –If CPU cycles (and bandwidth) are available If not, user should be able to monitor progress and receive updated partial results on an interactive time scale This responsiveness is obtained with parallel processing Input dataset is split Subdatasets are processed in parallel Results are merged Data and compute resources must be accessible on this time scale Most likely share resources with other activities, e.g. production

ATLAS D. Adams CHEP06 DIALFebruary 13, Model Input dataset describes the data to be processed Transformation is an operation to be performed on this data Output of this transformation is another dataset, the result Job preferences are hints on how to carry out processing But should not affect the essence of the result Job is an instance of applying a transformation to a dataset to produce a result Scheduler creates ands runs jobs Catalogs store objects and metadata

ATLAS D. Adams CHEP06 DIALFebruary 13, Service paradigm DIAL provides a web service framework with modules to implement scheduler and catalog services Based on gsoap With plugin for GSI authentication and authorization –Delegation to maintain user identity throughout the processing chain Motivation Provide users with a common interface for wide range of batch and workload management systems Insulate users from splitting, merging and error recovery –Users may express preferences but the work is done at a central point –User are free to disconnect during processing

ATLAS D. Adams CHEP06 DIALFebruary 13, Datasets Dataset describes a collection of data Properties Unique identifier for persistent reference –Two 32-bit integers –All persistent objects have this Location of the data (e.g. a list of files) Content is a collection of content blocks; each has –Content label (e.g. ESD, AOD, HIST, …) –List of content identifiers, each a type and key >E.g. Jets-cone7 or TH1F-jet_pt –Event ID list (may be restricted to count, first and last) List of subdataset ID’s –I.e. datasets are hierarchical

ATLAS D. Adams CHEP06 DIALFebruary 13, Datasets (cont) Classes Base class Dataset defines interface Subclass GenericDataset provides data and means to stream to and from a generic XML representation –Most complete dataset types inherit from this DIAL provides some other generic types –SingleFileDataset – holds a single file >Base for VO-specific classes, e.g. AtlasPoolEventDataset >Subclasses add content information –MultiFileDataset – a generic list of files >No content information >Use this for integration with ATLAS DQ2 –EventMergeDataset – merger of event datasets >E.g. VO-specific single file datasets –TextDataset – directly carries named text files >E.g. for log files

ATLAS D. Adams CHEP06 DIALFebruary 13, Transformations A transformation has two components The application carries out the processing The task carries data used to configure the application Task is a list of named text files Application defines the task interface –Expected file names and their meaning –Not formalized – test by running a job with the task Files may hold run time parameters, source code, … Binary data (e.g. libraries) may be put in an SE (storage element) and the file URL (e.g. LFN) carried in a task file

ATLAS D. Adams CHEP06 DIALFebruary 13, Transformations (cont) Application holds two scripts build_task –Used to build. e.g. compile the task –Run in a directory containing the task files –Writes output to that same directory run –Does the data processing –Run in a directory with the input dataset in file dataset.xml and task location in the file taskdir –Expected to write result dataset to result.xml Both run in minimal environment –Posix, g++ and pkgmgr –Latter provides means to locate software Both scripts return 0 to indicate success

ATLAS D. Adams CHEP06 DIALFebruary 13, Jobs Job is constructed from Job definition –Application, task, dataset and preferences Local run directory –Remote jobs also have a remote dir and a way to bring back result Name of run script (build_task, or run) Base class Job holds data and provides implementation for Reading job data Streaming data to and from XML Base class provides interface for Creation Submission Killing Updating the job data Subclasses implement this for different batch or WM systems

ATLAS D. Adams CHEP06 DIALFebruary 13, Jobs (cont) DIAL provides job subclasses: LsfJob for submission to LSF CondorJob for submission to Condor or Condor-G CondorCodJob to use Condor COD (computing on demand) ScriptedJob to control job through a user-supplied script –Rather than adding a new subclass –Used for globus gatekeeper and PANDA CompoundJob used internally by schedulers Job data includes State (initialized, running, done, failed, killed) –Last three are terminal >job is immutable when one of those states is reached Start, update and stop times Batch or WMS ID, worker node Input job definition and output result And more…

ATLAS D. Adams CHEP06 DIALFebruary 13, Schedulers Normally a scheduler is used to submit, monitor and kill jobs Local scheduler provides submission of single jobs to a particular batch or WMS (via choice of job type) Master scheduler provides distributed processing –Split input dataset –Use local scheduler to process each subdataset –Merge results as subjobs finish Scheduler may be provided as a web service Called an analysis service Web service client provides scheduler interface –Submit a job: job definition  job ID –Check status: job id  updated job object (including result) –Kill job with job ID if desired This is the typical means of accessing a scheduler

ATLAS D. Adams CHEP06 DIALFebruary 13, Catalogs Catalogs provide means to access and select objects Datasets, applications, tasks and jobs Repositories Provide means to store and retrieve objects Indexed by object ID MySQL and flat file implementations Selection catalogs Assign name and other metadata to an object SQL query can be used to select objects MySQL implementation Catalogs may be accessed directly or via web services Web service provides GSI authentication and authorization Removal and update restricted to entry owner

ATLAS D. Adams CHEP06 DIALFebruary 13, Catalogs (cont) Object references may be by value, ID, name or query Value  XML description is provided ID  XML obtained from a repository –Except for jobs that have not reached a terminal state, all persistent objects are immutable  the same object is always obtained. Name or query  selection catalog used to find ID –Selected object may be different for different catalogs –Or may change with time –E.g. use name to identify the latest version

ATLAS D. Adams CHEP06 DIALFebruary 13, ATLAS deployment DIAL services have been deployed at BNL Unique ID Repository Selection catalog Suite of analysis services –Local batch >Fast = LSF queue that overlays jobs (100 MB, 15 min) >Short = Condor preemptive (1 GB, 90 min) >Long = Condor low priority (1 GB, 1 day) –PANDA –Condor-G (as demonstration now) –Globus (usually not run; big load on gatekeeper) Each service has a dedicated URL Data is available at BNL All AOD from last production run (Rome data) All data from the current production (CSC) DIAL datasets are available for all the above

ATLAS D. Adams CHEP06 DIALFebruary 13, ATLAS deployment (cont) ATLAS transformations have been defined All run Athena – the ATLAS framework program In all cases, the task includes –ATLAS release tag, e.g –Top level job options (run time parameters) –List of output files to save (used to construct output dataset) Applications –atlasopt includes just the above –aodhisto adds code to be built inside the standard analysis package >Compiled before the job is run –atlasdev makes use of arbitrary changes to the ATLAS release >Packaged in a tarball constructed from user’s development area >User must compile (for now) –atlasxform runs any of the transformations used in data production Example tasks are provided for each of these transformations

ATLAS D. Adams CHEP06 DIALFebruary 13, Performance To assess performance a reference dataset was constructed Using much of the available CSC AOD data at BNL Mix of physics channels excluding single particle events 1872 files, each with 100 events Size is 25 GB, i.e. 130 kB/event Larger datasets were constructed by duplicating these files 0.5, 1, 1.5, 2, 3,4,5 and 6 times the reference sample Copies divided between two NFS file servers Transformation The atlasdev transformation was used The task was atlasdev_aod_histos version 3 –Opens four containers in each event >Truth, electrons, cone jets and jet tags –Fills a couple dozen histograms

ATLAS D. Adams CHEP06 DIALFebruary 13, Performance (cont) The reference dataset was run as a single job Athena clock time was 70 minutes –I.e. 43 ms/event, 3.0 MB/s –Actual data transfer is about half that value >Some of the event data is not read Following figure shows results Local fast queue (LSF) –Green squares Local short queue (Condor preemptive) –Blue triangles Condor-G to local fast –Red diamonds PANDA –Violet circles

ATLAS D. Adams CHEP06 DIALFebruary 13, Single job (Single job)/10

ATLAS D. Adams CHEP06 DIALFebruary 13, Conclusions A useful DIAL system has been deployed for ATLAS Common analysis transformations Access to current data For AOD to histograms and large samples, 15 times faster than a single process Easy to use Root interface Web-based monitoring Packaged datasets, applications and example tasks Demonstrated viability of remote processing Via Condor-G or PANDA Need interactive queues at remote sites –With corresponding gatekeeper or DIAL service Or improve PANDA responsiveness

ATLAS D. Adams CHEP06 DIALFebruary 13, Documentation DIAL home page at Service page User guide

ATLAS D. Adams CHEP06 DIALFebruary 13, DIAL contributors GANGA Karl Harrison, Alvin Tan DIAL David Adams, Wensheng Deng, Tadashi Maeno, Vinay Sambamurthy, Nagesh Chetan, Chitra Kannan ARDA Dietrich Liko ATPROD Frederic Brochu, Alessandro De Salvo AMI Solveig Albrand, Jerome Fulachier ADA (plus those above), Farida Fassi, Christian Haeberli, Hong Ma, Grigori Rybkine, Hyunwoo Kim