PROOF – Parallel ROOT Facility Fons Rademakers http://root.cern.ch Bring the KB to the PB not the PB to the KB March, 2003 CHEP'03

PROOF Collaboration between core ROOT group at CERN and MIT Heavy Ion Group Part of and based on ROOT framework Uses heavily ROOT networking and other infrastructure classes Currently no external technologies March, 2003 CHEP'03

Main Motivation Design a system for the interactive analysis of very large sets of ROOT data files on a cluster of computers The main idea is to speed up the query processing by employing parallelism In the GRID context, this model will be extended from a local cluster to a wide area “virtual cluster”. The emphasis in that case is not so much on interactive response as on transparency With a single query, a user can analyze a globally distributed data set and get back a “single” result The main design goals are: transparency, scalability, adaptability March, 2003 CHEP'03

Parallel Script Execution #proof.conf slave node1 slave node2 slave node3 slave node4 Local PC Remote PROOF Cluster proof proof = master server root stdout/obj proof proof = slave server ana.C proof proof proof *.root TFile node1 ana.C TNetFile *.root $ root root [0] tree->Process(“ana.C”) root [1] gROOT->Proof(“remote”) root [2] chain->Process(“ana.C”) $ root root [0] .x ana.C root [1] gROOT->Proof(“remote”) $ root root [0] .x ana.C $ root node2 TFile *.root node3 TFile *.root node4 March, 2003 CHEP'03

PROOF - Architecture Data Access Strategies Transparency Local data first, also rootd, rfio, SAN/NAS Transparency Input objects copied from client Output objects merged, returned to client Scalability and Adaptability Vary packet size (specific workload, slave performance, dynamic load) Heterogeneous Servers Migrate to multi site configurations March, 2003 CHEP'03

Workflow For Tree Analysis – Pull Architecture Slave 1 Master Slave N Process(“ana.C”) Process(“ana.C”) Initialization Packet generator Initialization GetNextPacket() GetNextPacket() Process 0,100 100,100 Process GetNextPacket() GetNextPacket() Process 200,100 300,40 GetNextPacket() Process GetNextPacket() Process 340,100 440,50 Process GetNextPacket() GetNextPacket() Process 490,100 590,60 Process SendObject(histo) SendObject(histo) Wait for next command Add histograms Wait for next command Display histograms March, 2003 CHEP'03

Additional Issues Error handling Authentication Death of master and/or slaves Ctrl-C interrupt Authentication Globus, ssh, kerb5, SRP, clear passwd, uid/gid matching March, 2003 CHEP'03

Running a PROOF Job // Analyze TChains in parallel gROOT->Proof(“proof.cern.ch”); TChain *chain = new TChain(“AOD"); chain->Add("lfn://alien.cern.ch/alice/prod2002/file1"); . . . chain->Process(“myselector.C”); // Analyze generic data sets in parallel gROOT->Proof(“proof.cern.ch”); TDSet *objset = new TDSet("MyEvent", "*", "/events"); objset->Add("lfn://alien.cern.ch/alice/prod2002/file1"); . . . objset->Add(set2003); objset->Process(“myselector.C++”); March, 2003 CHEP'03

PROOF Scalability 8.8GB, 128 files 1 node: 325 s 32 nodes in parallel: 12 s 32 nodes: dual Itanium II 1 GHz CPU’s, 2 GB RAM, 2x75 GB 15K SCSI disk, 1 Fast Eth, 1 GB Eth nic (not used) Each node has one copy of the data set (4 files, total of 277 MB), 32 nodes: 8.8 Gbyte in 128 files, 9 million events March, 2003 CHEP'03

PROOF and Data Grids Many services are a good fit Authentication File Catalog, replication services Resource brokers Monitoring  Use abstract interfaces Phased integration Static configuration Use of one or multiple Grid services Driven by Grid infrastructure March, 2003 CHEP'03

Different PROOF–GRID Scenarios Static stand-alone Current version, static config file, pre-installed Dynamic, PROOF in control Using grid file catalog and resource broker, pre-installed Dynamic, ALiEn in control Idem, but installed and started on the fly by AliEn Dynamic, Condor in control Idem, but allowing in addition slave migration in a Condor pool March, 2003 CHEP'03

Running PROOF Using AliEn TGrid *alien = TGrid::Connect(“alien”); TGridResult *res; res = alien->Query(“lfn:///alice/simulation/2001-04/V0.6*.root“); TDSet *treeset = new TDSet("TTree", "AOD"); treeset->Add(res); gROOT->Proof(res); // use files in result set to find remote nodes treeset->Process(“myselector.C”); // plot/save objects produced in myselector.C . . . March, 2003 CHEP'03

Near Future Working with some early users Ironing out of several remaining issues Writing install and user guide After CHEP general release Part of standard ROOT distribution Still to do for analysis: Support for: event lists, friend trees Still to do for grid: Interfacing to file catalogs, resource broker Multi site PROOF sessions March, 2003 CHEP'03