Grid data transfer tools Tier 3(g,w) meeting at ANL ASC – May 19, 2009 Marco Mambelli – University of Chicago

Grid/OSG components  Standard installation  Set of services or resources containing data and providing processing power  Computing elements (CE)  Storage elements (SE)  Information systems  Clients WN Disk CE Door Gatekeeper SE LRM Clients Information system Clients Auxiliary systems

Site ATLAS Distributed Data Management (DQ2) DQ2 dataset catalogs: -files in dataset -where are datasets DQ2 “Queued Transfers” Local File Catalog (LFN ->PFN) File Transfer Service File Transfer Service DQ2 Subscription Agents DQ2 (ATLAS) Grid provided ATLAS provided? Local Storage Remote Storage

ATLAS space token token namestorage type used ATLASDATATAPET1D0RAW data, ESD, AOD from re-proc XX ATLASDATADISKT0D1ESD, AOD from dataXXX ATLASMCTAPET1D0HITS from G4, AOD from ATLFAST X ATLASMCDISKT0D1AOD from MCXXX ATLASPRODDISKT0D1buffer for in-and exportX ATLASGROUPDISKT0D1DPDXXX ATLASUSERDISKT0D1User DataXX *) ATLASLOCALGROUP DISK T0D1Local User How data is organized at sites (SEs)

Name used to refer files  Logical File Name  lfn:/grid/dteam/vkoukis-gfal-test2  Grid Unique IDentifier (GUID)  guid:4f74b453-5aaf-4af5-af2d-b438d081bb63  Storage URL (for a specific replica, on a specific Storage Element)  srm://se01.athena.hellasgrid.gr/pnfs/athena.hellasgrid.gr/data/dteam/generated/ /file775276c2-7bbf-4f3d-92cf-5d5cdbaba254  Transport URL (for a specific replica, on an SE, with a specific protocol)  gsidcap://se01.athena.hellasgrid.gr:22128//pnfs/athena.hellasgrid.gr/data/dteam/generated/ /file775276c2-7bbf-4f3d-92cf-5d5cdbaba254 SURLGUID LFN TURL

Datasets and Containers  Container datasets are the logical equivalent of physics datasets. They contain the same files.  Their name is the dataset name before the tid (or T for cosmic data). Name ends with a ‘/’  Containers can contain only DDM datasets. You cannot make containers of containers  Dataset in DDM is a set of files  For example, MC containers contain the files belonging to the tid datasets (containing tidxxxx with xxxx as task number).  Most of the dq2 commands are similar for datasets and containers except few ones

Example  Data : data08_cosmag physics_RPCwBeam.recon.ESD_FIL TERED.o4_f70/  MC :mc PythiaWenu_1Lepton.recon.AOD.e352_s462_r 541/

How to move data  DQ2 Client Tools  command line  package developed by ATLAS  RPM and Pacman  uses Grid middleware  tested on WLCG-Client and gLiteUI

Discover and get data  DDM central catalog:  list of sites contain at least a fraction of dataset  List of GUIDs per dataset + checksum + size (for consistency check)  LFC catalog : List of files on a site:  Each catalog contains list of physical replicas for a GUID  Input : list of GUIDs  Middle : For each GUID, list of replicas managed by T1 LFC  Output: List of GUIDs on the site  In addition: list of SURLs of the files (used by jobs)  Storage Element catalog:  Map the SURL  WARNING : No check of consistency between catalogs : DDM/ LFC/SE

Client installation  Install DQ2 clients (and WLCG-client)  $ source /share/pacman/setup.sh  $ mkdir /share/wlcg-client; cd /share/wlcg-client  $ pacman -get  (choose local installation ‘l’)  source setup.sh  vi $VDT_LOCATION/vdt/etc/vdt-update-certs.conf  (choose OSG certificate set)  source $VDT_LOCATION/vdt-questions.sh; $VDT_LOCATION/vdt/sbin/vdt-setup-ca-certificates  DQ2 client itself  mkdir /share/dq2-client; cd /share/dq2-client  pacman -trust-all-caches -allow tar-overwrite -get pacman/cache:DQ2Clients

Client setup  Setup with WLCG client (superset of OSG/VDT Client)  $ source /share/osg-client/setup.sh  $ source /share/dq2-client/setup.sh  (you need a grid certificate in ATLAS VO)  $ voms-proxy-init –voms atlas:/atlas  Now you can:  move dataset  find files  copy files

Commands and Use   dq2-ls -fp DATASETNAME  dq2-ls -L SITENAME -fp DATASETNAME  dq2-get DATASETNAME  dq2-get -f FILENAME DATASETNAME  dq2-put -s SOURCEDIRECTORY DATASETNAME  dq2-list-datasets-container CONTAINERNAME  dq2-list-dataset-replicas-container CONTAINERNAME  dq2-get -f FILENAME1 CONTAINERNAME

References and other way to move data  DQ2 Clients  ement ement   OSG OSG  Subscriptions   Direct copy  Examples for the first two follow

Subscription service

DDM Dashboard Can be used to check the status of your (and other) subscriptions Use drill-down menus

DDM Dashboard detail Here you see the subscriptions to Chicago (BNL cloud)

Some observations  Works generally for O(TB) datasets  Large number of small files are inefficient (avoid O(100) files datasets  DDM operations for support

DQ2 Clients  Use the how-to page as reference (open in browser)  Live demo (terminal)  setup  voms proxy  dq2-ls, -L ROAMING, name*; hat’s in the output  containers (= physics datasets)  _tidXXXX datasets  non _tid datasets – DO NOT USE THIS!  dq2-list-dataset-replicas-container, explain output  which site (SE) has all your files?  # of complete (datasets) = Total datasets  verify that the files are there: dq2-ls -f (verify path -p)  use –L site_that_you_want_to_check

DQ2 Clients (2)  (cont)  Create PoolFileCatalog.xml dq2-ls –P  --replace-io=REPLACEIO ('regularexpression^newvalue’)  e.g. srm://uct2-dc1.uchicago.edu(:[0- 9]+)(/srm/managerv2SFN=)/pnfs/^dcache:/pnfs/  Errors in dq2-get  no replica available (check dq2-ls -r)  files not copied: try different ‘protocol’: ng,lcg,lcg1,srm,dcap,rfio,…  checksum error: report to DDM operations  Help  DDM operation savannah portal or hn-atlas-DDMOperations  hn-atlas-dist-analysis-help hypernews

Demo 1 [uct3-edge5] /ecache/marco/test_dq2-get >. /share/wlcg-client/setup.sh >. /share/dq2-client/setup.sh > voms-proxy-init -voms atlas Enter GRID pass phrase: Your identity: /DC=org/DC=doegrids/OU=People/CN=Marco Mambelli Creating temporary proxy Done Contacting lcg-voms.cern.ch:15001 [/DC=ch/DC=cern/OU=computers/CN=lcg-voms.cern.ch] "atlas" Done Creating proxy Done Your proxy is valid until Thu Feb 26 22:13: > dq2-ls -L ROAMING mc singlepart_e_Et25.recon.AOD.* mc singlepart_e_Et25.recon.AOD.e342_s462_r604_tid032272_sub mc singlepart_e_Et25.recon.AOD.e342_s462_r604_tid032272_sub mc singlepart_e_Et25.recon.AOD.e342_s462_r563 … mc singlepart_e_Et25.recon.AOD.e342_s462_r604/ mc singlepart_e_Et25.recon.AOD.e342_s462_r604_tid mc singlepart_e_Et25.recon.AOD.e342_s462_r604_tid032272

Demo 2 > dq2-list-dataset-replicas-container \ mc singlepart_e_Et25.recon.AOD.e342_s462_r563/ mc singlepart_e_Et25.recon.AOD.e342_s462_r563_tid027926: INCOMPLETE: WISC_MCDISK COMPLETE: AGLT2_MCDISK,BNL-OSG2_MCDISK,BNLPANDA,… mc singlepart_e_Et25.recon.AOD.e342_s462_r563_tid027925: INCOMPLETE: RRC-KI_MCDISK COMPLETE: AGLT2_MCDISK,BNL-OSG2_MCDISK,BNLPANDA,… Container name: mc singlepart_e_Et25.recon.AOD.e342_s462_r563/ Total datasets: 2 Summary: SITE / # COMPLETE / # INCOMPLETE / TOTAL UKI-SCOTGRID-GLASGOW_MCDISK WISC_MCDISK CERN-PROD_MCDISK … SARA-MATRIX_MCDISK MWT2_UC_MCDISK 2 0 2

Demo 3 > dq2-ls -L MWT2_UC_MCDISK -f mc singlepart_e_Et25.recon.AOD.e342_s462_r563_tid mc singlepart_e_Et25.recon.AOD.e342_s462_r563_tid [X] AOD _00507.pool.root.3 48B4D116-38AE-DD11-B876-00A0D1E7FB98 md5:ba76b8c771a11c d0fb [X] AOD _00196.pool.root.1 5E39D1C3-31AE-DD11-9C54-00A0D1E7BC5E md5:6a2fd3ad0464c4da30faaf05ca55b86a [X] AOD _00249.pool.root.1 0A AE-DD11-9E81-00A0D1E70C54 md5:797dfea0f1b e18f96c6e060e … [X] AOD _00209.pool.root.1 3C2D1AC7-31AE-DD A0D1E7B60E md5:2b85fcf c8ec3d5b7725e5f [X] AOD _00765.pool.root.1 B0773A0C-A8AF-DD11-ABD6-00A0D1E50051 md5:099b1c154adf4e4376def5afd93c total files: 720 local files: 720 total size: date: :53:23 > dq2-ls -L MWT2_UC_MCDISK -f mc singlepart_e_Et25.recon.AOD.e342_s462_r563_tid |\ tail -n 4 total files: 80 local files: 80 total size: date: :38:26 > dq2-ls -L MWT2_UC_MCDISK -fp mc singlepart_e_Et25.recon.AOD.e342_s462_r563_tid027926