GFAL 2.0 Devresse Adrien CERN lcgutil team lcgutil-support@cern.ch

What is GFAL 2.0 ? → One and only one Toolkit for all the Grid and Cloud data operations : → Client data access : → support for all common protocols → support for all common storage systems → Data management → One API for all data management → Data transfers → 3rd party transfer support CERN, IT-GT-DMS

Goal of GFAL 2.0 ? → Hide the Grid complexity from the client side. → Simplify the data access world → One API. → Reduce the number of dependencies. → Plugin system → Makes application independent of the technology / protocols. CERN, IT-GT-DMS

The complexity of a random read file operation Client LFC get file replicas list SRM Server DPM Head Node Disk node SRM GET rfio open rfio lseek rfio read lfc_stat GFAL 2.0 gfal2_open gfal2_pread SRM surl → turl CERN, IT-GT-DMS

An old-style Grid application : Grid Application X Core logic File access layer dCache, DPM, Castor lib Globus framework Xrootd framework SRM client Http lib CERN, IT-GT-DMS

A GFAL 2.0 Grid application : Grid Application X Core logic GFAL 2.0 API CERN, IT-GT-DMS

GFAL 2.0 architecture : Extensible... API POSIX File API Transfer Plugin layer Plugin LFC Plugin SRM Plugin RFIO Plugin Dcap Plugin Http/dav Plugin xrootd Plugin S3 ( ? ) experience catalogue plugin (???)  Any Cloud ( Http/Dav ) DPM Castor Dropbox Extensible... EOS dCache

What GFAL 2.0 can do ? In any protocol : Meta-data operations : → stat, rm, mkdir, → mv, rmdir, etc... → list directory, xattr, etc... Remote I/O in any protocol → open/read/write/close → pread/pwrite → etc.. In any protocol : → SRM → GRIDFTP → RFIO-Castor → RFIO-DPM → LFN → GUID → DCAP / GSIDCAP → XROOTD → HTTP(S) * → WebDav(s) * → local file CERN, IT-GT-DMS

Feature : Protocol generic filecopy Third party transfer copy : gfalt_copyfile(context, NULL, « srm://monfichier », « srm://madest/higgs ») gfalt_copyfile(context, NULL, « srm://monfichier », « gsiftp://madest/chuck_norris ») Automatic protocol resolution : gfalt_copyfile(context, NULL, « guid:1233 », «root://xrootd_dpm/curiosity») gfalt_copyfile(context, NULL, « file:///tmp/super_file», «gsidcap://42») CERN, IT-GT-DMS

Really protocol generic : Take a, b in srm, guid, lfn, , rfio, dcap , gsidcap, gsiftp, file, xrootd, (http, dav ) gfalt_copyfile(context, NULL, « a://src/file », «b://dst/file») FINISHED ! CERN, IT-GT-DMS

Speed Up x2-x4 for short transfers Feature : Transparent parallel operations Destination exist ? File copy operation : SRM GET SRM GET Destination exist ? SRM PUT SRM PUT transfer transfer PUTDONE PUTDONE Speed Up x2-x4 for short transfers Old tools... GFAL 2.0

thread-safe session reuse Feature : Transparent thread-safe session reuse Pool of Connections SRM, GSIFTP, HTTP, Etc,.. Transfer 1 Create session End Transfer 2 End One server connection

Feature : extended attributes GFAL 2.0 supports POSIX extended attributes ( getxattr, setxattr, etc...) → Clean way to uses protocol specific functionalities. → Simple get/list/set API → Easy to use.

NO ENVIRONMENT VAR ! Environment Variable User

Feature : Works out of the box : NO ENV VAR ! → Every parameter is in a configuration file : → /etc/gfal2.d/*.conf → No environment variable needed. → Tune it once, re-use everywhere. → Still compatible with classical environment variable of GFAL 1.0, for an easy transition.

Does not make the coffee Missing Feature : Does not make the coffee Coming soon …. ( RFC-2324 compliant ! )

How to use GFAL 2.0 ? : → libgfal2 : → gfal2-python : → a C Library → a set of independent plugins → gfal2-python : → simple and pythonic python bindings → gfalFS fuse module for gfal 2 → allows to mount any grid endpoint in a local folder → gfal-tools : experimental command line tools → gfal-ls, gfal-copy, gfal-mkdir, gfal-cat, …. CERN, IT-GT-DMS

Already in production state ! How to get GFAL 2.0 ? : Already in production state ! → Part of EMI 2 release: → EMI 2 repository : www.eu-emi.eu → Packaged on EPEL : → sudo yum install gfal2-all gfal2-doc gfalFS → is going to be packaged on Debian ( thx Mathias ) → Source available, designed to be portable. https://svnweb.cern.ch/trac/lcgutil/wiki/gfal2 CERN, IT-GT-DMS

Support of GFAL 2.0 : → Part of EMI will be supported after the end of EMI → GFAL 2.0 is the core of FTS 3.0 will be supported in any case. → Long term successor to : lcg clients : lcg-cp, lcg-cr, lcg util python gfal 1.0, etc... → Development Open to everybody CERN, IT-GT-DMS

From gfal1.0 to GFAL 2.0 : → GFAL 2.0 does a lot more than GFAL 1.0 : → a lot of Major changes : Not 100% backward compatible. gfal 1.0 + lcg-util (lcg-cp, .. ) + is-interface + gridftp-ifce + …. CERN, IT-GT-DMS

From gfal1.0 to GFAL 2.0 : But, We provides facilities for the transition The complete POSIX API is backward compatible We provide examples for common operations Doxygen documentation of the API. Any question is Welcome : lcgutil-support@cern.ch CERN, IT-GT-DMS

Lcg-util command line tools successor ? gfal-tools ( ? ) : Lcg-util command line tools successor ? →Users wants command line tools → lcg-util (lcg-cp, lcg-cr ) are heavily used. → lcg-util suffers from his age... → outdated features → hard to maintain → lots of dependencies → → a GFAL 2.0 based -tools ?

gfal-tools : → Experimental tools for lcg-util replacement on the long term. → We need feedback and informations from the experiments : → Which part of lcg-util is needed : → lcg-cp, lcg-ls, lcg-cr, ( and ? ) → What can be improved  in their point of view ?: → It's time for suggestion ! → How can we help to a smooth transition ?

Gfal-tools : Current status → A simple set of generic command line file utils  based on GFAL 2: → gfal-copy → gfal-ls → gfal-rm → gfal-cat → gfal-save → etc.... → Prototype already available for comments ... → Still under development ! → It's the right time for comments...

https://svnweb.cern.ch/trac/lcgutil/wiki/gfal2 lcgutil-support@cern.ch