1. Rucio & objectstores
James Perry

2. Rationale
Cloud-style objectstores are becoming increasingly popular for data storage
Amazon S3
OpenStack Swift
Google Cloud Platform
Existing Rucio support for objectstores had limitations:
AWS S3 support required clients to store objectstore credentials
OpenStack Swift not properly supported at all
Google Cloud Platform included URL signing support in Rucio core
But only used for downloads
Code was untidy with hacks, pseudo-protocols, special casing, etc.

3. S3 download in Rucio (old method)
Client
Objectstore
Credentials
Data file
Credentials

4. Signed URLs Objectstores support signed URLs (TempUrls in Swift)
URL, operation (GET, PUT, DELETE) and expiry time are cryptographically signed with a secret key
Server checks the URL signature and enforces operation and time restrictions
Can be generated offline (at least in Swift and in newest version of S3)
Instead of giving objectstore credentials to clients, keep them on the Rucio server and have the server generate signed URLs for the clients

5. Upload and download: completed
Added URL signing for S3 and Swift to Rucio core
Alongside existing GCP implementation
Downloads worked immediately
Rucio already signs the URLs returned by list_replicas when required
Added support for uploading to signed URLs
Required new Rucio API endpoint to get signed URL for file that does not yet exist
More complicated than downloads because (e.g.) checksums and existence checks cannot be performed on signed PUT URL

6. S3 download in Rucio (new method)
Rucio Server
Request
Credentials
Signed URL
Credentials
Client
Objectstore
Signed URL
Data file

7. Deletion: completed
URL signing code already added to core supported DELETE operation
Added a call to sign URL when required to Rucio’s reaper daemon
A bug in underlying GFAL2 library prevented this from working
It performed a stat on the URL, not allowed on URL signed for DELETE operations
Fixed in GFAL

8. 3rd party transfers (delegated to FTS)
Rucio uses FTS3 for all 3rd party (RSE-to-RSE) transfers
Signed URLS are not used for this
Technically possible but undesirable as it could result in leakage of signed URLs
Instead, FTS holds the objectstore credentials and performs authentication itself

9. Code clean up (in progress)
Legacy objectstore code is being removed from Rucio
Fits well with core Rucio team’s current focus

10. Future Plans Event Service Auditing Dynafed
Check that URL signing performs well enough for heavy usage by the Event Service
Auditing
Check that the existing Rucio auditing code works with objectstores and make any changes necessary
Dynafed
Can be used to make objectstores appear more like traditional grid storage systems
Check that it works properly with Rucio
Already in progress at CERN
Take advantage of objectstores fully, don’t just use as file systems
Store DUNE events as objects
Could implement a QoS prototype using them