1. The Data Bridge Laurence Field IT/SDC 6 March 2015

2. BOINC and Virtualization 2

3. Test4Theory Model  Avoid restarting the VM for every job  Reduces CVMFS related network traffic  CoPilot support challenges  Dependencies not available in the standard repositories  Reduce the operational cost  New standardized components available for some functions  Separation of VM management and job management  Inline with the cloud model  Can reuse cloud related tooling 3 VM BOINC Server Volunteer Agent Job Wrapper Co Pilot Job Agent Storage Agent Job Description Data I/O

4. The Challenge 4 Workload Manager VM Volunteer Job Wrapper Grid

5. Authentication  How to authenticate BOINC users?  In the VM, credential provided via /dev/fd0  BOINC_ID  BOINC_AUTHENTICATOR  BOINC Project DB is the user Identity Provider (IDP)  MySQL  User Table  mod_auth_mysql  Maps username/password to DB table  AuthMysqlUserTable user  AuthMySQLNameField id  AuthMySQLPasswordField authenticator  Enables reuse of apache-based HTTP technology 5

6. The Architecture 6 Workload Manager Messaging Service Messaging Service Data Bridge VM Volunteer Agent Job Wrapper Job Description FTS Grid Pull Push Plugin Data I/O

7. The Data Bridge  Spans authentication domains  BOINC user’s credential  Grid x509 credentials  Scalable data I/O  With sandboxing capabilities  Data isolation  Simple apache-based prototype  Supports HTTP PUT/GET  mod_auth_mysql to validate BOINC user’s credential  mod_auth_ssl to validate WMS x509 credential  HTTP Federation  Possibility to reuse standard DM tools 7

8. Dynamic HTTP Federations  Dynafed implements federated storage over HTTP  In testing in LHCb and Canada (ATLAS)  Federates WebDAV or S3 enabled storage systems  Apache front end  Can be used as a data bridge  S3 storage backend(s)  Acts as a security gateway between X509 or BOINC Auth  Clients then redirected directly to the storage  Great scalability potential  Global system, smart replica selection (availability, proximity)  http://svnweb.cern.ch/trac/lcgdm/wiki/Dynafeds 8

9. Apache The Data Bridge 9 ssl FTS S3 mysql WMS BOINC User BOINC User PUT/GET HTTP redirect & sign PUT/GET Grid DynaFed

10. Message Queue  Messaging service does not support BOINC authentication  Not clear if it is possible or worthwhile to provide functionality  Standard apache Web server approach  mod_auth_mysql to validate BOINC user’s credential  mod_auth_ssl to validate WMS x509 credential  Two simple cgi scripts  put-job.cgi  get-job.cgi  Simple file-based queue  python-dirq  Job descriptions from the WM  Supports arbitrary file types  Garbage in, Garbage out  Extensible 10 Web Server dirq put Get

11. Implementation 11 Workload Manager Message Queue Data Bridge VM Volunteer Agent Job Wrapper GET PUT Plugin PUT FTS S3 Grid

12. Adoption  Building upon vLHC@home and the data bridge as a platform  Require:  WM to POST job description to the data bridge message queue  Stage input data to the data bridge’s input bucket (if needed)  CernVM3 Image  Contextualized including CVMFS configuration  Credentials read from /dev/fd0  BOINC_ID and BOINC_AUTHENTICATOR  Job Agent  GET the job description  GET the input from the data bridge  Run job  PUT output on the data bridge  Read data from the data bridge’s output bucket  Similar to HLT  Fitter bad data etc. 12

13. Rollout Plans  CMS@Home  Pioneered the adoption of the data bridge  Will hopefully enter the beta testing phase soon  Test4Theory  Plan to migrate to the data bridge within the next 2 months  To address co-pilot support issues  Beauty@Home  Currently integrating the data bridge  Will solve their x509 credential distribution issue  Open the project up to the public 13

14. Summary  The Data Bridge spans auth domains  Grid and Volunteer computing  Reuses HTTP federation component for S3  Added BOINC authentication  A simple message delivery function  For the job description  Just provide an image along with a job agent  And interact with the data bridge  Towards a platform for volunteer computing 14