SYNC & SHARE FOR THE DUTCH RESEARCH & HIGHER EDUCATION SURFdrive

How it al started about a year ago We want our own

How it al started about a year ago But, Privacy of data is important. No NSA allowed! Dutch institutes agreed on security & privacy rules/guidelines. Large number of whishes and requirements.

Plan Select a product in the autumn of 2013 to do a pilot. Do a pilot in Q Production april 1 st 2014 (no joke).

Important requirements Federated AuthN (SAML2, SURFconext) Sync clients: Offline access to files. Mobile clients. Sharing data with people inside the SURFconext community but also people outside this community.

And the winner is…. Owncloud Enterprise Edition, version 6.0.x Branded Clients. Shibboleth plugin already available. Pilot proofed that the feature set was a good starting point.

Federated Login using SURFconext Institutes manage the identity. SURFdrive manages the authorization. Transparent for the users.

Pilot Period Due to lack of time two tests simultaneously:  Pilot environment with about 250 users from Dutch universities. Mainly functionality tests.  Test environment for:  Stress testing and reliability tests.  Wrote a webdav-based test suite.  1000 concurrent reads, 100 concurrent writes.  Different configuration scenario’s for storage, webservers etc.

The Result: Technical Components Proxy:  HAproxy, v1.5 with SSL  Keepalived for IP failover Application servers:  Apache + Shibboleth  Owncloud Application  GlusterFS server+client Database:  MariaDB  Galera Master/Master replication Proxy:  Dell R415, 2x 3.1 GHz Opteron  16GB memory & 10GB NICs Application servers:  Dell R515, 2x 3.1 GHz 6C Opteron  128GB memory & 10GB NICs  40 TB RAID6 SATA Storage Database:  Dell R515, 2x 3.1 GHz 6C Opteron  64GB memory & 10GB NICs  400 GB RAID10 SSD storage

Technical Choices Proxy:  HAproxy, because it’s fast and low on resources. Application servers:  Apache, because of Shibboleth module support.  NGINX & Lighttpd are still considered because they are more robust and less memory hungry.  GlusterFS because we can easily mount it on all servers, also there were already some positive experiences and Object Storage was not supported. Database:  Galera makes failover so easy, and it’s master/master.

GlusterFS: All data twice !! Server 3 Server 4 Server 2 Server 1 Replicated Volume 0 Replicated Volume 1 Distributed-Replicated Volume Brick01 File 1 File 2

SURFdrive The Numbers We went live on 1st April Now we have 36 institutes connected. Besides Universities also: Higher Education. Research Institutes. Medical Centres (pilot). New institutes get connected each month. New users login each day. Last night we were at 2413 users.

SURFdrive Users

SURFdrive Storage Usage (TB)

SURFdrive Storage Usage per User (GB)

SURFdrive Number of Files

SURFdrive Daily Concurrent Connections Diner Do a few things after dinner Sleep End of the day Start of the day Lunch

SURFdrive Daily Concurrent Connections

Issues Database issues related to Galera replication and the absence of primary keys in some tables. Owncloud Client (sync clients as well as mobile clients) -Login issues related to Shibboleth implementation -Synchronisation issues/bugs -GUI Issues/bugs

Custom Implementation of Webdav Federated login using shibboleth breaks WebDAV. HTTP basic auth with webdav to allow access to SURFdrive using third-party apps. Implemented by Owncloud on our request: -Generated Password -Revoke Password -Visible Once!

Roadmap Provisioning Portal -Institutes have to de-provision the users themselves; they are control. Disaster Recovery -Secondary site as full copy of production. -Galera MariaDB over WAN. -GlusterFS Georeplication. Storage -Object-based (SWIFT). -Geographical distribution. Security -Two factor Authentication; part of SURFconext roadmap. -Client-side encryption.

More Roadmap

Service summary Status:Prod Number of users (current, target):2413|20k+ Default and Maximum quota:100GB and 100GB Linux/Mac/Win user ratio:6% / 18% / 76% Desktop clients/Mobile Clients/Web access ratio: 99.3%, 0.2%, 0.5% Technology:Owncloud Target communities:Universities, Institutes, Higher Education Integration in your current environment (examples): Not at all Risk factors:financial Most important functionality:Sync and share, federated login Missing functionality (if any):Admin functionality like accounting, simultaneous document editting

User feedback Very little actually