12.04.2018 Abstract Title: Experience with running Owncloud on virtualized infrastructure (Openstack/Ceph) (continuation of talk at C3 2016) Owncloud is quite popular among sync & share service providers. As the name implies, Owncloud was build with home users in mind. It can run on devices as small as a raspberry pi. At the same time this product is also sold to service providers who support with one single Owncloud instance more then 20k users. This already being an astounding achievement, it is not yet good enough. Service providers would need Owncloud to scale up to 100k users or even more. At the CS3 in Zürich 2016, we presented our service SWITCHdrive. A sync & share service based on Owncloud which we run ontop of our IAAS offering (SWITCHengines) based on Openstack/Ceph. We discussed its advantages but also its limitations. This year we will present how the service evolved. It grew quite a bit and we replaced the database (from PostgreSQl to Galera cluster). We'll talk about the motivation for that change and what our experience was. There is still one major pain point left: the NFS servers. We are currently addressing that problem and we will explore possible solutions in our talk.

SWITCHdrive Experience with running Owncloud on top of Openstack/Ceph 12.04.2018 Experience with running Owncloud on top of Openstack/Ceph Present SWITCH, team, self… Simon Leinen simon.leinen@switch.ch Christian Schnidrig christian.schnidrig@switch.ch Amsterdam, January 2017

I’m a cloud engineer in the Infrastructure and Data Team @ SWITCH 12.04.2018 I’m a cloud engineer in the Infrastructure and Data Team @ SWITCH We offer two services to swiss academia SWITCHdrive (Sync & Share based on owncloud) SWITCHengines (IAAS based on Openstack) We started 3 years ago. We presented this last year at CS3 in Zürich already. This year we want to present how the service evolved during 2016. https://www.flickr.com/photos/jimculp/865599033/in/album-72157594310567262/

Openstack (Mitaka) 118 servers 4760 logical cores Ceph (Hammer) 12.04.2018 Openstack (Mitaka) Ceph (Hammer) 118 servers 4760 logical cores 24.5 TiB RAM 2.3 PB storage Our little tiny cloud…. Paid service since 1.1.2016

ownCloud 8.1 (upgrade 9.0 in March) Users: 24’000 Quota: 50 GiB / user 12.04.2018 ownCloud 8.1 (upgrade 9.0 in March) Users: 24’000 Quota: 50 GiB / user Number of files: 62M Storage: 61 TiB 50 TiB live data 7.2 TiB trash 2.2 TiB versions 1.5 TiB … Paid service since 1.1.2016

Plan for 2016/7 To that From this architecture 12.04.2018 What we set out to be our goal for SWITCHdrive in 2016/7 To that

What we were lacking Jan 2016? 12.04.2018 What we were lacking Jan 2016? Last Slide from last year: Object store instead of NFS We had several service outages in 2016 due to NFS. Sometimes when ceph was slower than normal an NFS server would stall. When a hypervisor with a NFS server on it died. DB: scale out with PostgreSQL proves to be very difficult -> migration to Galera/Maxscale. Scale out with “federated cloud sharing” We know how to run an OC instance < 20k Users. We can easily run several such instances in parallel. So sharding could help us.

Current Status 12.04.2018 The db scale out is done and works fine NFS replacement with object store: we are investigating, looks good, however there is still a lot to be done. Sharding: we are planing to get this done in March. (With the help from Owncloud)

DB Scale out Standard procedure: (weeks) Custom migration: (4 hours) 12.04.2018 DB Scale out Standard procedure: (weeks) occ db:convert-type: is ok but very slow Custom migration: (4 hours) Edit „converttype.php“: comment out „oc_filecache“ Run occ db:convert-type -> copies schema and all other tables Manually dump filecache table from postgres & load into mariadb There was no ‚production ready solution for postgresql horizontal scaling So we opted to replace Postgresql with Mariadb/Galera. Mariadb is popular amongst NRENs. Almost noone uses PostgreSQL for Owncloud -> we wanted to go mainstream. Migration: Extremely slow with occ db:convert-type -> would have taken weeks to finish. Custom migration: Edit „core/command/db/converttype.php“ comment out copyTable for oc_filecache, then run db:convert-type for schema and other tables. Dump data from postgres pg_dump -Fp --data-only --column-inserts --disable-dollar-quoting --encoding='UNICODE' --table=oc_filecache -U postgres owncloud –f dump.sql Load into mariadb: Cat dump.sql | tail -n +17 | head -n -11 | mysql --batch -u owncloud --password=<passwd> --default-character-set=utf8

Performance (measured on load balancer) 12.04.2018 Performance (measured on load balancer) PostgreSQL master/slave Mariadb/Galera 3 nodes maxscale read/write split Response times were better and we were quite happy with the change.

DB-Scalability with Galera 12.04.2018 DB-Scalability with Galera theory reality Number of db servers in cluster Max number of Users it could handle 3 (min with galera) Up to 22.3 K 4 Up to 23.2 K 5 ? K Theorie: Very little write load -> read can be scaled by adding new slaves -> perfect! Reality: It is not good enough. We need to add to many server for only a few more users.

Sharding (March 2017) Federated Cloud Sharing https://drive.switch.ch/ 12.04.2018 Sharding (March 2017) Federated Cloud Sharing https://drive.switch.ch/ cookie haproxy All shards can be accessed through same domain. Client sends cookie with request Haproxy nows to which server the client belongs. We develop this together with Owncloud Planned deployment is in early March. redis In development together with Owncloud

Object Gateway oC 9.1 has multibucket support There are still some obstacles: plan to upgrade to 9.0 in March. (not 9.1) migration of existing users -> no out of box solution from oC backup (maybe take etag from radosgw) performance: SWITCHdrive needs: 15 reads/s & 15 writes/s We need to scale out radosgw first.

? simon.leinen@switch.ch christian.schnidrig@switch.ch https://github.com/switch-ch/switchdrive-ansible ? simon.leinen@switch.ch christian.schnidrig@switch.ch oc@lists.geant.org