1.
Abstract
Title: Experience with running Owncloud on virtualized infrastructure (Openstack/Ceph) (continuation of talk at C3 2016)
Owncloud is quite popular among sync & share service providers. As the name implies, Owncloud was build with home users in mind. It can run on devices as small as a raspberry pi. At the same time this product is also sold to service providers who support with one single Owncloud instance more then 20k users. This already being an astounding achievement, it is not yet good enough. Service providers would need Owncloud to scale up to 100k users or even more.
At the CS3 in Zürich 2016, we presented our service SWITCHdrive. A sync & share service based on Owncloud which we run ontop of our IAAS offering (SWITCHengines) based on Openstack/Ceph. We discussed its advantages but also its limitations.
This year we will present how the service evolved. It grew quite a bit and we replaced the database (from PostgreSQl to Galera cluster). We'll talk about the motivation for that change and what our experience was.
There is still one major pain point left: the NFS servers. We are currently addressing that problem and we will explore possible solutions in our talk.

2. SWITCHdrive Experience with running Owncloud on top of Openstack/Ceph
Experience with running Owncloud on top of Openstack/Ceph
Present SWITCH, team, self…
Simon Leinen
Christian Schnidrig
Amsterdam, January 2017

3. I’m a cloud engineer in the Infrastructure and Data Team @ SWITCH
I’m a cloud engineer in the Infrastructure and Data SWITCH
We offer two services to swiss academia
SWITCHdrive (Sync & Share based on owncloud)
SWITCHengines (IAAS based on Openstack)
We started 3 years ago.
We presented this last year at CS3 in Zürich already.
This year we want to present how the service evolved during 2016.

4. Openstack (Mitaka) 118 servers 4760 logical cores Ceph (Hammer)
Openstack (Mitaka)
Ceph (Hammer)
118 servers
4760 logical cores
24.5 TiB RAM
2.3 PB storage
Our little tiny cloud….
Paid service since

5. ownCloud 8.1 (upgrade 9.0 in March) Users: 24’000 Quota: 50 GiB / user
ownCloud 8.1 (upgrade 9.0 in March)
Users: 24’000
Quota: 50 GiB / user
Number of files: 62M
Storage: 61 TiB
50 TiB live data
7.2 TiB trash
2.2 TiB versions
1.5 TiB …
Paid service since

6. Plan for 2016/7 To that From this architecture 12.04.2018
What we set out to be our goal for SWITCHdrive in 2016/7
To that

7. What we were lacking Jan 2016?
What we were lacking Jan 2016?
Last Slide from last year:
Object store instead of NFS
We had several service outages in 2016 due to NFS.
Sometimes when ceph was slower than normal an NFS server would stall.
When a hypervisor with a NFS server on it died.
DB: scale out with PostgreSQL proves to be very difficult -> migration to Galera/Maxscale.
Scale out with “federated cloud sharing”
We know how to run an OC instance < 20k Users. We can easily run several such instances in parallel. So sharding could help us.

8. Current Status 12.04.2018 The db scale out is done and works fine
NFS replacement with object store: we are investigating, looks good, however there is still a lot to be done.
Sharding: we are planing to get this done in March. (With the help from Owncloud)

9. DB Scale out Standard procedure: (weeks) Custom migration: (4 hours)
DB Scale out
Standard procedure: (weeks)
occ db:convert-type: is ok but very slow
Custom migration: (4 hours)
Edit „converttype.php“: comment out „oc_filecache“
Run occ db:convert-type -> copies schema and all other tables
Manually dump filecache table from postgres & load into mariadb
There was no ‚production ready solution for postgresql horizontal scaling
So we opted to replace Postgresql with Mariadb/Galera.
Mariadb is popular amongst NRENs. Almost noone uses PostgreSQL for Owncloud -> we wanted to go mainstream.
Migration:
Extremely slow with occ db:convert-type -> would have taken weeks to finish.
Custom migration:
Edit „core/command/db/converttype.php“
comment out copyTable for oc_filecache, then run db:convert-type for schema and other tables.
Dump data from postgres
pg_dump -Fp --data-only --column-inserts --disable-dollar-quoting --encoding='UNICODE' --table=oc_filecache -U postgres owncloud –f dump.sql
Load into mariadb:
Cat dump.sql | tail -n +17 | head -n -11 | mysql --batch -u owncloud --password=<passwd> --default-character-set=utf8

10. Performance (measured on load balancer)
Performance (measured on load balancer)
PostgreSQL
master/slave
Mariadb/Galera 3 nodes
maxscale read/write split
Response times were better and we were quite happy with the change.

11. DB-Scalability with Galera
DB-Scalability with Galera
theory
reality
Number of db servers in cluster
Max number of Users it could handle
3 (min with galera)
Up to 22.3 K 4
Up to 23.2 K 5
? K
Theorie: Very little write load -> read can be scaled by adding new slaves -> perfect!
Reality: It is not good enough. We need to add to many server for only a few more users.

12. Sharding (March 2017) Federated Cloud Sharing https://drive.switch.ch/
Sharding (March 2017)
Federated Cloud Sharing
cookie
haproxy
All shards can be accessed through same domain.
Client sends cookie with request
Haproxy nows to which server the client belongs.
We develop this together with Owncloud
Planned deployment is in early March.
redis
In development together with Owncloud

13. Object Gateway oC 9.1 has multibucket support
There are still some obstacles:
plan to upgrade to 9.0 in March. (not 9.1)
migration of existing users -> no out of box solution from oC
backup (maybe take etag from radosgw)
performance:
SWITCHdrive needs: 15 reads/s & 15 writes/s
We need to scale out radosgw first.

14. ? simon.leinen@switch.ch christian.schnidrig@switch.ch ?