1. Eos at 6,500 kilometres wide An Australian Experience
David Jericho – Solutions Architect, AARNet

2. We’re geographically large…
Australia is a huge continent
47ms from Sydney to Perth
27ms from Brisbane to Melbourne
90ms from Darwin to Perth
Bandwidth is not the problem
RTT * TCP Window = Goodput
Campus networks
User equipment choices
© AARNet Pty Ltd |

3. …Because we’re spread thin
The dark blue areas show 50% of our 24 million people
98% of the population live within 150 kilometres of the coast line
Collaborations occur with groups thousands of kilometres apart
© AARNet Pty Ltd |

4. Cloudstor sites 3 major sites
Brisbane -> Melbourne is 22ms
Melbourne -> Perth is 43ms
At least two geographically disperse replicas
Network splits do occur despite redundancy
At least 24 servers providing metadata, application, storage and additional compute in each site
This is excessive for our current needs
It is a large number of servers for a small team
© AARNet Pty Ltd |

5. Solutions we have tried
Hadoop
MapR, Hortonworks, Apache official
XtreemFS
Ceph
GlusterFS
pNFS
OrangeFS
… and others
© AARNet Pty Ltd |

6. Our primary EOS cluster
© AARNet Pty Ltd |

7. EOS clusters in AARNEt CloudStor CloudStor support infrastructure
2.5PB presently
12 machines
Not homogenous
40Gbps connected each
44 or 36 disks per machine
Multiple MGMs
Encrypted at rest
Only 4% of files are larger than 10 megabytes
CloudStor support infrastructure
Uses Seagate Kinetic FSTs
Connects back to developer’s workstations
Content Delivery Network
6 FSTs
48 TB, 12 disks each
Acts as a canary for CloudStor
Extremely read heavy with only one write client
Test CloudStor
Only 30TB
3 machines
Treated as entirely disposable
Interest in using EOS elsewhere too
© AARNet Pty Ltd |

8. CloudStor uses EOSD Fastest way to production EOS usage
Everything understands files
eosd has had many improvements since we’ve been using it
> has been almost entirely trouble free
48 web servers
eosd runs in foreground within a container
Volume mapped /eos to other containers
© AARNet Pty Ltd |

9. Completely containerised
Via Docker Engine
All containers bind to the host’s network stack
Orchestrated by Rancher
Each FST geotag is its own pod/stack
MGM Master is own pod/stack
Slaves all reside in the same pod/stack
Each component containerised
MGM, MQ, Sync, eossync on same host
FSTs also run on MGM hosts as a container
FSTs mount encrypted storage inside the container
eosd running foreground in container
Extremely quick to deploy and very consistent
© AARNet Pty Ltd |

10. Successes we’ve had IT WORKS! Obvious write latency penalty
Stable, server issues have been almost exclusively container related
Fast
Obvious write latency penalty
Users don’t notice
Hello all, I know it’s Monday…
CERN have been very responsive, THANKYOU!
© AARNet Pty Ltd |

11. observations IO performs at disk speed IO even at 65ms
Does require suitable tuning and stable network
Latency can be obvious with many small files
“Square” groups can mask this through average performance
Xroot latency is insignificant with respect to network latency
Moving towards direct xroot manipulation for IO
FileSender tool streams concatenated chunks
Mass ingest tool for ownCloud environment
GeoTag functionality works well
Will eventually result in imbalanced FST counts in some sites
This isn’t a problem
© AARNet Pty Ltd |

12. Problems along the way eosd has matured a lot FST start up can be slow
Many user space tools do silly things (rsync)
FST start up can be slow
Mostly resolved now
Better group/fs layouts has helped
Documentation is rare
Word of mouth training
Internal documentation
© AARNet Pty Ltd |

13. Possible improvements
Bundling of balancing transfers
Many small files limits speed of balancing
UDT or similar protocol for FST communications
Or at least a software tunable window size
Latency sensitive geotag reading
AARNet contributed test infrastructure for Citrine
Help with maturity of software in non-CERN use cases
© AARNet Pty Ltd |

14. All images CC0 unless otherwise noted