Eos at 6,500 kilometres wide An Australian Experience David Jericho – Solutions Architect, AARNet

We’re geographically large… Australia is a huge continent 47ms from Sydney to Perth 27ms from Brisbane to Melbourne 90ms from Darwin to Perth Bandwidth is not the problem RTT * TCP Window = Goodput Campus networks User equipment choices © AARNet Pty Ltd |

…Because we’re spread thin The dark blue areas show 50% of our 24 million people 98% of the population live within 150 kilometres of the coast line Collaborations occur with groups thousands of kilometres apart © AARNet Pty Ltd |

Cloudstor sites 3 major sites Brisbane -> Melbourne is 22ms Melbourne -> Perth is 43ms At least two geographically disperse replicas Network splits do occur despite redundancy At least 24 servers providing metadata, application, storage and additional compute in each site This is excessive for our current needs It is a large number of servers for a small team © AARNet Pty Ltd |

Solutions we have tried Hadoop MapR, Hortonworks, Apache official XtreemFS Ceph GlusterFS pNFS OrangeFS … and others © AARNet Pty Ltd |

Our primary EOS cluster © AARNet Pty Ltd |

EOS clusters in AARNEt CloudStor CloudStor support infrastructure 2.5PB presently 12 machines Not homogenous 40Gbps connected each 44 or 36 disks per machine Multiple MGMs Encrypted at rest Only 4% of files are larger than 10 megabytes CloudStor support infrastructure Uses Seagate Kinetic FSTs Connects back to developer’s workstations Content Delivery Network 6 FSTs 48 TB, 12 disks each Acts as a canary for CloudStor Extremely read heavy with only one write client Test CloudStor Only 30TB 3 machines Treated as entirely disposable Interest in using EOS elsewhere too © AARNet Pty Ltd |

CloudStor uses EOSD Fastest way to production EOS usage Everything understands files eosd has had many improvements since we’ve been using it > 0.3.222 has been almost entirely trouble free 48 web servers eosd runs in foreground within a container Volume mapped /eos to other containers © AARNet Pty Ltd |

Completely containerised Via Docker Engine All containers bind to the host’s network stack Orchestrated by Rancher Each FST geotag is its own pod/stack MGM Master is own pod/stack Slaves all reside in the same pod/stack Each component containerised MGM, MQ, Sync, eossync on same host FSTs also run on MGM hosts as a container FSTs mount encrypted storage inside the container eosd running foreground in container Extremely quick to deploy and very consistent © AARNet Pty Ltd |

Successes we’ve had IT WORKS! Obvious write latency penalty Stable, server issues have been almost exclusively container related Fast Obvious write latency penalty Users don’t notice Hello all, I know it’s Monday… CERN have been very responsive, THANKYOU! © AARNet Pty Ltd |

observations IO performs at disk speed IO even at 65ms Does require suitable tuning and stable network Latency can be obvious with many small files “Square” groups can mask this through average performance Xroot latency is insignificant with respect to network latency Moving towards direct xroot manipulation for IO FileSender tool streams concatenated chunks Mass ingest tool for ownCloud environment GeoTag functionality works well Will eventually result in imbalanced FST counts in some sites This isn’t a problem © AARNet Pty Ltd |

Problems along the way eosd has matured a lot FST start up can be slow Many user space tools do silly things (rsync) FST start up can be slow Mostly resolved now Better group/fs layouts has helped Documentation is rare Word of mouth training Internal documentation © AARNet Pty Ltd |

Possible improvements Bundling of balancing transfers Many small files limits speed of balancing UDT or similar protocol for FST communications Or at least a software tunable window size Latency sensitive geotag reading AARNet contributed test infrastructure for Citrine Help with maturity of software in non-CERN use cases © AARNet Pty Ltd |

All images CC0 unless otherwise noted