1. OpenStack Swift Where do big data go? Eben van Zyl
21 April 2016
Eben van Zyl
Business System Architect
North-West University

2. OpenStack Swift
“If South Africa can develop a reservoir of data science talent, the world will look here for the next wave of growth.”
- Sunil Geness, The Information Technology Association of South Africa

3. OpenStack Swift
What is the ARC?
ARC provides flexible scalable computing power to researchers, with computing infrastructure, software and services that allow the research community to store, access and run data, remotely, rapidly and autonomously.
ARC’s self-service structure allows fast, efficient desktop access to data and a powerful platform for collaboration.

4. OpenStack Swift
Data Sources
IDIA - The Inter-University Institute for Data Intensive Astronomy
Will support SKA, the world’s largest public science data project
Once completed the SKA will generate data at a rate more than ten times today’s global internet
Bioinformatics
Advancements in the field including next generation sequencing
Existing research data that goes missing
Data store for existing applications (Sakai / Matterhorn)

5. OpenStack Swift

6. OpenStack Swift

7. Data Storage Options RAID / VSA / 3PAR
OpenStack Swift
Data Storage Options
RAID / VSA / 3PAR
Fast, expensive, fairly reliable, long rebuild times, limited capacity
Ceph
Cloud based file, block and object storage
OpenStack Swift
Cloud based object storage

8. Advantages of Swift Extremely scalable Lower capital expenditure cost
OpenStack Swift
Advantages of Swift
Extremely scalable
Lower capital expenditure cost
Heterogeneous commodity hardware
Lower operational costs
New capabilities - API driven
By design a distributed system – eventual consistency
Supports geofencing

9. OpenStack Swift

10. OpenStack Swift
A Swift Cost Use Case
Purchased 12 servers – 9 Storage servers, 3 ring nodes
162 8 TB HDD’s
Total cost R2,9m
Delivered 1,2PB Storage, 400TB usable storage
Cost 19c / GB / month
Expansion with 162 8TB R1,3m – 800TB usable 14c /GB / month

11. Thank You