1. Elastic Consistent Hashing for Distributed Storage Systems
Wei Xie and Yong Chen
Texas Tech University
31st IEEE International Parallel & Distributed Processing Symposium 
May 29 – June 2, 2017
Buena Vista Palace Hotel
Orlando, Florida USA

2. Elastic Distributed Data Store
Elasticity means resize (the number of active servers) dynamically as workload varies (to save power)
More difficult than elastic computing
Reduce power consumption (cost), or improve resource utilization

3. How Resizing Works Power down last 3,
need to migrated red and blue data
(otherwise they are not available)
3-way replication, 10 server, same color represents same data
Last 3 servers can be powered-down now
Data re-replicated
Power on 8th server
Data re-integration

4. Resize Agility is Critical
The resizing agility (how fast resize can take place)
Determines how good the workload variation is tracked
Xu et al. SpringFS: Bridging Agility and Performance in Elastic Distributed Storage

5. Resize Footprint Should be Small
If resizing introduces lots of IO workload, the number of server needed to be active increases
The longer and more intensive the resize IO, the less machine hour savings

6. What Determines Agility and Resize Footprint?
Size down: need to migrate data to avoid data on the powering-down servers unavailable. Agility
Size up: need to migrate data to keep the data on the powering-on servers up-to-date. Resize footprint

7. Consistent Hashing based Distributed Storage Systems
Distributed hash table/database:Cassandra, Dynamo, Voldemort, Riak
Distributed data Storage/file system: GlusterFS, Sheepdog, Ceph (uses a variant of CH)
Advantage:
Scalability
Add or remove server only requires small data to be moved

8. Elasticity with Consistent Hashing
Consistent hashing is good fit for achieving elasticity
Resize with small data movement
Size-down: move the data on the deactivating server to other servers
Size-up: move data back to the activating server
Can we do better?

9. Elasticity with Consistent Hashing
Can we do better?
Size-down: how about no data movement?
Size-up: how about only moving modified data？
How to do it?
Make sure data always has copies on active servers
Need to know what data are modified

10. Size-down with No Movement
The problem is to prevent the servers being power-down to be a copyset of data.
Copyset: a combination of servers that contain the replicas of a data item
E.g., {1, 2, 3}, {2, 3, 4}, {3, 4, 5}

11. Size-down with No Movement
Solution 1: shutdown those servers that do not make copysets, e.g. shutdown {1, 5, 9}
Limitation: CH usually uses virtual node for load balance that significantly increases the number of copysets
Solution 2: change data layout
Reserve some servers that all copysets have one of these servers and they never shut down

12. Non-elastic Data Layout
A typical pseudo-random based data layout as seen in most CH-based distributed FS
Almost all server must be “on” to ensure 100% availability
No elastic resizing capability

13. Elastic Data Layout General rule Take advantage of replication
Always keep the first (primary) replicas “on”
The other replicas can be activated on demand

14. Primary Server Layout
Peak write performance: N/3 (same as non-elastic)
Limited scaling to N/3

15. Equal-work Data Layout
To scale even smaller (less primary servers), each server will have different size
To achieve proportional performance scaling (performance ~ num_active_servers), an equal-work data layout needed
Servers may need different capacity to ensure balanced storage utilization

16. Primary-server Layout with CH
Modifies data placement in original CH so that one replica is always placed on a primary server
To achieve equal-work layout, the cluster must be configured accordingly

17. Equal-work Data Layout
Number of data chunks on primary:
Number of data chunks on secondary:

18. Data Re-integration
After a node is turned down, no data will be written to it. When this node joins again, any newly created data/modified data might need to re-integrate to it.

19. Data Re-integration
Data re-integration incurs lots of I/O operations and degrades performance (or require extra machines) when scaling up
3-phase workload: high load -> low load -> high load
No resizing: 10 servers always on
With resizing: 10 servers -> 2 servers -> 10 servers

20. Selective Data Re-Integration
Each resize is associated with a version and a membership table
Dirty table to track all OIDs that are dirty
When re-integration finishes, OID is removed from table
The rate of re-integration is controlled by issuing number of OIDs per minute

21. Implementation
Primary-secondary data placement/replication implemented in Sheepdog
10 Sheepdog servers, 1 client server
Manually resize the cluster by killing & starting sheepdog process
Sheepdog uses Corosync to manage membership
Dirty data tracking implemented using Redis

22. Evaluation
Without selective re-integration, the size-up has a delay, which indicates that it requires powering up more servers to achieve the same performance (which means less machine hour savings)
With selective re-integration, the size-up delay is much less
The experiment indicates the effectiveness but it is small scale

23. Large-scale Trace Analysis
Use the Cloudera trace (CC-a: <100 machines, 1 month, 69TB; CC-b: 300 machines, 9 days, 473TB)
Apply our policy and analyze the effect of resizing
Number of servers needed for workload ideally
Add delays by clean-up
Add number of servers to compensate for re-integration workload

24. Machine hour compared to the ideal resizing
Machine Hour Saving
Machine hour compared to the ideal resizing
Trace
Original CH
+ full
+selective
CC-a
1.32
1.24
1.21
CC-b
1.51
1.37
1.33
Elastic layout gives 6% to 10% extra saving in machine hours
Selective data reintegration gives 2% improvement

25. Summary
We propose primary-secondary data placement/replication scheme to provide better elasticity in consistent hashing based data store
We use selective background data re-integration technique to reduce the I/O footprint when re-integrating nodes to a cluster
First work studying elasticity for saving power in consistent hashing based store
Further study the problem that the capacity may be full when scaling down
Testing in larger cluster configurations

26. Welcome to visit our website for more details.
Questions!
Welcome to visit our website for more details.
DISCL lab:
Personal site: