1. Flat Datacenter Storage
Edmund B. Nightingale, Jeremy Elson, and Jinliang Fan, Owen Hofmann, Jon Howell and Yutaka Suzue
Presented by Kuangyuan Chen, Qi Gao
EECS 582 – W16

2. Outline Motivation Overview Distributed Metadata Management
Dynamic Work Allocation
Replication and Failure Recovery
Performance Evaluation

3. Motivation move computation to data?
because of bandwidth shortage in the datacenter
e.g. MapReduce
locality constraints hinder efficient resource utilization
stragglers, retasking
As we have seen in the mapreduce paper, we learned that network bandwidthis a scarce resource.
This restriction leads to the design of moving computation to data in the MapReduce paper.
However, this optimization is not free.
We gain performance through locality at the expense of efficient resource utilization.
One example is straggler. If there exists a slow machine, the entire job cannot complete until the slow machine finishes, while most of other machines are idle.
The common solution is to re-execute the task in the other machine. But because of the locality constraints, we have to move the data first before we can restart the task on another machine.
This is expensive but worthwhile if network bandwidth is a scarce resource. But what if datacenters are no longer in shortage of network bandwidth.

4. Motivation CLOS network supports full bisection bandwidth
CLOS network can provide full bisection bandwidth, which make datacenter bandwidth abundant.
A typical CLOS network looks like this.
On the two sides are Top of Rack Routers, and each of them are connected to many spine routers in the middle.
Network traffic can be balanced between different spine routers, thus providing full bisection bandwidth.
Top of Rack(TOR) routers
spine routers

5. Flat Datacenter Storage
All compute nodes can access all data with equal throughput
Simple and easy to program
All data operations are remote.
All machine have as much network bandwidth as disk bandwidth.
Flat datacenter storage is based on such an assumption that network bandwidth is abundant.
In FDS, all compute nodes can access all data with equal throughput.
Therefore, applications are programmed without the consideration of locality.
More specifically, all data are stored in remote servers , and network bandwidth is as much as disk bandwidth for each machine.

6. Overview Logically Centralized Storage Array Blob Tract FDS API
Per-Blob
Metadata
8MB
Blob 0x5fab97ff da5c7c00:
Tract -1
Tract 0
Tract 1
...
Tract N
Logically Centralized Storage Array
Blob
byte sequence named with GUID
Tract
unit of read and write
constant sized (e.g. 8MB)
FDS API
e.g. CreateBlob(), WriteTract(), ReadTract()
asynchronous/non-blocking: can be issued in parallel
Basically, FDS provides the abstraction of a logically centralized storage array.
Data are logically stored in blobs with a global unique identifier.
A blob contains a variant number of tracts.
Reads and writes are done in the unit of tracts, which are fixed-size and 8MB in this paper.
Users interact with FDS through a set of APIs provided by FDS and
these APIs are non-blocking,
they can be issued in parallel and the underlying storage system can process these request concurrently.

7. Distributed Metadata Management
Tractserver
a process that manages a disk
lay out tracts on the disk directly
using raw disk interface
Tract Locator Table (TLT)
a list of active tractservers
Tract_Locator = (Hash(GUID) + i) mod TLT_Length
deterministic, and produce uniform disk utilization
One key feature of FDS is its distributed metadata management.
There are several components cooperating to achieve distributed metadata management.
A tractsever is a process residing with a disk.
It manages the disk and services the reads and writes from clients.
Tracts in the disk are accessed by the tractserver directly through raw disk interface.
So if a client want to read a tract, in addition to the tract number, the only thing it need to know is the address of that specific tractserver.
Such information is kept in a data structure called tract locator table.
A tractserver can be located by computing the tract locator, which is a hash function of blob’s GUID and tract number.
One thing to notice is that the process of finding a tractserver is deterministic. This means we don’t have to consult a centralized metadata server every time we issue read and write.
And this eliminates the bottleneck bottleneck a centralized metadata server.
Moreover, hash function is used to randomize data accesses among different tractservers , and thus improving the disk utilization.

8. Tract Locator List Locator(Row) Version Disk 1 Disk 2 Disk 3 122 A G H
122 A G H 1 5 B D F 2 6 C T 3 E V 4
373 R
...
TLT_Length
160 U I
Tractserver Address
Here is an example of the tract locator list.
We compute the row number using the hash function. Within each entry, there are the addresses of trackserver that contain the tract.
there might be multiple tractservers in an entry, which serves as replication.
The version number in each entry is for failure recovery, which Qi will introduce later.
As you can see, TLT only keeps information about tractservers, so size is relatively small which makes such a design scalable.
Tract_Locator = (Hash(GUID) + i) mod TLT_Length
Tractserver versioning for failure recovery

9. Distributed Metadata Management(cont)
Metadata Server
create TLT by balancing among tractservers
distribute TLT to clients
assign version number to tractservers
in critical path only at client startup
tract locator table is created and managed by the metadata server.
And it will distribute TLT to clients when they start.
Since each client has a copy of TLT, and in normal cases this table will not change,
further read and write operations can go directly to tractservers without going to the metadata server.
So the metadata is essentially distributed and clients can fully utilize the network bandwidth.

10. Dynamic Work Allocation
mitigate stragglers
decouple data and computation
assign work to workers dynamically and at fine granularity
reduce dispersion to time of a single work unit
Another important feature is dynamic work allocation.
Since now data and computation are decoupled, retasking have very low cost and then we can retask work with higher flexibility.
FDS divides work into small units and assigns them to clients dynamically.
It assigns a unit to a client upon the completion of the previous unit.
So even if there is a straggler, only a small number of work units will be run on that straggler.
So the effect of stragglers will not be significant.
Next, Qi will talk about replication and failure recovery.

11. Replication
When a disk fails, redundant copies of the lost data are used to restore the data to full replication.

12. Replication
As long as the lost data tracts are restored somewhere in the system, we are good.

13. Replication All disk pairs appear in the table O(n^2) table size
When a disk fails, the lost data can be recovered using the rest of disks in parallel
Locator
Disk 1
Disk 2
Disk 3 1 A B C 2 Z 3 D H 4 E M 5 F Y 6 G
...
1234 W Q
1235 X
1236 U

14. Failure Recovery - Metadata Server
Increment the version number of each row in which the failed tractserver appears
Pick random tractservers to fill in the empty spaces in the TLT
Sends updated TLT assignments to every server affected by the changes
Wait for each tractserver to ack the new TLT assignments, and then begins to give out the new TLT to clients when queried for it
Locator
Version
Disk 1
Disk 2
Disk 3 1 8 A B C 2 17 Z 3
324 D H 4 E M 5
456 F Y 6 7 G
...
1234 W Q
1235 43 X
1236 U
Locator
Version
Disk 1
Disk 2
Disk 3 1 9 A B C 2 18 Z 3
325 D H 4 E M 5
457 F Y 6 8 G
...
1234 W Q
1235 43 X
1236
324 U M R T Y U O

15. Failure Recovery - Tract Server
When a tractserver receives an assignment of a new entry in the TLT, it contacts the other replicas and begins copying previously written tracts B M
Locator
Version
Disk 1
Disk 2
Disk 3 1 9 A B C 2 18 Z 3
325 D H 4 E M 5
457 F Y 6 8 G
...
1234 W Q
1235 43 X
1236
324 U M R T Y U O C R D T

16. Failure Recovery - Client
All client operations are tagged with TLT entry version number
Client
Tract Server
Single tract server failure
Multiple tract server failure
Metadata server failure/metadata & tractserver failure concurrently
Metadata Server

17. Evaluation

18. Evaluation
Failure recovery time reduces with more disks!

19. Evaluation
Question:
Speed gain breakdown between full bisection bandwidth and FDS?

20. Conclusion Flat storage provides simplicity for applications.
Deterministic data placement enables distributed metadata management.
Without locality constraints, dynamic work allocation increase utilization.
Highly scalable and fast failure recovery

21. Q&A