1. sRoute: Treating the Storage Stack Like a Network
Authors: Iona Stefanovici, Bianca Schroeder, Greg O’Shea, and Eno Thereska
Presented by: Arnav Mishra

2. Agenda Introduction/Background Routing Types & Challenges Design
sSwitches
Controllers
Fault Tolerance
Implementation
Control Applications
Tail Latency
Replica Set
File Cache
Conclusion
Discussion Points

3. Introduction Current Process: Solution:
IO routing doesn’t exist in storage stack
Endpoints are predetermined, rigid structure
Solution:
sRoute = architecture for routing abstraction from storage stack
Uses centralized control panel + sSwitches to route IO requests
10x better latency for tail IO
60% better throughput for customized replication protocol
2x better throughput for customized caching

4. High Level Ideas/Summary
Bringing dynamic routing to IO
Components
Control Panel: Sets forwarding rules and sSwitch locations
sSwitches: Execute forwarding rules on IO as it comes through
Modifies endpoints and paths for improved IO Performance
Sends Reads/Writes to less loaded servers

5. Background
IO Routing is nearly impossible with current systems: pre-determined destinations, static movement channels
Goal:
Dynamically re-route read/write
Send write to less loaded servers
Send read to latest data
Challenges
Keeping consistency with changing endpoints for read/write
Minimizing number of times IO has to traverse sSwitches
IOFlow Currently:
Bypasses stages on IO Path
Cannot change endpoints/path

6. Types of IO Routing Endpoint Waypoint Scatter
From Source P to File X routed to File Y
Improves tail latency, copy-on-write, file versioning, and data re-encoding
Dynamically changes endpoint of new data
Waypoint
From Sources P and R to File X routed via special stage W
Inject specialized waypoint routing to selectively route IO to correct stages
Scatter
From Source P and R to subset of data replicas
Enables specialized replication and erasure coding policies
Dynamically chooses endpoint to read/write from

7. Visual Representation of Routing

8. Routing Challenges
Prevent system-wide configuration updates from making system instability
sSwitches can’t send IO through incorrect paths
Some applications require reads to follow writes
Must keep track of last data’s location/path
Some file systems require knowing destination endpoint of IO
Must maintain file system functionality and semantics in storage stack
Must maintain a low overhead in adding dynamically changing direction to IO

9. Design
Notice the sSwitch’s control over IO direction in the new architecture
Tenants runs vms/containers with IOFlow
sRoute built on top of IOFlow
sSwitches used via 4 API calls: Insert, Delete, Quiesce, and Drain
Control Panel gives sSwitches locations and forwarding rules

10. sSwitches Forwards IO via rules from Control Panel
Rule = IO Header + Action/Delegate Function
IO Packets matched with IO Header  First matched rule is executed
Responses don’t have to be routed same was as requests
Takes care of file connections, handling namespace conflicts
Open/Create are expensive operations: tradeoff = open all files at start and have empty files vs open when IO begins and have expensive first IO

11. sSwitch API Calls

12. Controllers
Tenant policies changed into sSwitch API calls and sent as forwarding rules
Identifies locations for sSwitches (hypervisors/storage servers/etc.)
Control Delegates
Restricted Control Panel at sSwitches
Increases efficiency by making control decisions
Can only make API calls, no changes to IO Header/Data (security issue)

13. Consistency Per-IO Per-Flow
IO flows through either old rules or updated rules, not hybrid mix
Taken care of by quiesce and drain calls to queue
Per-Flow
Read requests sometimes need data from last write request
Not an issue if both read/write come from same source
Solutions:
Central controller updates forwarding rules after each write (inefficient)
Insert sSwitches with delegate functions near sources

14. Fault Tolerance Loss of controller can’t affect data integrity
Would cause slower IO
Controllers can choose to replicate forwarding rules
Rules created at delegates either sent to central controller or forwarded as a header on IOs
If sSwitch on Kernel fails, entire server could fail due to added code

15. Limitations sRoute lacks verification tools
Currently possible for controller to route IO to arbitrary, incorrect locations causing data loss
Storage controller doesn’t have access to network
Storage Controller and Network Controller could send data to different places

16. Routing Implementation
sSwitch in kernel (in C) and in user-level (in C#) – total of 25 kLOC
Within Server’s IO Stack:
Uses filter driver architecture of Windows
Kernel attaches control code to beginning of filter driver
Bypassing is done by returning early from driver
Across Remote Servers:
Kernel sSwitch sends IO to user-level sSwitch
Transmits to remote location using TCP/RDMA
sSwitch on remote server intercepts arriving packet

17. sSwitch Implementation
sSwitch = stage that is identified by host name and driver name
Driver identified by device driver name, device name, altitude
sSwitch stores files in ”sroute-folder” directory to decrease namespace conflicts
Limitations:
sSwitches can’t intercept individual IO to memory mapped files
Does not support byte-range file locking for multiple accesses to same file
Intercepting IO response in Windows is costly =

18. Control Applications Testbed 3 Workloads
12 servers, running 16 Intel Xeon 2.4 GHz cores
Run Windows Server 2012
Act as Hyper-V or Storage Servers
3 Workloads
Unmodified SQL server with small IOs
Public IO trace from Exchange server
IoMeter

19. Tail Latency Control
IOs often arrive in bursts; load balancing must be implemented
Endpoint Routing
Detects extra load by comparing average receive rate with current receive rate
Forward reads to less loaded servers where data is up to date
If no writes happen, everything is still latest, send to Vmmax
When spikes end:
Trigger data reclaim to point to original VHDmax
Rules are deleted, sSwitches removed
CPU overhead < 1%
Much better latency at Smax, little change at Smin

20. IOs Arrive in Bursts
Notice the peak in the request rate.

21. Latency Improvement with Tail Latency Control
Notice the huge decrease in latency for high # of IOs with sRoute

22. Replica Set Control
Write-Set/Read-Set = Num of servers to contact for a read/write request
Workload = Read-Only  read-set should be only replicas of primary server
Writes in Workload  all requests happen at primary server which writes to write set
After 30 seconds if no writes:
sSwitch inserted to send reads to randomly chosen servers using control delegate
If write received, reset to write stream
Latency overhead for first write can be reduced with higher time interval before assuming read-only

23. Latency Improvement with Tail Latency Control
Notice the throughput jumps whenever write requests end
Notice the latency spike when the first write comes in to notify the controller

24. File Cache Accessing data in cache >3x faster than accessing disks
Filter rules gain explicit control over customizing cache size, evictions, and write policies
IO is cached if header matches installed filter rule
Cache isolation: Controller sets sizes for each cache, sSwitches isolate them
sRoute can forward to or bypass caches
sRoute improved throughput 57%, performance 2x in test
Throughput increases when both workloads run because they are isolated
Caches don’t overwrite each other (one process won’t take cache from another)
With IoMeter, controller gives it ¾ of cache  40% cache hit ratio
File Cache Control
accessing data in cache is >3x faster than accessing disks
benefit from customized cache size, eviction, and write policies —> explicit control can be gained via filter rules
cache isolation: controller determines size of cache for each tenant, the sSwitches isolate each cache
sRoute controls IO and can forward to a cache or bypass a cache
with sRoute (figure 10b) throughput increases, but with today’s caching it drops (improved throughput by 57% with sRoute, improved performance 2x)
when IoMeter runs, controller gives it 3/4 of cache resulting in 40% cache hit ratio

25. Conclusion Routing is inherently programmable and dynamic
Substituting hard-coded implementation for common routing core improves performance
Final restated hypothesis:
Routing is inherently programmable and dynamic —> substitute hard-coded implementation with common routing core

26. Piazza Discussion Points
No detailed discussion of consistency concerns with path/endpoint changes
Agreed, something that they mention needs to be researched further
Control Plane as single point of failure
Slide 14: Losing control panel won’t make system worse than current line
Added overhead from managing hops/security code/etc.
CPU added overhead = 1% (no further discussion)
Installed priorities could conflict with network priorities
“desirable to have a control plane that understands both the network and storage” (203)
No discussion of exponential growth of routing rules
No mentions of this
Does the overhead make this better for scenarios with a lot of IO
Slide 20, latency improved at all levels

27. Questions?