Flamingo: Enabling Evolvable HDD-based Near-Line Storage Sergey Legtchenko, Xiaozhou Li, Antony Rowstron, Austin Donnelly, Richard Black

Storing Cold Data in the Cloud Pelican rack (Microsoft, [OSDI 14]) 8% HDDs active 576 HDDs/server Open Compute Cold Storage rack (Facebook) 6% HDDs active 240 HDDs/server Custom racks: trading latency for cost Only fraction of HDDs concurrently active Reduced #servers (1 or 2 per rack) Cold data: rarely accessed data Challenge: storing cold data at low cost Benefits Lower capital cost Capped resource consumption Higher storage density

Designing Cold Storage Racks is Hard Resources are constrained in the rack Resource constraints: 1 HDD / cooling column 2 HDDs / tray Vibration, bandwidth… Power Cooling Software: co-designed, constraint-aware Data Layout IO Scheduler Experience from building Pelican Design complexity Storage stack is brittle to design changes Impact of resource provisioning on end performance? Pelican: 8% HDDs active

Flamingo: a Tool to Help Cold Storage Rack Design In the rest of the talk Input Generic Storage Stack Data layout IO scheduler parameters Online Constraint Solver Data layout, IO scheduler + Storage stack configuration Set of rack descriptions: same topology varying resource provisioning Perf. Analysis (simulator) Rack description Constraints Hardware properties Resource provisioning exploration Performance goals Rack Resource provisioning specification

Rack Description Resource Domain: set of HDDs sharing a limited resource {D1,D2,D3,D4}: 40 {D5,D6,D7,D8}: 40 {D4,D8}: 1 {D3,D7}: 1 {D1,D5}: 1 {D2,D6}: 1 Domain A: type: power, budget: 40W Domain B: type: vibration, budget: 1HDD D1 D2 D3 D4 D5 D6 D7 D8 Standby Active Spin up power: 10W vibration: 1 power: 20W power: 2W vibration: 0 HDD: operating states + resource consumption IO-capable state Expresses constraints: e.g. only 1 HDD can be spinning up in A 40W 2 + 20+ 20 = 44W 40W 2 + 20 = 26 can be hard or soft

Storage Stack Configuration Data layout Groups of HDDs that concurrently transition state Minimize inter-group conflicts Constraint Solver Rack description

Storage Stack Configuration Data layout Groups of HDDs that concurrently transition state Minimize inter-group conflicts Generic Storage Stack Data Layout Spin ups/downs, IOs   Blob-store API IO Scheduler Blob Group of HDDs Constraint Solver D1 D2 D4 D5 D6 D7 D8 D3 Conflict Conflicts minimized vibration, budget: 1 power budget: 40 G1: {D3, D8} G2: {D4, D7} G3: {D1, D6} G4: {D2, D5} Group definition D1 D2 D4 D5 D6 D7 D8 D3 vibration, budget: 1 power budget: 40 Each group conflicts with 2 Conflict D1 D6 D1 D5 D4 D2 D7 D2 D3 D4 D8 D4 D5 D6 D7 D8 Sd A Su 10, 1 20, 1 2, 0 power budget: 40 vibration, budget: 1 Rack description Conflicts between groups {D4,D8}: 1 {D3,D7}: 1 {D2,D6}: 1 {D1,D5}: 1 {D1,D2,D3,D4}: 40 {D5,D6,D7,D8}: 40 Rack description Inter-group constraints {G1,G2,G3,G4}: 40 {G1,G2}: 1 {G3,G4}: 1 {G2,G1}: 1 {G1,G2}: 1 {G4,G3}: 1 {G3,G4}: 1 {G3,G4,G1,G2}: 40 {G4,G3,G2,G1}: 40

Resource Provisioning Exploration N resource types: N-dimensional space of rack descriptions Fully provisioned Resource 1 (e.g. power) D1 D2 D4 D5 D6 D7 D8 D3 40 1 Fully provisioned Resource 2 (e.g. vibration)

Resource Provisioning Exploration N resource types: N-dimensional space of rack descriptions Fully-provisioned rack (JBOD), per domain: all HDDs in most resource-consuming state Sd A Su 10, 1 20, 1 2, 0 D1 D2 D4 D5 D6 D7 D8 D3 vibration, budget: 2 power budget: 80 Fully provisioned Bottleneck resource constraint Relaxing Resource 1 (e.g. power) Least-provisioned rack, per domain: 1 HDD in IO-capable state n-1 in lowest resource-consuming D1 D2 D4 D5 D6 D7 D8 D3 vibration, budget: 1 power budget: 26 Fully provisioned Resource 2 (e.g. vibration)

Resource Provisioning Exploration N resource types: N-dimensional space of rack descriptions Fully-provisioned rack (JBOD), per domain: all HDDs in most resource-consuming state Fully provisioned Resource 1 (e.g. power) Discrete surface in the N-dimensional space For Pelican: 747 rack descriptions Bottleneck resource: vibration Bottleneck resource: power Fully provisioned Resource 2 (e.g. vibration)

Evaluation - Pelican Pelican Simulator – Poisson workload, 1GB reads.

Execution Time - Pelican 9 minutes

Execution Time - Pelican 9 minutes Rack name OCP Pelican Rack_A Rack_B Rack_C Rack_D Rack_E #rack descriptions 1921 747 1421 1152 973 649 683

Execution Time for Different Racks 9 minutes 3 hours Rack name OCP Pelican Rack_A Rack_B Rack_C Rack_D Rack_E #rack descriptions 1921 747 1421 1152 973 649 683

Conclusion Cold storage racks: Co-design: resource-constrained hardware + constraint-aware software low cost but hard to (re)design Flamingo simplifies design of cold storage racks Synthesizes Data Layout and IO Scheduler parameters Explores impact of resource provisioning on end performance Redesign in days vs months manually