1. Open Stack Summit – Hong Kong - 2013 OPENSTACK HA @PAYPAL

2. 2 PayPal offers flexible and innovative payment solutions for consumers and merchants of all sizes. 137,000,000 users $300,000 payments processed each minute 193 markets / 26 currencies The World’s Most Widely Used Digital Wallet ABOUT PAYPAL

3. 3 Why HA is important for PayPal? Our Learning Our Solution What is not solved? Q&A AGENDA

4. 4 WHY HA IS IMPORTANT? “no perceived downtime” for cloud users Enterprise Class Auto Scaling & Flex up/down can never break API Integrations always succeed Everyone expected to use the cloud

5. 5 No SPOF “Under the Cloud” Scale Across the Data Center(s) Scale Across Racks & Containers Respect natural availability zones within the data centers No ‘cloud’ can impact any other ‘cloud’ AVAILABILITY REQUIREMENTS

6. 6 INFRASTRUCTURE RACK 10g Active 10g Passive 1g Mgmt 10g Passive 10g Active 10g Passive 1g Mgmt 10g Passive 10g Active LB Active LB Passive Access Compute Racks …Infrastructure / Controller Racks … Layer 2 versus Layer 3 Cattle & Puppies

7. 7 INFRASTRUCTURE RACK OpenStack Services are all VM on KVM Every infra component resides on 2+ nodes Redundant physical racks Redundant power/switches in each rack Layer-3 connectivity between racks (no Layer 2) Enterprise Grade Physical LB (floating VIP)

8. 8 COMPUTE LB Active LB Passive Compute Node 96 Hyperscale 16 Core 256GB Ram 1.1T Disk Compute Node 96 Hyperscale 16 Core 256GB Ram 1.1T Disk 10g Active 10g Passive 1g Mgmt 10g Passive 10g Active Compute Node 96 Hyperscale 16 Core 256GB Ram 1.1T Disk Compute Node 96 Hyperscale 16 Core 256GB Ram 1.1T Disk 10g Active 10g Passive 1g Mgmt 10g Passive 10g Active Compute Node 96 Hyperscale 16 Core 256GB Ram 1.1T Disk Compute Node 96 Hyperscale 16 Core 256GB Ram 1.1T Disk 10g Active 10g Passive 1g Mgmt 10g Passive 10g Active Compute Node 96 Hyperscale 16 Core 256GB Ram 1.1T Disk Compute Node 96 Hyperscale 16 Core 256GB Ram 1.1T Disk 10g Active 10g Passive 1g Mgmt 10g Passive 10g Active LB Active LB Passive Access 1 2 3

9. 9 COMPUTE Active Passive 10g 1g 10g 1g 10g Management 1g bond0 Hyperscale Raid-10 Hyperscale Raid-10 Top Of Rack

10. OPENSTACK SERVICES

11. 11 LB VIP for every service (unless it can’t) Connect to LB VIP, not individual nodes Script to close Server Connections Pacemaker only works inside a single Layer-2 (not a large enterprise) Auto Restart using Monit MySQL Swift Cluster OPENSTACK CONSIDERATIONS

12. 12 HEAT with Corosync/Pacemaker/keepalived (for now) KeyStone / Nova / Glance / Swift Proxy Rabbit MQ Cluster Cinder Volume Service CONTINUED…

13. 13 Figure shows a typical interaction between Cinder components to serve a end user request. (create new volume in this example). CINDER SERVICES WORKFLOW Cinder API Cinder Scheduler Cinder Volume AMPQ Storage Back- end1 Storage Back- end1 Storage Back- end2 Storage Back- end2 User request (create volume) 1 2 3 4 5 6

14. 14 How HA is implemented for Cinder Components: API (stateless) – Load Balancer (A/A or A/P); Scheduler (stateless) – Pacemaker, Queue itself (A/A or A/P); Volume – Pacemaker, Queue itself (A/A or A/P). CINDER SERVICES WITH HA Cinder API A Cinder Scheduler B Cinder Volume A AMPQCluster Storage Back- end1 Storage Back- end1 Storage Back- end2 Storage Back- end2 User request (create volume) 1 2 5 6 Load Balancer Cinder API B 3 Cinder Scheduler A 4 Cinder Volume B

15. 15 VIP-friendly Cinder Volume service Seamless Upgrade Flip Failed DB TX Reconciliation Consistent API Response Time UNRESOLVED

16. 16Confidential and Proprietary cloud@paypal.com

17. THANK YOU HTTP://GITHUB.COM/PAYPAL/AURORA SCOTT CARLSON - @RELAXED137 RAJ GEDA ZHITENG HUANG IRC:WINSTON-D