PayPal Cloud Journey & Architecture Anand Palanisamy Sr. Manager, Cloud Platform Engineering Oct 27 2015

About PayPal Founded in 1998 Gobal Company and head quartered in San Jose, CA, USA Public company, trading in Nasdaq (PYPL) 169 million active accounts Operating in 203 markets with 100+ currencies Processed 4 Billion payments in 2014 $235 Billion (Payment Volume) More info at: https://www.paypal.com/about

Design Goals A platform to colloborate internally and externally Agility for both Cloud Users and Cloud Builder Well-defined Cloud APIs

About our PayPal Cloud Background Business Enablement Started in July 2012 with 1 engineer and 16 decommissioned servers Today, one of the world’s Largest OpenStack Private Cloud Number of Physical Servers: 8064 Total Cores: ~400,000+ Number of VMs: 82,000+ Block Storage: 2 peta bytes Availability Zones: 10+ Largest AZ with 2,500+ hypervisors Business Enablement Hosting ~100% of PayPal’s production traffic (except Databases and Messaging) Powers 100% of PaaS, Dev/QA and M&As First production workload on SDN in 2013

Provide a platform that enables agility, availability and innovation. The Vision Provide a platform that enables agility, availability and innovation. Risk Payments Wallet Mobile Ads CI, ALM, Monitoring, Release/Change Mgmt (Home Grown, Jenkins, Docker,Mesos..) Platform Frameworks (Java, C++, node.js, Python, Scala..) PAAS CMS (Configuration Management System) Homegrown CMS (CMDB) Open Source Puppet Salt Ansible Homegrown Reparo Blink StackWatch StackMetrics CloudInfo FlyWay Open Source ELK (Elastic Search, Log Stash, Kibana) Zabbix Monitoring Alerting Remediation Compliance Patching Upgrades DEV QA L&P PROD M&A’s Infrastructure-as-a- Service (IaaS) Nova, Neutron, Cinder, Swift, KeyStone, Glance PayPal Compute SKUs, ToRs/Routers, LBs, Firewalls, Storage

The Cloud Lab + First Seed

The Journey so far…

Cloud Deployment Architecture 10,000 foot level Regions Availability Zones- AZs Region 2 AZ1 AZ2 AZn Region 1 Region m AZ = fault-domain, single OpenStack control-plane instance AZ maps to one or more physical network bubbles Multiple nova cells in AZ

Inside AZ Internet Core Availability Zone Aggregation Access SDN Gateways LBs Firewalls Racks Cloud (IaaS + PaaS) Controllers, VMs, SDN controllers

Multi-tenancy model Virtual private cloud (VPC) VPC is a collection of OpenStack tenants It is a security zone Keystone changes Ideally single large (virtual) router or VRF per VPC VPC model introduces scale issues VPC Isolation evolved over period of time based on the network gears

Challenges in managing 10+ AZs 1000s of services across multiple data centers GBs of logs every hour to dig for identifying potential issue Many moving parts (Backbone network, LDAP Infra, DNS Infra, CORP LDAP & Network) Network Infrastructure differences Overall failure Vs 1 or 2 failures Global Keystone Sync issue between AZ and Cells Firewall between Control Plane and Hypervisors Generations of Hardware, Firmware Versions, BIOS difference Config drift management Capacity Management and Cloud Backoffice

Challenges with DEV/QA VPC Unpredictable API Usage pattern Single point of failure for VMs Variety of workloads Adhoc Connectivity requirements Patching VMs Enforcing discipline

https://jobsearch.paypal-corp.com/jobs?keywords=cloud&page=1 Questions? We are hiring  https://jobsearch.paypal-corp.com/jobs?keywords=cloud&page=1