1. PayPal Cloud Journey & Architecture
Anand Palanisamy
Sr. Manager, Cloud Platform Engineering
Oct

2. About PayPal Founded in 1998
Gobal Company and head quartered in San Jose, CA, USA
Public company, trading in Nasdaq (PYPL)
169 million active accounts
Operating in 203 markets with 100+ currencies
Processed 4 Billion payments in 2014
$235 Billion (Payment Volume)
More info at:

3. Design Goals
A platform to colloborate internally and externally
Agility for both Cloud Users and Cloud Builder
Well-defined Cloud APIs

4. About our PayPal Cloud Background Business Enablement
Started in July 2012 with 1 engineer and 16 decommissioned servers
Today, one of the world’s Largest OpenStack Private Cloud
Number of Physical Servers: 8064
Total Cores: ~400,000+
Number of VMs: 82,000+
Block Storage: 2 peta bytes
Availability Zones: 10+
Largest AZ with 2,500+ hypervisors
Business Enablement
Hosting ~100% of PayPal’s production traffic (except Databases and Messaging)
Powers 100% of PaaS, Dev/QA and M&As
First production workload on SDN in 2013

5. Provide a platform that enables agility, availability and innovation.
The Vision
Provide a platform that enables agility, availability and innovation.
Risk
Payments
Wallet
Mobile
Ads
CI, ALM, Monitoring, Release/Change Mgmt (Home Grown, Jenkins, Docker,Mesos..)
Platform Frameworks (Java, C++, node.js, Python, Scala..)
PAAS
CMS (Configuration Management System)
Homegrown
CMS (CMDB)
Open Source
Puppet
Salt
Ansible
Homegrown
Reparo
Blink
StackWatch
StackMetrics
CloudInfo
FlyWay
Open Source
ELK (Elastic Search, Log Stash, Kibana)
Zabbix
Monitoring Alerting
Remediation
Compliance
Patching
Upgrades
DEV QA
L&P
PROD
M&A’s
Infrastructure-as-a- Service (IaaS)
Nova, Neutron, Cinder, Swift, KeyStone, Glance
PayPal Compute SKUs, ToRs/Routers, LBs, Firewalls, Storage

6. The Cloud Lab + First Seed

7. The Journey so far…

8. Cloud Deployment Architecture
10,000 foot level
Regions
Availability Zones- AZs
Region 2
AZ1
AZ2
AZn
Region 1
Region m
AZ = fault-domain, single OpenStack control-plane instance
AZ maps to one or more physical network bubbles
Multiple nova cells in AZ

9. Inside AZ Internet Core Availability Zone Aggregation Access
SDN Gateways
LBs
Firewalls
Racks
Cloud (IaaS + PaaS) Controllers, VMs, SDN controllers

10. Multi-tenancy model Virtual private cloud (VPC)
VPC is a collection of OpenStack tenants
It is a security zone
Keystone changes
Ideally single large (virtual) router or VRF per VPC
VPC model introduces scale issues
VPC Isolation evolved over period of time based on the network gears

11. Challenges in managing 10+ AZs
1000s of services across multiple data centers
GBs of logs every hour to dig for identifying potential issue
Many moving parts (Backbone network, LDAP Infra, DNS Infra, CORP LDAP & Network)
Network Infrastructure differences
Overall failure Vs 1 or 2 failures
Global Keystone
Sync issue between AZ and Cells
Firewall between Control Plane and Hypervisors
Generations of Hardware, Firmware Versions, BIOS difference
Config drift management
Capacity Management and Cloud Backoffice

12. Challenges with DEV/QA VPC
Unpredictable API Usage pattern
Single point of failure for VMs
Variety of workloads
Adhoc Connectivity requirements
Patching VMs
Enforcing discipline

13. https://jobsearch.paypal-corp.com/jobs?keywords=cloud&page=1
Questions?
We are hiring 