Securing Containerized Applications by Pino de Candia, CTO at Midokura O’Reilly Software Architecture Conference November 15, 2016

About Me, Midokura, MidoNet MIDOKURA ARCHITECTURE FOR IIOT About Me, Midokura, MidoNet Pino de Candia: CTO and Chief Architect at Midokura Previously developer and dev manager Background in Distributed Systems and early NoSQL databases Small tech company (50+ employees) Focus on Network Virtualization for virtualized workloads Maker of MidoNet Customers are public IaaS clouds, web-scale companies, enterprise private clouds for dev/test and production. Open Source network virtualization Integrations with OpenStack, Eucalyptus, Kubernetes, vSphere Centralized network intent database; distributed network controller

Securing container networks East-west traffic L3, L4 Security MIDOKURA ARCHITECTURE FOR IIOT What this talk is about Container networking Securing container networks East-west traffic L3, L4 Security L7 Security QoS (a little bit) NOT container image management NOT preventing break-out from containers

Securing Containerized Applications Pods and Containers I’m not going to use the term “pod” in this talk. If you’re more familiar with Kubernetes, my use of the term “Container” may confuse you. I’m referring to the unit of work that has network and filesystem isolation, requiring Ethernet/IP/TCP to communicate with other work units. What about security within one of these work units? Not covered by this talk, but there are systems to verify container images and code. Frankly, I’m not well-versed with security “inside” the container/pod.

Hosts, Minions and Agents Securing Containerized Applications Hosts, Minions and Agents I’ll refer to hosts when I’m discussing containers without regard to any higher-level container orchestration platform. Host may be physical or virtual server (VM). I use the term “minion” for hosts that are part of a higher-level orchestration platform. I use the term “agent” for the software running on a minion, that takes orders from the orchestration platform’s control/management plane.

Network graphs Labels: role: frontend C7r C7r C7r TCP to port 5002 Securing Containerized Applications Network graphs Labels: role: frontend C7r C7r C7r TCP to port 5002 Labels: role: middleware C7r C7r C7r TCP to port 5004 Labels: role: db C7r C7r C7r

Network graphs Labels: role: frontend C7r C7r C7r TCP to port 5001 VIP Securing Containerized Applications Network graphs Labels: role: frontend C7r C7r C7r TCP to port 5001 VIP TCP to port 5002 Labels: role: middleware C7r C7r C7r TCP to port 5003 VIP TCP to port 5004 Labels: role: db C7r C7r C7r

Network graphs External Endpoints Container Set Container Set Securing Containerized Applications Network graphs External Endpoints Container Set Container Set Container Set External Endpoints Container Set Container Set External Endpoints Container Set Container Set

Securing Containerized Applications Network graphs

Block un-intended network access Securing Containerized Applications What would we like to do? Block un-intended network access Layers of security, possibly managed by different teams Block horizontal attacks in allowed traffic Without encumbering developers Monitor/understand network traffic patterns

Potential container security requirements Securing Containerized Applications Potential container security requirements Capability Notes Ingress rules (L3/L4) Control who sends traffic, on what port and proto. Egress rules (L3/L4) Control where the container sends traffic. Intrusion Prevention/Detection Block various forms of code injection. Deep Packet Inspection Inspect traffic intent for various other purposes. Different admin levels Different teams control different layers of policy. Auto-generate policies Generate policy from service/container templates Learning Learning policy from behavior during initial period. Apply policy after launch Testing policy before applying How do I know my policy won’t break something? Port/traffic mirroring Must-have trouble-shooting tool or ad-hoc intrusion detection.

QoS Policy Capability Notes Rate-limiting container bandwidth Securing Containerized Applications QoS Policy Capability Notes Rate-limiting container bandwidth One container can’t hog the network Rate-limiting specific connections One connection can’t hog the network DSCP marking Preferential treatment on the physical networking Queuing, traffic policing/shaping For preferential treatment within the host

Can’t network Containers like we network VMs Securing Containerized Applications Can’t network Containers like we network VMs VM VM VM VM VM Linux bridging br0 eth0

Can’t network Containers like we network VMs Securing Containerized Applications Can’t network Containers like we network VMs Some of the ways that VMs are bridged to networks aren’t really scalable for containers: Huge number of Containers requires a big IP subnet and DHCP address pool to assign IP addresses VMs are fewer per host and usually stick around longer Containers come and go a log faster. The physical network is not expected to change that fast (whether churn on required VLANs or assigned IP addresses) It’s not “service-friendly” – keeping track of the IP addresses of Containers providing a service requires integration with the physical environment or Container must register after receiving an address. VM VM VM VM VM ipvlan, macvlan, or SR-IOV eth0

Containers Source NATed by host Securing Containerized Applications Containers Source NATed by host C7r C7r C7r C7r C7r Container IP is not visible outside the host. Port mapping is painful, problematic and insecure. Most container systems have been moving to “IP-per-container” model. Example: Container port 80 exposed as host port 9080 Source NAT and port-mapping br0 eth0

SNAT/Port-mapping security flaws Securing Containerized Applications SNAT/Port-mapping security flaws C7r C7r C7r C7r C7r You get little bit of security by obfuscation In some implementations containers are on the same bridge so containers can port and address-scan each other. Mapped/exposed ports can be scanned from off the box. Unless you’re doing some sort of authn/authz, anyone can use services they find. And how can we decide who should have access to that port? SNAT obfuscates the client address. Rogue containers can port-scan their neighbors on the bridge Source NAT and port-mapping br0 eth0 Anyone can access Containers via exposed ports Rogue containers can scan IP ranges and ports of any endpoint in the data center

SNAT/Port-mapping security flaws Securing Containerized Applications SNAT/Port-mapping security flaws C7r C7r C7r C7r C7r br0 br1 Source NAT and port-mapping Using multiple bridges doesn’t fix all the security flaws. eth0

IP-per-Container model Securing Containerized Applications IP-per-Container model Minion 1 Minion 2 C7r C7r C7r C7r C7r C7r 10.0.1.0/24 10.0.2.0/24 Container IP addresses are provided at host-level. Hosts use different IP address ranges for their local Containers. Container IP address ranges may be routable from the Data Center network OR NOT. If NOT routable: puts minimal requirements on the minions’ network, which might be physical or virtual (the Containers might be running in a VM). Container-to-container communication is tunneled across different minions. Ranges are static, so container churn does not make it harder for Minions to correctly perform tunneling. Container-to-external communication must be Source NATed at the host If they are: Minion is probably announcing its local Container address range (i.e. participating in dynamic routing, like in Calico) Tunneling is not required. Source NAT for Container to Internet communication is performed by host’s network br0 10.0.1.1 br0 10.0.2.1 Tunneling, Source NAT and port-mapping Tunneling, Source NAT and port-mapping eth0 eth0 192.168.1.1/24 192.168.2.1/24 IP Network

IP-per-Container security? Securing Containerized Applications IP-per-Container security? Minion 1 Minion 2 C7r C7r C7r C7r C7r C7r 10.0.1.0/24 10.0.2.0/24 Anybody can talk to anybody. Rogue containers can scan IP ranges and ports of other Containers, Minions or any endpoint in the data center br0 br0 Tunneling, Source NAT and port-mapping Tunneling, Source NAT and port-mapping eth0 eth0 192.168.1.1/24 192.168.2.1/24 IP Network

Services in Kubernetes Securing Containerized Applications Services in Kubernetes Environment Variables: SERVICE1_HOST=10.2.0.5 SERVICE1_PORT=7890 Container-facing DNS: Service1.namespace1 => 10.2.0.5 http.tcp.Service1.namespace1 => 10.2.0.5:7890 Node 1 Pod Pod Pod Client containers can discover service “Cluster IPs” either via DNS or via environment variables. Local proxy load-balances the Cluster IP to the backend Pods, whatever Node they’re on. Proxy has a limited form of health-checking (verify that, NOT sure); but Pods can be configured for various checks by their local Node. Local checks can remove Pods from Endpoint set (and therefore from proxies). Local port 5000 is reserved for exposing Service1 (on all Pods). A twist on Docker “expose”. Kubernetes has a built-in Service Registry and load-balancing proxy that understands the service registry. A variation is to have the container clients explicitly do their own lookup in the Service Registry... 10.2.0.5 is called a “ClusterIP” 10.0.1.0/24 br0 Load balance 10.2.0.5:7890 to {Pod1:1234; Pod2:1234} DNAT 192.168.1.1:5000 to 10.2.0.5:7890 5000 is a NodePort: Service1 can be contacted on port 5000 of ANY node eth0 192.168.1.1/24

Current state of security for containers? Securing Containerized Applications Current state of security for containers?

Label/annotation signals externally managed policy Securing Containerized Applications Label/annotation signals externally managed policy % docker exec… --network=net1 –policy=poli1 apiVersion: v1 kind: Pod metadata: name: nginx labels: app: nginx annotations: network: net1 policy: poli1 spec: containers: - name: nginx image: nginx ports: - containerPort: 80 Initially SDNs wrapped docker, then Docker’s libnetwork provided a pluggable approach to networking. SDN-specific driver receives parameters from exec call and can interpret them as desired. CNI (container native interface) does the same for Kubernetes (and more recently Mesos) But network and policy details are managed outside the container system – in the specific SDN. No single pane of glass to show which networks are managed by what policies. Enabled by libnetwork for Docker and CNI driver Kubernetes… but: NO NATIVE WAY TO DEFINE POLICY Example policy: Allow UDP from Group1 on port 50 Allow TCP from Group2 on port 80 Allow TCP to Group3 on port 40 Default: drop

CNI/libnetwork driver with MidoNet (via Kuryr+Neutron) Securing Containerized Applications CNI/libnetwork driver with MidoNet (via Kuryr+Neutron) Node 1 1. Container creation C7r veth0-0 MidoNet is an open source SDN from Midokura. Think of it as Enterprise-grade Neutron. OpenStack is an IaaS cloud platform. Neutron is the main OpenStack network project. It’s both an API and a default implementation, which can be replaced. Kuryr is a project to integrate Neutron in container platforms. 2. Invokes Kuryr’s libnetwork driver veth0-1 3. Creates Neutron network port and apply policy MidoNet Switch eth0 4. Bind Container interface to MN 192.168.1.1/24

Requirements met by integration with Neutron/MidoNet Securing Containerized Applications Requirements met by integration with Neutron/MidoNet Capability Notes Ingress rules YES Egress rules Intrusion Prevention/Detection NO (not conveniently) Deep Packet Inspection NO Different admin levels Auto-generate policies Learning Apply policy after launch YES (but may not be practical) Testing policy before applying Port/traffic mirroring

Requirements met by integration with Neutron/MidoNet Securing Containerized Applications Requirements met by integration with Neutron/MidoNet Capability Notes Rate-limiting container bandwidth YES Rate-limiting specific connections NO DSCP marking Queuing, traffic policing/shaping

Controlling ingress traffic natively in Kubernetes Securing Containerized Applications Controlling ingress traffic natively in Kubernetes Labels: role: frontend C7r C7r C7r TCP to port 5002 Labels: role: middleware C7r C7r C7r TCP to port 5004 Labels: role: db C7r C7r C7r

Kubernetes NetworkPolicy template (beta) Securing Containerized Applications Kubernetes NetworkPolicy template (beta) apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: front-to-mid  namespace: upsell-app spec: podSelector:  matchLabels:    role: middleware ingress:  - ports:     - protocol: TCP       port: 6379    from:     - podSelector:       matchLabels:         role: frontend apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: mid-to-db  namespace: upsell-app spec: podSelector:  matchLabels:    role: db ingress:  - ports:     - protocol: TCP       port: 3306    from:     - podSelector:       matchLabels:         role: middleware

Kubernetes NetworkPolicy template (beta) Securing Containerized Applications Kubernetes NetworkPolicy template (beta) apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: front-to-mid  namespace: upsell-app spec: podSelector:  matchLabels:    role: middleware ingress:  - ports:     - protocol: TCP       port: 6379    from:     - podSelector:       matchLabels:         role: frontend apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: mid-to-db  namespace: upsell-app spec: podSelector:  matchLabels:    role: db ingress:  - ports:     - protocol: TCP       port: 3306    from:     - podSelector:       matchLabels:         role: middleware

Requirements met by NetworkPolicy template Securing Containerized Applications Requirements met by NetworkPolicy template Capability Notes Ingress rules YES Egress rules NO Intrusion Prevention/Detection Deep Packet Inspection Different admin levels Auto-generate policies Learning Apply policy after launch Testing policy before applying Port/traffic mirroring YES (depending on what SDN you use)

Requirements met by integration with Neutron/MidoNet Securing Containerized Applications Requirements met by integration with Neutron/MidoNet Capability Notes Rate-limiting container bandwidth NO Rate-limiting specific connections DSCP marking Queuing, traffic policing/shaping

PoC: control L7 policy via template Securing Containerized Applications PoC: control L7 policy via template apiVersion: v1 kind: Pod metadata: name: nginx labels: app: nginx annotations: network: net1 policy: poli1 spec: containers: - name: nginx image: nginx ports: - containerPort: 80 apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: mid-to-db  namespace: upsell-app spec: podSelector:  matchLabels:    role: db layer7: policy: database

PoC: control L7 policy via template in Kubernetes Securing Containerized Applications PoC: control L7 policy via template in Kubernetes 1. Launch pod with L7 policy Node 1 C7r veth0-0 2. Kuryr API watcher notifies Security Orchestrator Containerized NGFW C7r 3. Intel Open Security Controller launches Forcepoint Stonesoft NGFW on same Minion as workload veth0-1 MidoNet Switch eth0 4. Intel Security Controller calls MidoNet APIs to service chain workload traffic through the NGFW 192.168.1.1/24

Thoughts on Kubernetes Namespaces Securing Containerized Applications Thoughts on Kubernetes Namespaces Scopes for named objects within a cluster Pods, Services, etc. are all namespaced Logical grouping of related things Could be 1 user ...or 1 app ...or 1 tier of an app No relationship to nodes or networks All Namespaces exist on all nodes Network is not segmented The NetworkPolicy template implicitly turns Namespaces into Policy domains.

Q&A and Useful Links Pino de Candia pino@midokura.com Securing Containerized Applications Q&A and Useful Links Pino de Candia pino@midokura.com Midokura Company Website www.midokura.com MidoNet Community Site www.midonet.org Project Git Repo https://github.com/midonet/midonet Try MidoNet with one command: $> curl -sL quickstart.midonet.org | sudo bash Kuryr Community Site https://wiki.openstack.org/wiki/Kuryr Project Git Repo https://github.com/openstack/kuryr