1. Securing Containerized Applications
by Pino de Candia, CTO at Midokura
O’Reilly Software Architecture Conference
November 15, 2016

2. About Me, Midokura, MidoNet
MIDOKURA ARCHITECTURE FOR IIOT
About Me, Midokura, MidoNet
Pino de Candia:
CTO and Chief Architect at Midokura
Previously developer and dev manager
Background in Distributed Systems and early NoSQL databases
Small tech company (50+ employees)
Focus on Network Virtualization for virtualized workloads
Maker of MidoNet
Customers are public IaaS clouds, web-scale companies, enterprise private clouds for dev/test and production.
Open Source network virtualization
Integrations with OpenStack, Eucalyptus, Kubernetes, vSphere
Centralized network intent database; distributed network controller

3. Securing container networks East-west traffic L3, L4 Security
MIDOKURA ARCHITECTURE FOR IIOT
What this talk is about
Container networking
Securing container networks
East-west traffic
L3, L4 Security
L7 Security
QoS (a little bit)
NOT container image management
NOT preventing break-out from containers

4. Securing Containerized Applications
Pods and Containers
I’m not going to use the term “pod” in this talk. If you’re more familiar with Kubernetes, my use of the term “Container” may confuse you. I’m referring to the unit of work that has network and filesystem isolation, requiring Ethernet/IP/TCP to communicate with other work units.
What about security within one of these work units? Not covered by this talk, but there are systems to verify container images and code. Frankly, I’m not well-versed with security “inside” the container/pod.

5. Hosts, Minions and Agents
Securing Containerized Applications
Hosts, Minions and Agents
I’ll refer to hosts when I’m discussing containers without regard to any higher-level container orchestration platform. Host may be physical or virtual server (VM).
I use the term “minion” for hosts that are part of a higher-level orchestration platform.
I use the term “agent” for the software running on a minion, that takes orders from the orchestration platform’s control/management plane.

6. Network graphs Labels: role: frontend C7r C7r C7r TCP to port 5002
Securing Containerized Applications
Network graphs
Labels:
role: frontend
C7r
C7r
C7r
TCP to port 5002
Labels:
role: middleware
C7r
C7r
C7r
TCP to port 5004
Labels:
role: db
C7r
C7r
C7r

7. Network graphs Labels: role: frontend C7r C7r C7r TCP to port 5001 VIP
Securing Containerized Applications
Network graphs
Labels:
role: frontend
C7r
C7r
C7r
TCP to port 5001
VIP
TCP to port 5002
Labels:
role: middleware
C7r
C7r
C7r
TCP to port 5003
VIP
TCP to port 5004
Labels:
role: db
C7r
C7r
C7r

8. Network graphs External Endpoints Container Set Container Set
Securing Containerized Applications
Network graphs
External Endpoints
Container Set
Container Set
Container Set
External Endpoints
Container Set
Container Set
External Endpoints
Container Set
Container Set

9. Securing Containerized Applications
Network graphs

10. Block un-intended network access
Securing Containerized Applications
What would we like to do?
Block un-intended network access
Layers of security, possibly managed by different teams
Block horizontal attacks in allowed traffic
Without encumbering developers
Monitor/understand network traffic patterns

11. Potential container security requirements
Securing Containerized Applications
Potential container security requirements
Capability
Notes
Ingress rules (L3/L4)
Control who sends traffic, on what port and proto.
Egress rules (L3/L4)
Control where the container sends traffic.
Intrusion Prevention/Detection
Block various forms of code injection.
Deep Packet Inspection
Inspect traffic intent for various other purposes.
Different admin levels
Different teams control different layers of policy.
Auto-generate policies
Generate policy from service/container templates
Learning
Learning policy from behavior during initial period.
Apply policy after launch
Testing policy before applying
How do I know my policy won’t break something?
Port/traffic mirroring
Must-have trouble-shooting tool or ad-hoc intrusion detection.

12. QoS Policy Capability Notes Rate-limiting container bandwidth
Securing Containerized Applications
QoS Policy
Capability
Notes
Rate-limiting container bandwidth
One container can’t hog the network
Rate-limiting specific connections
One connection can’t hog the network
DSCP marking
Preferential treatment on the physical networking
Queuing, traffic policing/shaping
For preferential treatment within the host

13. Can’t network Containers like we network VMs
Securing Containerized Applications
Can’t network Containers like we network VMs VM VM VM VM VM
Linux bridging
br0
eth0

14. Can’t network Containers like we network VMs
Securing Containerized Applications
Can’t network Containers like we network VMs
Some of the ways that VMs are bridged to networks aren’t really scalable for containers:
Huge number of Containers requires a big IP subnet and DHCP address pool to assign IP addresses
VMs are fewer per host and usually stick around longer
Containers come and go a log faster. The physical network is not expected to change that fast (whether churn on required VLANs or assigned IP addresses)
It’s not “service-friendly” – keeping track of the IP addresses of Containers providing a service requires integration with the physical environment or Container must register after receiving an address. VM VM VM VM VM
ipvlan, macvlan, or SR-IOV
eth0

15. Containers Source NATed by host
Securing Containerized Applications
Containers Source NATed by host
C7r
C7r
C7r
C7r
C7r
Container IP is not visible outside the host.
Port mapping is painful, problematic and insecure.
Most container systems have been moving to “IP-per-container” model.
Example: Container port 80 exposed as host port 9080
Source NAT and port-mapping
br0
eth0

16. SNAT/Port-mapping security flaws
Securing Containerized Applications
SNAT/Port-mapping security flaws
C7r
C7r
C7r
C7r
C7r
You get little bit of security by obfuscation
In some implementations containers are on the same bridge so containers can port and address-scan each other.
Mapped/exposed ports can be scanned from off the box. Unless you’re doing some sort of authn/authz, anyone can use services they find.
And how can we decide who should have access to that port? SNAT obfuscates the client address.
Rogue containers can port-scan their neighbors on the bridge
Source NAT and port-mapping
br0
eth0
Anyone can access Containers via exposed ports
Rogue containers can scan IP ranges and ports of any endpoint in the data center

17. SNAT/Port-mapping security flaws
Securing Containerized Applications
SNAT/Port-mapping security flaws
C7r
C7r
C7r
C7r
C7r
br0
br1
Source NAT and port-mapping
Using multiple bridges doesn’t fix all the security flaws.
eth0

18. IP-per-Container model
Securing Containerized Applications
IP-per-Container model
Minion 1
Minion 2
C7r
C7r
C7r
C7r
C7r
C7r
/24
/24
Container IP addresses are provided at host-level. Hosts use different IP address ranges for their local Containers.
Container IP address ranges may be routable from the Data Center network OR NOT.
If NOT routable:
puts minimal requirements on the minions’ network, which might be physical or virtual (the Containers might be running in a VM).
Container-to-container communication is tunneled across different minions.
Ranges are static, so container churn does not make it harder for Minions to correctly perform tunneling.
Container-to-external communication must be Source NATed at the host
If they are:
Minion is probably announcing its local Container address range (i.e. participating in dynamic routing, like in Calico)
Tunneling is not required.
Source NAT for Container to Internet communication is performed by host’s network
br0
br0
Tunneling, Source NAT and port-mapping
Tunneling, Source NAT and port-mapping
eth0
eth0
/24
/24
IP Network

19. IP-per-Container security?
Securing Containerized Applications
IP-per-Container security?
Minion 1
Minion 2
C7r
C7r
C7r
C7r
C7r
C7r
/24
/24
Anybody can talk to anybody.
Rogue containers can scan IP ranges and ports of other Containers, Minions or any endpoint in the data center
br0
br0
Tunneling, Source NAT and port-mapping
Tunneling, Source NAT and port-mapping
eth0
eth0
/24
/24
IP Network

20. Services in Kubernetes
Securing Containerized Applications
Services in Kubernetes
Environment Variables:
SERVICE1_HOST=
SERVICE1_PORT=7890
Container-facing DNS:
Service1.namespace1 =>
http.tcp.Service1.namespace1 =>
:7890
Node 1
Pod
Pod
Pod
Client containers can discover service “Cluster IPs” either via DNS or via environment variables.
Local proxy load-balances the Cluster IP to the backend Pods, whatever Node they’re on.
Proxy has a limited form of health-checking (verify that, NOT sure); but Pods can be configured for various checks by their local Node. Local checks can remove Pods from Endpoint set (and therefore from proxies).
Local port 5000 is reserved for exposing Service1 (on all Pods). A twist on Docker “expose”.
Kubernetes has a built-in Service Registry and load-balancing proxy that understands the service registry.
A variation is to have the container clients explicitly do their own lookup in the Service Registry...
is called a “ClusterIP”
/24
br0
Load balance :7890 to
{Pod1:1234; Pod2:1234}
DNAT :5000 to :7890
5000 is a NodePort: Service1 can be contacted on port 5000 of ANY node
eth0
/24

21. Current state of security for containers?
Securing Containerized Applications
Current state of security for containers?

22. Label/annotation signals externally managed policy
Securing Containerized Applications
Label/annotation signals externally managed policy
% docker exec… --network=net1 –policy=poli1
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: nginx
annotations:
network: net1
policy: poli1
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
Initially SDNs wrapped docker, then Docker’s libnetwork provided a pluggable approach to networking. SDN-specific driver receives parameters from exec call and can interpret them as desired.
CNI (container native interface) does the same for Kubernetes (and more recently Mesos)
But network and policy details are managed outside the container system – in the specific SDN. No single pane of glass to show which networks are managed by what policies.
Enabled by libnetwork for Docker and CNI driver Kubernetes… but:
NO NATIVE WAY TO DEFINE POLICY
Example policy:
Allow UDP from Group1 on port 50
Allow TCP from Group2 on port 80
Allow TCP to Group3 on port 40
Default: drop

23. CNI/libnetwork driver with MidoNet (via Kuryr+Neutron)
Securing Containerized Applications
CNI/libnetwork driver with MidoNet (via Kuryr+Neutron)
Node 1
1. Container creation
C7r
veth0-0
MidoNet is an open source SDN from Midokura. Think of it as Enterprise-grade Neutron.
OpenStack is an IaaS cloud platform.
Neutron is the main OpenStack network project. It’s both an API and a default implementation, which can be replaced.
Kuryr is a project to integrate Neutron in container platforms.
2. Invokes Kuryr’s libnetwork driver
veth0-1
3. Creates Neutron network port and apply policy
MidoNet Switch
eth0
4. Bind Container interface to MN
/24

24. Requirements met by integration with Neutron/MidoNet
Securing Containerized Applications
Requirements met by integration with Neutron/MidoNet
Capability
Notes
Ingress rules
YES
Egress rules
Intrusion Prevention/Detection
NO (not conveniently)
Deep Packet Inspection NO
Different admin levels
Auto-generate policies
Learning
Apply policy after launch
YES (but may not be practical)
Testing policy before applying
Port/traffic mirroring

25. Requirements met by integration with Neutron/MidoNet
Securing Containerized Applications
Requirements met by integration with Neutron/MidoNet
Capability
Notes
Rate-limiting container bandwidth
YES
Rate-limiting specific connections NO
DSCP marking
Queuing, traffic policing/shaping

26. Controlling ingress traffic natively in Kubernetes
Securing Containerized Applications
Controlling ingress traffic natively in Kubernetes
Labels:
role: frontend
C7r
C7r
C7r
TCP to port 5002
Labels:
role: middleware
C7r
C7r
C7r
TCP to port 5004
Labels:
role: db
C7r
C7r
C7r

27. Kubernetes NetworkPolicy template (beta)
Securing Containerized Applications
Kubernetes NetworkPolicy template (beta)
apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: front-to-mid
 namespace: upsell-app spec: podSelector:  matchLabels:    role: middleware ingress:  - ports:     - protocol: TCP       port: 6379
   from:
    - podSelector:       matchLabels:         role: frontend
apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: mid-to-db
 namespace: upsell-app spec: podSelector:  matchLabels:    role: db ingress:  - ports:     - protocol: TCP       port: 3306
   from:
    - podSelector:       matchLabels:         role: middleware

28. Kubernetes NetworkPolicy template (beta)
Securing Containerized Applications
Kubernetes NetworkPolicy template (beta)
apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: front-to-mid
 namespace: upsell-app spec: podSelector:  matchLabels:    role: middleware ingress:  - ports:     - protocol: TCP       port: 6379
   from:
    - podSelector:       matchLabels:         role: frontend
apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: mid-to-db
 namespace: upsell-app spec: podSelector:  matchLabels:    role: db ingress:  - ports:     - protocol: TCP       port: 3306
   from:
    - podSelector:       matchLabels:         role: middleware

29. Requirements met by NetworkPolicy template
Securing Containerized Applications
Requirements met by NetworkPolicy template
Capability
Notes
Ingress rules
YES
Egress rules NO
Intrusion Prevention/Detection
Deep Packet Inspection
Different admin levels
Auto-generate policies
Learning
Apply policy after launch
Testing policy before applying
Port/traffic mirroring
YES (depending on what SDN you use)

30. Requirements met by integration with Neutron/MidoNet
Securing Containerized Applications
Requirements met by integration with Neutron/MidoNet
Capability
Notes
Rate-limiting container bandwidth NO
Rate-limiting specific connections
DSCP marking
Queuing, traffic policing/shaping

31. PoC: control L7 policy via template
Securing Containerized Applications
PoC: control L7 policy via template
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: nginx
annotations:
network: net1
policy: poli1
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata:  name: mid-to-db
 namespace: upsell-app spec: podSelector:  matchLabels:    role: db layer7: policy: database

32. PoC: control L7 policy via template in Kubernetes
Securing Containerized Applications
PoC: control L7 policy via template in Kubernetes
1. Launch pod with L7 policy
Node 1
C7r
veth0-0
2. Kuryr API watcher notifies Security Orchestrator
Containerized NGFW
C7r
3. Intel Open Security Controller launches Forcepoint Stonesoft NGFW on same Minion as workload
veth0-1
MidoNet Switch
eth0
4. Intel Security Controller calls MidoNet APIs to service chain workload traffic through the NGFW
/24

33. Thoughts on Kubernetes Namespaces
Securing Containerized Applications
Thoughts on Kubernetes Namespaces
Scopes for named objects within a cluster
Pods, Services, etc. are all namespaced
Logical grouping of related things
Could be 1 user
...or 1 app
...or 1 tier of an app
No relationship to nodes or networks
All Namespaces exist on all nodes
Network is not segmented
The NetworkPolicy template implicitly turns Namespaces into Policy domains.

34. Q&A and Useful Links Pino de Candia pino@midokura.com
Securing Containerized Applications
Q&A and Useful Links
Pino de Candia
Midokura Company Website
MidoNet Community Site
Project Git Repo
Try MidoNet with one command:
$> curl -sL quickstart.midonet.org | sudo bash
Kuryr
Community Site
Project Git Repo