1. ISTIO & ENVOY – Security
For Security Subcommittee

2. Micro Service Architecture - Background
Internet
Internet
SLB/ADC/API-Gateway
SLB/ADC/API-Gateway ES ES ES EM EM
IS1
IS1
IS1
IS1
IS2
IS2
IS3
IS3
IM1
IM2
IM3
IM1
IM2
IM3
IS4
IS4
IS5
IS5
EM – Module with external interface
IM – Internal Modules
ES – External Service
IS – Internal Service
IM4
IM5
IM4
IM5
Micro Service based
In monolithic application, entire application is scaled-out, waste of memory, no rolling upgrades, massive integration time, big maintenance.
Micro Service architecture:
Each individual module is a micro service
Each module can scale-out independently
Typically each micro service is implemented as container
Image
Image
Monolithic Application

3. Micro Service Architecture - Challenges
Need for load balancing among internal hierarchy
Special load balancers – Expensive
Each consumer balancing its connections across producers. Independent LB decision. May load one instance of producers.
Special logic at each service.
Need for discovery of service instances
Special logic at the each service.
Need for health checks
Special logic at each service
Need for tracing/visibility
Need for network firewall capability
Authenticate and Authorize consumers
Special logic at each service level
Need for Mutual TLS among producers and consumers.
Special logic to get certificate enrolled.
Special logic to initiate/accept TLS connections
Internet
SLB/ADC/API-Gateway ES ES ES
IS1
IS1
IS1
IS1
IS2
IS2
IS3
IS3
EM – Module with external interface
IM – Internal Modules
ES – External Service
IS – Internal Service
IS4
IS4
IS5
IS5

4. Micro Service Architecture – Development Challenges
Polyglot
Any special logic at each service level need to support multiple language bindings.
Coordination among all development teams
Third party micro services (binary) – Can’t put our special logic there.
In some instances, it was found that 50% of the logic is related to non-application specific.
Internet
SLB/ADC/API-Gateway ES ES ES
IS1
IS1
IS1
IS1
IS2
IS2
IS3
IS3
EM – Module with external interface
IM – Internal Modules
ES – External Service
IS – Internal Service
IS4
IS4
IS5
IS5
Let application service developers not worry about micro-service tax.

5. Micro Service Architecture – Service Mesh technology
Internet
SC (Side Car) – One for each Service instance (A container in each POD)
Envoy proxy
Service discovery, Load balancing, Failure handling, Circuit breaking (Limits), Fault injection (for troubleshooting), Health checks
Mutual TLS
ISTIO Control plane:
Pilot:
Service discovery glue between Envoy and K8S
Traffic rule configuration
Security
Role based Access control (Pluggable RBAC engine, support for local RBAC, K8S RBAC adapters, but facility to add new adapters – Example AAF RBAC adapter)
Certificate Authority
Istio-ingress envoy ES ES ES SC SC SC SC
IS2
IS1 SC
ISTIO Control Plane
Pilot/Mixer/CA SC
IS2
IS1 SC
EM – Module with external interface
IM – Internal Modules
ES – External Service
IS – Internal Service SC
IS3
IS1 SC SC
IS3 SC SC SC SC
IS4
IS4
IS5
IS5

6. Service to Envoy proxy – Security Concerns & Mitigation
K8S POD2
K8S POD2
K8S POD 1
K8S POD 1
Envoy
SVC2
Instance1
SVC1
Instance1
Envoy
Name space 1
IP tables tproxy
Name space 2
TLS
Name space 2
Name space 1
IP Tables Tproxy
Linux Kernel
Linux Kernel
EM – Module with external interface
IM – Internal Modules
ES – External Service
IS – Internal Service
ISTIO Pilot
Even though traffic is clear between instance and side car, it is not considered a security issue as both side car and service belong to same POD, hence same network name space. Clear traffic is never seen on the wire.
Pilot configures IP Tables to redirect the traffic coming from services to Envoy and Envoy to services.
No configuration required at the service level to use Envoy as proxy
Envoy acts as transparent proxy. Ensure that Source IP of the connection is maintained on both sides.

7. ISTIO CA - Features Can generate self signed CA root certificate
Can take externally generated CA certificate and private key
Support for certificate chain (CA certificate signed by Intermediate CA)
CA redundancy
Two types of user certificate issue methods
Native K8S based for K8S orchestrated services
Non-K8S orchestrated services (VMs/Baremetal)
Certificate renewal
SPIFFE naming convention
Envoy:
Auto reload of renewed certificates/keys
Integration with PKCS11 based security
Authentication – Mutual TLS
Authorization - ISTIO RBAC or adapters to remote RBAC
1. Create Service account and Deploy service
K8S Master
2. POD deployment event
ISTIO CA
3. Create user key/certificate (signed by CA cert)
4. Send key/cert using secret mount
SVC SC
ISTIO CA
EM – Module with external interface
IM – Internal Modules
ES – External Service
IS – Internal Service
3. Generate Cert, Sign using CA key
2. Send CSR
4. Send Cert
VM/Baremetal
Node agent
1. Key pair generation and CSR

8. HW Security - Casablanca work
ISTIO CA
Envoy proxy
Secure CA private key
Distribute CA private key securely to the Hardware (TPM/SGX) – Started conversations with ISITO security group.
Extend the work done on AAF CA to ISTIO CA – Note : Current AAF CA is only testing purpose only.
Secure user private key
Envoy takes care of RBAC.
ISTIO RBAC is quite powerful, but if lacks features, can add new adapter to talk to ONAP RBAC engine (AAF)
Go Crypto
OpenSSL
Crypto11 (PKCS11 Interface)
LibP11
Crypto11 (PKCS11 Interface)
SSHSM
SSHSM
TPM /SGX plugin
EM – Module with external interface
IM – Internal Modules
ES – External Service
IS – Internal Service
TPM /SGX plugin
Hardware
Hardware
Since Mutual TLS/RBAC is implemented in one way (C++ in case of envoy), provides opportunity to do good job of HW security & Accelerating TLS, thereby universal security and improving performance

9. Summary
Service mesh technologies take away the tax of Micro-Service architecture out of services and consolidate in side cars.
Less error prone
Faster development
Easy maintenance
Polyglot independent.
K8S native support
Security is one big piece of Service mesh
Enable Mutual TLS
Enable RBAC
Enables security without each developer putting effort independently in their projects
Provides opportunities to provide HW security and acceleration with one implementation.

10. Inter working with AAF

11. Overlap items CA
AAF has CA and is being used by some services in Beijing release.
ISTIO has its own CA.
RBAC
AAF has RBAC and some services in Beijing release is already using AAF RBAC.
ISTIO has RBAC, which is simpler (Need to see if is good enough to satisfy ONAP requirements)
Three choices (Assuming that ISTIO Service mesh is chosen)
Choice 1: Use ISTIO CA and ISTIO RBAC across all ONAP services (Presented before)
Choice 2: Use ISTIO CA and use AAF RBAC
Choice 3: Use ISTIO CA and AAF CA together and use AAF RBAC (Being shown in next slide as it is superset)

12. Choice 3 (Keeping services that use AAF CA )1
ONAP Administrator creates CA key pair and self-signed CA certificate (or it could be CA key pair and signed by root/intermediate CA)
ONAP Administrator using CA CLI/GUI, upload the CA private key/certificate and any chain in both ISTIO CA instances and AAF CA instances.
When S1 or S2 is brought up by OOM, corresponding CA will issue service certificates. Services get its own key/certificate & CA chain.
TLS establishment : Based on peer ID, service (or its side car) authenticates using CA certificate it has.
Authorization: When there is HTTP operation, its method, URL and other operation specific information, requester ID is sent to the RBAC engine to figure out the roles and permissions.
RBAC plugin talks to AAF RBAC to get the permissions (in case of ISTIO)
If permissions are allowed for that operation, operation proceeds. Otherwise, operation is denied.
CA Key Generation Station (Secure system) 2 2
ISTIO CA
AAF CA
AAF RBAC 3 3
ISTIO Mixer 5 6
AAF RBAC Plugin 5 S1
Side car S2
CADI 4 7
No implication (based on current understanding) on existing services that are using CADI for authentication & authorization
Additional RBAC plugin development at the ISTIO is required

13. Recommendation Assumptions:
AAF RBAC is the right solution for ONAP, which is not supported ISTIO RBAC.
Recommendation for default setup
ISTIO CA
ISTIO Mixer with AAF RBAC plugin and use AAF RBAC
Options:
If some deployment likes to use “ISTIO RBAC”, it should be possible.
If some deployment likes to use their own RBAC, they just need to develop their own RBAC plugin to ISTIO