1. Ingress Policy

2. Agenda – New Features Feature Summary Data Plane Flow of current model Policy enforcement for current model Limitations of current model Data Plane Flow for Ingress Policy model Policy enforcement for Ingress Policy model Information Flow for Ingress Policy model Policy enforcement for pcEnfDir=egress Troubleshooting Ingress Policy User Configuration Restrictions for Ingress Policy model

3. Feature Summary Saves resources on border leaf. Reduces SG TCAM (zone-rule) consumption on border TOR Policy will be only applied on Non-Border TOR with this model For backward compatibility, PcEnfDir (ingress/egress) knob provided on per VRF level. Alleviates some of the policy and tenant configuration scale issues incurred during WAN configuration in ACI External prefixes configured under L3out on Border leaf will be leaked onto Non- Border leaf with “remote” flag for policy application purpose.

4. Dataplane Flow For Current Model Host to WAN NorthStar Broadcom On Non Border Leaf Si Border Leaf Leafs Spines External Router Host WAN to Host NorthStar Broadcom On Border Leaf WAN to Host NorthStar Broadcom On Non-Border Leaf Host to WAN NorthStar Broadcom On Border Leaf

5. Policy Enforcement For Current Model Zone-rules installed on both Non-Border & Border leaf Assuming host & WAN on different TORs Host to WAN Policy is applied on Border TOR Destination class derivation cannot happen on Non-Border leaf in this case WAN to Host Policy is applied on Border TOR if destination host is learnt on Border TOR Policy is applied on Non-Border TOR otherwise WAN to WAN Policy is applied on ingress Border TOR Destination class derivation happens on ingress Border TOR in this case

6. Limitations Of Current Model For all the hosts that need WAN access, contract needs to be installed on both Non- Border as well as Border TOR for appropriate policy application. This causes a scale issue on the border leaf. Increased SG TCAM utilization on Border TOR. SG TCAM limit: 4k NorthStar ASIC/64k Donner ASIC.

7. Dataplane Flow For Ingress Policy Model Host to WAN NorthStar Broadcom On Non Border Leaf Si Border Leaf Leafs Spines External Router Host WAN to Host NorthStar Broadcom On Border Leaf WAN to Host NorthStar Broadcom On Non-Border Leaf Host to WAN NorthStar Broadcom On Border Leaf

8. Policy Enforcement For Ingress Policy Model Policy enforcement direction for VRF should be set to “Ingress” Zone-rules installed only on Non-Border leaf Assuming host & WAN on different TORs Host to WAN Policy is applied on Non-Border TOR Destination class derived on Non-Border leaf from “remote” actrlPfxEntry WAN to Host Policy is applied on Non-Border TOR irrespective of whether or not the EPG is learnt on BL WAN to WAN Policy is applied on ingress Border TOR Destination class derived on ingress Border leaf from “remote” actrlPfxEntry

9. Policy Enforcement For Ingress Policy Model APIC (PE) Policy Mgr Aclqos Shim NorthStar MTS msg sent from APIC to switch policy mgr process for prefix/rule add/modify/delete PPF session between SUP & LC process Aclqos requests shim/nsausd to program the entry in Northstar ASIC Shim programs the entry in h/w Switch

10. Policy Enforcement for PcEnfDir = Egress When PcEnfDir is set to egress, the behavior is same as old model (mentioned on slide # 4 & 5) Eltmc command to display VRF policy mode: FIB command to display VRF policy mode:

11. Troubleshooting Shared – L3 Out Troubleshooting ufib If each route is advertised by the VRF from multiple nexthops, we should see a q-in-q entry used in the data plane for each nexthop.

12. Troubleshooting Shared – L3 Out (cont.)

13. Troubleshooting broadcom / North-star (for host to WAN path) Identify the route from broadcom LPM, follow the nexthop and interface to identify q-in-q tag. Use q-in-q tag in NS to identify adjacency.

14. Troubleshooting Shared – L3 Out (cont.) # vsh_lc -c "show platform internal ns forwarding epg ingress" | grep fdb004 900 0 fdb004 1 1 200002 0 0 0 0 S/D N/N 0 0 0 0 1 c 0 0 0 # vsh_lc -c "show platform internal ns forwarding adj 0xc ingress" ENCP T U USE D S RM S SRC POS SEG-ID PTR D P PCI M DST-MAC M IDX R SEG-ID CLSS ----------------------------------------------------------------------------------------------------------- 12 200000 2ff4 1 0 0 0 00:00:00:00:00:00 0 0 1 200000 0

15. Troubleshooting Ingress Policy On Border TOR: 172.20.30.0/24 is the actrlPfxEntry under consideration. 3699788 is VRF VNID. leaf102# vsh_lc module-1# show system internal aclqos prefix VrfVni AddrMaskScopeClassSharedRemote ======== ========== ====== ======= ===== ======= ======= 3699788172.20.30.0ff449155FALSEFALSE leaf102# show zoning-rule | grep 3699788

16. Troubleshooting Ingress Policy (cont.) On Non-border TOR: leaf101# vsh_lc module-1# show system internal aclqos prefix VrfVni AddrMaskScopeClassSharedRemote ======= ========= ===== ===== ===== ====== ======= 3699788172.20.30.0ff450266FALSETRUE leaf101# show zoning-rule | grep 3699788 Rule IDSrcEPGDstEPGFilterIDoperStScopeActionPriority ======= ========= ========= ======= ======== ======= ====== ============ 48603277049155defaultenabled2588677permitsrc_dst_any(5) 48614915532770defaultenabled2588677permitsrc_dst_any(5)

17. User Configuration A new property named “PcEnfDir” i.e. Policy Enforcement Direction has been introduced on fvCtx (VRF). It has two possible values: Egress: Maintains the old behavior. −Actrl rules (represented by model class actrl::Rule) are installed on both border & non-border leaves. −Actrl Prefix Entry (represented by model class actrl::PfxEntry) are installed only on border leaf, other then the following case: Application EPG is in contract with an InstP named InstP-1 under L3Out named L3Out-1. If there is another outside (other then L3Out1 & InstP1) deployed on the EPGs tor or in EPGs VRF (if EPG and L3Out are in different VRF) then actrl prefix entry for the external subnets defined under InstP1 will also get installed on the EPGs tor or EPGs VRF (if EPG and L3Out are in different VRF). Ingress: New behavior −Actrl rules are installed only on the non-border leaf. −Actrl Prefix Entry are installed both on border and non-border leaf.

18. User Configuration It is used for defining policy enforcement direction for the traffic coming to or from an L3Out. Egress and Ingress directions are wrt L3Out. Default will be Ingress. But on the existing L3Outs during upgrade it will get set to Egress so that right after upgrade behavior doesn’t change for them. This also means that there is no special upgrade sequence needed for upgrading to the release introducing this feature. After upgrade user would have to change the property value to Ingress. Once changed, system will reprogram the rules and prefix entry. Rules will get removed from the egress leaf and will get installed on the ingress leaf. Actrl prefix entry, if not already, will get installed on the ingress leaf.

19. Restrictions For Ingress Policy Model This feature doesn’t work for the following cases: Transit: Rules already get applied at Ingress vzAny contract Taboo contract