1. **DRAFT** Doctor+Congress PoC @ OPNFV Summit 2016 15 June 2016 Doctor+Congress PoC team

2. PoC Demo Description Doctor: fast and dynamic fault management in OpenStack (DOCOMO, NTT, NEC, Nokia, Intel) Doctor is an OPNFV project implementing fault management framework for high service availability in OpenStack. In this framework, there are two choices for handling fault events which will be notified to users immediately, by OpenStack Congress and OpenStack Vitrage. –Congress is the Policy Evaluation Engine for enforcing flexible and dynamic failure identification policy defined by an Operators demands. –Vitrage is a new Root Cause Analysis Engine for organizing, analyzing and expanding OpenStack alarms & events, yielding insights regarding the root cause of the problems, and deducing the existence of faults before they are directly detected. This PoC shows how fast fault recovery is performed by using these two options, thereby ensuring the required service availability of telecom nodes. http://events.linuxfoundation.org/events/opnfv-summit/extend-the-experience/opnfv-poc-zone 2

3. Demo Scenario 3 Monitor Aodh Ceilometer Manager Virtualized Infrastructure (Resource Pool) Alarm Conf. Application Controller Neutron Resourc e Map Congress 6. Notify all 7. Notify Error 1. Set Alarm on Port event Failur e Policy Monitor Collectd 3. Notify Raw Failure 5. Update State 4. Find Affected 2. Monitor 0. Setup VM2 VM1 VM0 Port0 Bonding Doctor Plugin Doctor Driver A: When both of NIC are down, propagate error to status of Ports connecting DPDK Switch B: When 1 NIC is down, propagate error to status of Ports connecting DPDK Switch API extension for Port state update Port1 Port2 DPDK SwitchSR-IOV NIC 0 NIC 1 NIC 2

4. 4 0. Setup Make Congress fetches Neutron Port info (vif_type and hostname) periodically 1.Create Aodh Alarm definition, specifying notification URI of the App manager, Neutron Port ID and context of port update event (Trigger Error) 2.The collectd gets NIC status from DPDK 3.If collectd gets failure(*), Doctor Plugin of collectd posts doctor event (containing vif_type=normal and hostname) to Congress 4.Congress (Policy engine) evaluates policy with received event and find effected 5.Congress enforce mark status of effected ports to down (using Neutron driver) 6.Neutron notify port update events to Ceilometer 7.Aodh fire the notifications of those port failures to Manager Notification Logic for Demonstration

5. Difference from the PoCs so far Event-driven notification with DPDK stats and collectd Failure mapping in logical resource view 5 Host (Hypervisor) Server (VM) Port (vNIC) Network @ OPNVF Summit 2015, OpenStack Summit Austin @ OPNVF Summit 2016

6. Options (Backup Plans / Improvements) Option A. VM0 uses another port of DPDK Switch instead of SRIOV (The networks are divided by VLAN) Option B. VM1 and VM2 use normal Open vSwitch instead of DPDK Switch Option C. Map raw failure to Nova Instance instead of Neutron Port Option D. Monitor packet loss instead of link state of pNIC port (Improvements) 6

7. Alarm in Aodh aodh alarm create -t event --name "NICFailureAlarm" \ --alarm-action \ --description "NIC failure" \ --enabled True \ --repeat-actions False \ --severity "moderate" \ --event-type port.update.end \ --query "traits.forced_down=string::True;traits.resource_id=string:: “ 7

8. Rules in Congress [All bonded NIC ports down] execute[neutronv2:force_down_port(portid)] :- neutronv2:ports(id=portid, hostid=hostname, vif_type=viftype), doctor:events(hostname=hostname,vif_type=viftype,type="host.nic1.down"), doctor:events(hostname=hostname,vif_type=viftype,type="host.nic2.down") [One NIC port down] execute[neutronv2:force_down_port(portid)] :- neutronv2:ports(id=portid, hostid=hostname, vif_type=viftype), doctor:events(hostname=hostname,vif_type=viftype,type="host.nic1.down") execute[neutronv2:force_down_port(portid)] :- neutronv2:ports(id=portid, hostid=hostname, vif_type=viftype), doctor:events(hostname=hostname,vif_type=viftype,type="host.nic2.down") 8

9. Main Monitor (40 inch) 9 Globe App Status App Manager Log time Recovery---0.04 Reported0.01@ mon 0.02 time Recovery---0.04 Reported0.01@ mon 0.02

10. Sub Monitor (Laptop) Console Slide Deck Horizon –VM List w/ status –Congress Rules (TBC) 10