NFV Infrastructure Maintenance Automation by OPNFV Doctor

Doctor: Fault Notification → Maintenance Allow APP owner to get VM’s fault and maintenance notifications from the Cloud Infra https://wiki.opnfv.org/display/doctor/Doctor+Home

Doctor Maintenance Benefits Fast Minimize maintenance sequence (pre-steps) Safe Handshake between APP Manager and Infra Flexible Further steps and policies can be in place

Doctor Maintenance PoC: What’s in? Nova Tag-notification, versioned (object) notification service.update notification (disable nova-compute) Ceilometer/Aodh Event transformer, Event Alarm Horizon – instances - Show server tags Sample Admin Tool Sample App Manager and Web server (Active-Standby)

Doctor Maintenance PoC App Manager Floating IP Nova Ceilometer/Aodh VM1 VM2 Compute1 Compute2 Compute3 Admin Tool

Doctor Maintenance Sequence Admin Tool Nova Ceilometer/Aodh App Manager 1. Disable Compute Host 2. “Service Disabled” Message 3. Mark VMs “going_to_maintenance” 7. “VM Update” Message (“allow_migration” tag) 8. Migrate VM Perform Further Maintenance Steps 4. “VM Update” Message (“going_to_maintenance” tag) 6. Tag “allow_migration” on the VM 5. Switch Active and Standby node

Doctor Profiler Total time cost: 472(ms) ==============================================================================> |Monitor|Inspector |Controller|Notifier|Evaluator | |112 |730 |? |? |? | | | | | | | | | | | link down:0 | | | | | | | | | raw failure:112 | | | | | | | | found affected:? | | | | | | | set VM error:312 | | | | | | marked host down:842 | | | | | notified VM error:? | | | | transformed event:? | | | evaluated event:? | | fired alarm:? | received alarm:472

Doctor: How to Setup Infra as an Admin Nova/Neutron Make sure Nova/Neutron sends out Notification https://docs.openstack.org/developer/nova/notifications.html https://docs.openstack.org/admin-guide/networking-adv-operational-features.html#notifications Ceilometer/Aodh Enable Event Transformer https://docs.openstack.org/admin-guide/telemetry-events.html Congress Set Correlation Rules http://docs.opnfv.org/en/latest/submodules/doctor/docs/release/configguide/feature.configuration.html

Doctor: How to Setup Alarm as an User Nova/Neutron Create Resource (VM, Port, ...) Aodh Create Event Alarm with Resource-ID https://docs.openstack.org/developer/aodh/event-alarm.html OR Create Event Alarm with HOT https://docs.openstack.org/developer/heat/template_guide/openstack.html#OS::Aodh::EventAlarm

Doctor: Alarm Example { --- snip --- u'reason_data': { u'type': u'event', u'event': { u'event_type': u'instance.update', u'traits': [ [u'display_name', 1, u'server1'], [u'uuid', 1, u'6bf4f683-7a17-44df-a2eb-aaf3c38e6e90'], [u'service', 1, u'nova-api:controller'], [u'availability_zone', 1, u'default'], [u'tenant_id', 1, u'1f5ebf625b95479abc98e3ae85dd3871'], [u'tags', 1, u"[u'active', u'no_migration', u'retirement', u'retirement2', u'retiremet']"], [u'display_description', 1, u'server1'], [u'state', 1, u'active'], [u'power_state', 1, u'running'], [u'request_id', 1, u'req-0856f167-07b6-4bf6-9b7e-af0122da4963'], [u'user_id', 1, u'111c2c5f88cc44eca4e35b16f996932a'], [u'project_id', 1, u'1f5ebf625b95479abc98e3ae85dd3871'], [u'metadata', 1, u"{u'metering.stack': u'b890d2db-c9b2-47e5-b6ff-d0136c05eb7b'}"] ], }}} https://github.com/openstack/nova/blob/master/nova/objects/flavor.py