Doctor Host Maintenance Maintenance changes to OpenStack Nova 13 Jun 2016 Tomi Juvonen Nokia

Preface 2 This document describes roughly changes expected to OpenStack Nova project. Initial discussions have been done with OpenStack Nova project and in OpenStack development and operator mailing lists. Finally all this came together in OpenStack Austin summit with session “Ops: Nova Maintenance - how do you do it?”.

Current state 3 Nova PUT /v2.1/​{tenant_id}​/os-services/enable and PUT /v2.1/​{tenant_id}​/os-services/disable APIs to enable/disable nova-compute service to allow/disallow VM scheduling for a host. Service status notification that is sent when nova-compute service status changes (disabled, forced down,…) GET /v2.1/​{tenant_id}​/servers/detail and GET /v2.1/​{tenant_id}​/servers/​{server_id} with microversion 2.16 to have host_status parameter to indicate host status to server owner.

Expected changes 4 Nova do not want more specific maintenance states or flow in its own content, but there should be free text field where one can have some external URL / Jira ticket to have more specific state information. This is what is concluded till now with Nova team. Also some further discussion risen to have change to address different kind of host detailed information. There is a spec up for review:

New host_details field to services table in Nova DB. 5 host_details should be added to services table in Nova DB. This free text field is something that admin can set for nova-compute service to have information about maintenance state or it can also be used for example to give some version information about the host. Because other than maintenance usage possibilities the field might be host_details instead of something like host_maintenance_reason. Setting of host_details should be done by new API or by modifying existing one.

How to expose host_details 6 Services table is already exposed to servers and os-hypervisors API. Exposing a new field from table to those APIs should not therefor be a very costly operation. While it is self-evident that one need to have all state information relevant to VM owner in servers API, that API is not something that one should poll frequently to know this. This is why changes to host_details should also be available via notification. External tool will use this notification to form alarm to be sent to VM owners/tenants. This fits to Doctor Inspector nicely as it have the topology information to know VMs for host and a capability to send alarms for VM owner trough AODH.

Setting “host_details” API Proposal alternative 1: 7 Have new “host_details” parameter for existing disable-log-reason API: PUT /v2.1/​{tenant_id}​/os-services/disable-log-reason​ { "host": "host1", "binary": "nova-compute", "disabled_reason": “Maintenance“, “host_details”: " } 200 OK { "service": { "binary": "nova-compute", "disabled_reason": “Maintenance", "host": "host1", "status": "disabled“ “host_details”: " }

Setting “host_details” API Proposal alternative 2: 8 Set “host_details” with new maintenance-reason API: PUT /v2.1/​{tenant_id}​/os-services/maintenance-reason { "host": "host1", "binary": "nova-compute", “host_details”: " } 200 OK { "service": { "binary": "nova-compute", "disabled_reason": "Maintenance", "host": "host1", "status": "disabled“ “host_details”: " }

Expose “host_details” to servers and hypervisors APIs 9 GET /v2.1/{tenant_id}/servers/{server_id} GET /v2.1/{tenant_id}/servers/detail PUT /v2.1/{tenant_id}/servers/{server_id} POST /v2.1/{tenant_id}/servers 200 OK { "server": { "host_status": "MAINTENANCE", "host_details": " } GET /v2.1/​{tenant_id}​/os-hypervisors/​{hypervisor_id}​ 200 OK { "hypervisor": { "status": “disabled", "service": { "host": “host1", "disabled_reason": “Maintenance”, "id": 6, "host_details": " "forced_down": false, }, … } Note! Also ”forced_down” added to hypervisors API!

10 GET /v2.1/​{tenant_id}​/os-services 200 OK { "services": [ { "id": 1, "binary": "nova-compute", "disabled_reason": "Maintenance", "host": "host1", "state": "up", "status": "disabled", "updated_at": " T13:42: ", "forced_down": false, "zone": "internal“, "host_details": " }, … ] } Expose “host_details” to services API

Notification Proposal 11 There should be a notification when host_details changes. This is for service as parameter is also set for service (Note! Cannot send VM/instance specific notification here. That needs to be formed externally by tool/Inspector catching this): { "priority":"INFO", "event_type":"service.update", "timestamp":" :46: ", "publisher_id":"nova-compute:controller", "payload":{ "nova_object.version":"1.0", "nova_object.name":"ServiceStatusPayload", "nova_object.namespace":"nova", "nova_object.data":{ "binary":"nova-compute", "report_count":1, "topic":"compute", "host":“host1", "version":3, "disabled":true, "forced_down":false, "last_seen_up":" T00:46:25Z", "disabled_reason":“Maintenance", "host_details": " } }, "message_id":"8516b5c7-c6a7-43a4-9ad1-df447f318afb“ }