1. **DRAFT** Doctor Host Maintenance Maintenance changes to OpenStack Nova 17 May 2016 Tomi Juvonen Nokia

2. Preface 2 This document describes roughly changes expected to OpenStack Nova project. Initial discussions have been done with OpenStack Nova project and in OpenStack development and operator mailing lists. Finally all this came together in OpenStack Austin summit with session “Ops: Nova Maintenance - how do you do it?”.

3. Current state 3 Nova PUT /v2.1/​{tenant_id}​/os-services/enable and PUT /v2.1/​{tenant_id}​/os-services/disable APIs to enable/disable nova-compute service to allow/disallow VM scheduling for a host. Service status notification that is sent when nova-compute service status changes (disabled, forced down,…) GET /v2.1/​{tenant_id}​/servers/detail and GET /v2.1/​{tenant_id}​/servers/​{server_id} with microversion 2.16 to have host_status parameter to indicate host status to server owner.

4. Expected changes 4 Nova do not want more specific maintenance states or flow in its own content, but there should be free text field where one can have some external URL / Jira ticket to have more specific state information. This is what is concluded till now with Nova team.

5. Change proposed 5 There is a spec up for review: https://review.openstack.org/310510/2 https://review.openstack.org/310510/2 This has now the smallest possible change to reuse disabled_reason for the external maintenance status and expose it to servers API. This was not the way this should be. Operators do not want to expose existing disabled_reason for VM owner, so there needs to be separate maintenance reason/state field. There should be a notification when this information is changed. External tool needs to catch the notification and it can alarm/notify user about change in maintenance reason/state for his server(s). Polling of servers API will not be acceptable solution while the API also need to be exposed to this new information, so it has all relevant state information to user. New maintenance reason/state field will be as maintenance_reason further in this document.

6. New maintenance_reason to services table in Nova DB. 6 maintenance_reason should be added to services table in Nova DB. This free text field is something that admin can set for nova-compute service to have information about maintenance state or it can also be used for example to give version information about the host. maintenance_reason can have URL to external tool that can tell about maintenance state in more detail. One could see there when underlying host is going for maintenance, host is in maintenance or what ever more detailed information is needed. Version information is then something telling if host has gone trough maintenance already. This information could be used by user who wants to schedule new VM for a host that is already gone trough maintenance. Surely there a things like host aggregate, but this gives some deployments alternative way to approach this problem. Setting maintenance_reason should be done by new API or by modifying existing one.

7. How to expose maintenance_reason 7 Services table is already exposed to servers and os-hypervisors API. Exposing a new field from table to those APIs should not therefor be a very costly operation. While it is self-evident that one need to have all state information relevant to VM owner in servers API, it is not something that anybody should poll frequently to know this. This is why changes in maintenance_reason should also be send via notification. External tool will use this notification to form alarm/notification to be sent to VM owners. This fits to Doctor inspector nicely as it have the topology information to know VMs for host and a capability to send alarms for VM owner.

8. Set “maintenance_reason” API Proposal alternative 1: 8 Have new “maintenance_reason” parameter for existing disable-log-reason API: PUT /v2.1/​{tenant_id}​/os-services/disable-log-reason​ { "host": "host1", "binary": "nova-compute", "disabled_reason": “Maintenance“, “maintenance_reason”: "http://mydetailedreasonurl" } 200 OK { "service": { "binary": "nova-compute", "disabled_reason": “Maintenance", "host": "host1", "status": "disabled“ “maintenance_reason”: "http://mydetailedreasonurl" }

9. Set “maintenance_reason” API Proposal alternative 2: 9 Set “maintenance_reason” with new maintenance-reason API: PUT /v2.1/​{tenant_id}​/os-services/maintenance-reason { "host": "host1", "binary": "nova-compute", “maintenance_reason”: "http://mydetailedreasonurl" } 200 OK { "service": { "binary": "nova-compute", "disabled_reason": "Maintenance", "host": "host1", "status": "disabled“ “maintenance_reason”: "http://mydetailedreasonurl" }

10. Expose “maintenance_reason” API Proposal 10 Expose “maintenance_reason” to APIs already having service state information: GET /v2.1/​{tenant_id}​/servers/detail and GET /v2.1/​{tenant_id}​/servers/​{server_id} 200 OK { "server": { "host_status": "MAINTENANCE", "OS-EXT-SRV-ATTR:maintenance_reason": "http://mydetailedreasonurl"... } GET /v2.1/​{tenant_id}​/os-hypervisors/​{hypervisor_id}​ 200 OK { "hypervisor": { "status": “disabled", "service": { "host": “host1", "disabled_reason": “Maintenance”, "id": 6 "maintenance_reason": "http://mydetailedreasonurl", }, … }

11. Notification Proposal 11 There should be a notification when maintenance_reason changes. This is for service as parameter is also set for service (Note! Cannot send VM/instance specific notification here. That needs to be formed externally by tool/Inspector catching this): { "priority":"INFO", "event_type":"service.update", "timestamp":"2016-03-22 00:46:25.211575", "publisher_id":"nova-compute:controller", "payload":{ "nova_object.version":"1.0", "nova_object.name":"ServiceStatusPayload", "nova_object.namespace":"nova", "nova_object.data":{ "binary":"nova-compute", "report_count":1, "topic":"compute", "host":“host1", "version":3, "disabled":true, "forced_down":false, "last_seen_up":"2016-03-22T00:46:25Z", "disabled_reason":“Maintenance", "maintenance_reason": "http://mydetailedreasonurl", } }, "message_id":"8516b5c7-c6a7-43a4-9ad1-df447f318afb" }