Octavia Hands-On Lab Adam Harwell Carlos Garza Franklin Naval German Eichberger Michael Johnson Al Miller Susanne Balle Stephen Balukoff

Are you ready? Grab a USB key or download from http://tinyurl.com/octaviaWorkshop-tar-bz A computer with 8+ GB of RAM VMware Workstation/Fusion - trial versions of both included on the USB drive Windows users need ssh client (putty) or work in the vmware window Copy OctaviaWorkshop from USB drive Unzip with tar –jxvf or 7zip on windows Start VMWare - make sure VT-X and nested Virtualization is enabled VM also at ... johnsom can cover this topic

VMWare Fusion Setup (VT-X Setting) Mac I had to double-click the image in Finder, it didn't want to load from within VMWare's file select, and when I did so, it said "This virtual machine appears to be in use." and I had to click "Take Ownership". It then asked me if I wanted to upgrade the VM, and I said "don't upgrade". --Adam

Agenda Introduction and Architecture Operations Troubleshooting How to contribute Q&A

Introduction and Architecture

Octavia High Level Architecture

Operations

Ground rules Slide: https://goo.gl/O9c5jn To login User: ubuntu Pwd: ubuntu sudo su - stack cd /opt/stack/devstack/ . openrc admin admin script /dev/null ← This will fix the devstack’s screen problem now you are good to go

Create a load balancer The vm contains two web servers on port 80, so as user admin: nova list - take note of the two ips LB: neutron lbaas-loadbalancer-create --name lb1 private-subnet Check if ACTIVE with neutron lbaas-loadbalancer-list - take note of VIP Listener: neutron lbaas-listener-create --loadbalancer lb1 --protocol HTTP --protocol- port 80 --name listener1 Pool: neutron lbaas-pool-create --lb-algorithm ROUND_ROBIN --listener listener1 -- protocol HTTP --name pool1 For each member: neutron lbaas-member-create --subnet private-subnet --address <ip> --protocol-port 80 pool1 Test: curl <vip>

Get information about amphora - 1 As admin: Show all amphora vms: nova list --name amphora Amphora info commands: curl -k --cert /etc/octavia/certs/client.pem https://<ip on lb-mgmt-net>:9443/0.5/info curl -k --cert /etc/octavia/certs/client.pem https://<ip on lb-mgmt-net>:9443/0.5/details curl -k --cert /etc/octavia/certs/client.pem https://<ip on lb-mgmt- net>:9443/0.5/listeners curl -k --cert /etc/octavia/certs/client.pem https://<ip on lb-mgmt- net>:9443/0.5/listeners/ <listener-id>

Get information about amphora - 2 As admin: Log in to a VM: ssh -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@<ip on lb-mgmt-net> -- good for debugging amphora changes Check on agent: sudo service amphora-agent status Check on haproxy: sudo service haproxy-<listener-id> status

Force/Simulate a fail over -1 Set up health monitoring: Edit /etc/octavia/octavia.conf Uncomment controller_ip_port_list and set to controller_ip_port_list = 172.24.4.1:5555 (this is some voodoo due to some bug in devstack we still need to fix) Restart o-cw: ./rejoin-stack.sh find o-cw screen; ctrl-c; cursor up; return find o-hm screen; ctrl-c; cursor up; return ctrl-a d Create another loadbalancer (see the other slide) – make sure to use different names, e.g. lb2, listener2, etc.

Force/Simulate a fail over - 2 Perform a failover: ssh into the amphora: ssh -i /etc/octavia/.ssh/octavia_ssh_key ubuntu@<ip on lb- mgmt-net> Stop the amphora agent: sudo service amphora-agent stop Observe the fail over: in one window curl the vip: while :; do curl <vip>; done; in the other one monitor nova (repeat until ip changes): nova list --name amphora

Troubleshooting

Increase Log Level Edit /etc/octavia/octavia.conf Uncomment debug = False and replace False with true. Alternatively you can also uncomment verbose=False and replace with True if you only require INFO level logging. We recommend DEBUG in most cases. Restart o-cw, o-api, o-*: ./rejoin-stack.sh find o-* screen; ctrl-c; cursor up; return repeat until all services you like to debug are restarted ctrl-a d

Log files on the host Review /opt/stack/logs ls /opt/stack/logs/o-*.log o-api.log o-cw.log o-hk.log o-hm.log o-api.log - logs the api server. Useful to see if requests made it to the system and if they actually got executed o-cw.log - logs the controller worker. This does most of the work and this logs will usually aid in debugging problems with nova, neutron, etc. o-hk.log - logs the housekeeping manager. Here we do the deletion of amphora marked for deletion, certificate rotation (the ones on the amphora), etc. o-hm.log - logs the information about the healthmanager and heartbeats the amphorae send; this is good for failover debugging

Log files on the amphora Ssh into the amphora cd /var/log/upstart/ sudo sudo less amphora-agent.log Useful to review if the amphora is sending heartbeats and if the API commands were successful; essential for contoller-worker <- -> amphora communication debugging

Octavia Database mysql use octavia show tables; There is some interesting stuff but let’s just look at Amphora table - information about all the amphora in the system Load_balancer, Listener, Member, Pool, Health_monitor - what you expect amphora_health - this has a timestamp of the last time we received a heartbeat from the amphora

How to contribute? For more information: Our next presentation “LBaaS V2 - Liberty and beyond” – 2:40 Thursday Weekly meetings Wednesdays, 20.00 UTC Freenode IRC: #openstack-lbaas https://wiki.openstack.org/wiki/octavia http://www.octavia.io https://launchpad.net/octavia https://github.com/openstack/octavia

Q&A