1. Octopus Deploy Customization and Automation with .NET and PowerShell
There and back again

2. Who am I? Ian Paullin Architect/DevOps/.NET Programmer
5 years with Avanade, 10 years in .NET
New Dad
Seahawks Fan

3. What are we talking about?
Our story with Octopus
Intro to Octopus REST API
How to get started
Demos
Octo.exe
PowerShell Demo
Post message to Slack

4. Why are we talking about this?
Octopus is awesome
Automation is awesome
Octopus is extremely extensible
Octopus extensibility + automation = more awesome

5. I thought it would be like..

6. In retrospect..

7. Our Story
Take Octopus Deploy to the masses. Ensure 24/7 uptime. Limit specific users to deploy to production.
February 2014 – May 2014 (Pilot approx. 10 teams)
June 2014 – February 2015
Goal projects on board by Feb/Mar 2015
February 2015 upgrade: from to (eventually to )
Problems emerged with upgrade
March 2015 – October 2015
Stabilization + growth
November 2015 – January 2016
More stabilization and automation
More users/projects/servers
Testing Octopus 3.1.x – 3.2.x

8. Where are we today? 5 Octopus servers with 5 NuGet servers
Total tentacles: 1564
Total # of project groups: 255
Total # of projects: 509
Busiest Day: Wednesday – avg. 105 deployments
Weekly deployment avg:
NuGet packages: A lot (e.g. 200Gb on just one of our NuGet servers)

9. Scaling
Octopus can do it, but we didn’t realize all contributing factors
NuGet issues – with free NuGet.Server
Was recommended to use NuGet.Lucene or NuGet Gallery
MyGet.org/ProGet
No feed maintenance means large feeds over time (3-10gb and thousands of packages)
Solution
Reduce packages in feeds (manually ))
IIS Fileshares circumvent issue with NuGet client DLL
Octopus Server File Handles
Max capacity of listening tentacles (listening uses 1 thread per machine!)
NuGet communication issues
Determine upper limit
Balance team tentacle size across number of servers
Switch to polling mode for tentacles (no thread per each polling tentacle like listening mode)
Deleting old/stale projects

10. Lessons Learned Research was difficult (at the time) Estimation
No real documentation on Octopus on massive scaling (everything is relative)
My failure to reach out before hand to determine how to best scale
Our use case maybe outside of scope of Octopus? Not quite – but 3.x changes everything about Octopus Deploy
Estimation
Pilot proved it could work but estimating scaling or load was very obfuscated by numerous aspects (NuGet, tentacle communications, server configuration, etc.)
Initial estimate of teams moving to Octopus was way too low.
Monitoring
SumoLogic – we had for IIS for NuGet.Server but didn’t use it correctly at beginning
AppDynamics – added late to really foresee issues but invaluable today
Not constantly watching environments – too many machines added

11. Lessons Learned (con’t)
RavenDB limitations of scale
Primarily licensing issue limiting to 3 CPU cores and 6GB (embedded/ISV)
NancyFX is nice but..
Shielded information about user sessions from us
Now Octopus 3.x has logging feature for sessions
Understand Limits/Tolerance
If you don’t know them, find them out (break your stuff if you can)
If you can’t find them, constantly watch out for multiple factors (monitoring)
Monitoring especially at enterprise scale is not optional

12. What really worked well
Provisioning
Automate process to create a team within Octopus. By hand, could take 25 to 30 minutes – sometimes a lot longer. Automating process meant 30 to 45 seconds of filling out fields and executing a deployment
Automatically create NuGet feed on NuGet server for each project (isolation)
Integration with Active Directory Groups and TFS AD Groups makes life much easier
Can build a customize new Octopus Deploy server with one provisioning script
Permissions
Permissions model is complicated, but extremely effective for our needs. This is 100% automated from the provisioning process
Reporting
Reports on how many deployments; how frequent; to which environments, etc.
Error reports and frequency tracking

13. What really worked well (con’t)
Synchronizing user accounts with internal service
Get list of users that are approved for production deployments for specific security teams – checks every hour across all our instances
Monitoring
AppDynamics – late start but very helpful in keeping a watch on stability
Server metrics to gauge stability and measure improvements
Maintenance
Weekly reboots – fully automated
NuGet cleanup – not automated (yet)
Changing IIS feeds to filesharing
Constant improvements
Weigh pros/cons, time available and payoff
Target daily or common tasks/procedures
Backlog one-offs, random issues, odds/ends

14. Odds and ends User Education
Some of our users still don’t use optimize Octopus. They can cause lots of problems with an entire instance and can make server unstable at least for (backups, canceling/starting deployments, long running processes, etc.)
Documentation + videos + guidance sessions still not enough for some
Good users are always willing to learn and improve but it takes time to see benefits
Excellent users are fewer in number, but really good in maximizing octopus
Teams change therefore users change; re-education is cyclical and therefore problematic over time

15. Support directly from Octopus Deploy
Great support from small company
Weekly meetings, research and testing extremely helpful
Helping us now with our plans to migrate to 3.1.x – 3.2.x
Continuing guidance and support on architecture design and migration to 3.x

16. Keys to winning Perseverance Watch closely and at high-level
Keep at it
Watch closely and at high-level
Monitoring + reports help make sense of what’s going on and what may happen
Continually improve aspects in and around Octopus Deploy
Improvements or tweaks, no matter how small, add value over time
Automate
With Octopus REST API, we can script repeatable tasks for everything
Customize
With Octopus REST API, we can extend Octopus in almost any way we want

17. One key to winning: The Octopus REST API
View the API at octopus server>/api
Documentation:

18. Octopus REST API Everything flows through the REST API Internally
Externally
Octo.exe

19. Octopus REST API - Demos
Look at the octo.exe
Query Octopus Deploy server directly via PowerShell
Simple test message to Slack during deployment

20. Octopus REST API – DiY with Octo.exe
Great for automating deployments after build
Quick Demo with Octo.exe

21. Octopus REST API – DiY with PowerShell/C#
You need 3 DLLs (found in tentacle/server installation folder)
Newtonsoft.Json.dll
Octopus.Client.dll
Octopus.Platform.dll
REST API URL exposes what’s available
APIKey and appropriate permissions for the user account (or service account) dictate what’s allowed for interaction; no username/password!
Quick Demo for PowerShell
Report example

22. Octopus REST API – Sample Report

23. Automation/Customization Ideas
Examples
Any task done more than two or three times
Windows Tasks that can be scheduled indefinitely
Write to log or database for auditing history
Integration with 3rd party tools/platforms
Synchronization of users – if a small subset of users is allowed
Scheduled reboots – if necessary
Security checks – especially for AD groups
Integration with Slack for messages of deployments

24. Integration with Slack
Slack is a 3rd party collaboration/messaging platform that allows users to chat as well as programmable bots or external parties
Idea: for every deployment in Octopus, send message to Slack
Demo

25. Octopus REST API conclusion
Not enough to just automate deployments. There’s lots of things around deployments that need automation as well.
Without the REST API, it’s much harder if not impossible to customize and automate as we did.
Many creative solutions possible – imagination and time are only barriers.

26. Thanks to.. Octopus Deploy team Thadd Parker (@tptechgeek)
Damian Brady
Vanessa Love
Paul Stovell
Thadd Parker
For stats and sample report and sample C# code
Gandalf the Pug