POP: Building Automation Around Secure Server Deployment Kevin Lux Penn Medicine luxk@upenn.edu @luxk on Slack https://kevinlux.info/

Talk Overview Background info Integrated server deployment Background work for new process Implementation Value-add Demo

Background: Me Intern in Penn Security Lab (SEAS) in 2001 while attending Drexel. Transitioned to full-time while earning a MSE in CIS. Moved to PSOM in 2006. Started building POP in late 2016.

Background: POP POP: PMACS Operations Portal. The integration point for operational systems in PSOM. Built by me. Current codebase is over 43k lines of code.

Background: POP, cont. Users utilize web forms to create, track and execute requests on operational systems. Back-end has multiple modes of execution: schedules, events, triggers, email, sms, slack, etc. Everything codified into the POP ecosystem is usable by everything else.

Background: POP Integration Points The operational groups of PMACS interact with a wide variety of systems on a daily basis. Most of these systems do not talk to each other. Penn Assignments

Background: POP, cont. Full discussion on POP is far beyond the scope of this talk. For more information on POP, see the “Presentations” section of my website at https://kevinlux.info/. Also will be discussed as part of the ISC IT Staff Convention in May.

Server Deployment: Wants/Haves Want: dynamic data. Have: manually updated lists. Want: integration. Have: going to multiple systems. Want: repeatable results. Have: manual processes. Want: verifiable. Have: manual audits. Essentially: Want: simple. Have: not simple.

Server Deployment: Basic Needs Basic wiki page describing the server is created. Server is in KACE inventory. Server is added to critical components. A security scan of the server is sent to the admin.

Staging Work: KACE Got all servers added to KACE (GPO/Ansible) Custom asset type configured for servers in KACE. Network rules established to allow agent check-ins.

Staging Work: Code Created an API for Critical Components. Created an API for KACE. Created an API for Security Center. Built out POP forms to support new systems.

Implementation Admins use a POP form to update server info in KACE… Project use of the server Importance of the server Admins for the server The type of the data on the server The application role of the server

Implementation, cont. Upon submitting the form, POP… Updates KACE with all the information. Checks Critical Components for the server. Adds the server if not found. Requests a security scan from Security Center. Checks the systems wiki for the server page. Creates it if it doesn’t exist.

Implementation, cont. After the form completes, POP will… Periodically check with Security Center to wait for the scan to complete. Upon completion, the scan is reformatted to a report and emailed to the admins. Refreshes the server wiki page daily with the most recent information about the machine. Virtual/physical hardware changes. Networking changes on the server and firewall. These are the core ideas of POP.

Implementation, cont. Server deletion Critical Components and KACE are both updated when servers are decommissioned. These are the core ideas of POP.

Unintended Value-Add Using the data in KACE, we can create an Icinga configuration to: Monitor basic OS-dependent services. Server-specific application status. The default configuration will at least check the server is alive. More exotic checks (e.g. DB specific ports), make this work even more valuable. These are the core ideas of POP.

Value-Add, cont. Using POP as a controller eliminates the “mess of configuration files” that always seems to pop up. Makes the monitoring server hands-off. Custom monitoring configurations are still supported – they are stored in KACE. These are the core ideas of POP.

Conclusion And lastly and somewhat obviously… Single step vs multiple steps means better chance of the work being performed completely and accurately. Addition of server to required systems (e.g. CC) is compulsory. Time reduction for admins. Reduced administration for managers maintaining admins in external applications. These are the core ideas of POP.

Demo Adding a new server.

Q&A Thanks for your attention! Questions? Follow-up communication channels: luxk@upenn.edu @luxk on Slack https://kevinlux.info/