PMACS Operations Portal (POP) Kevin Lux, Senior Systems Programmer Penn Medicine luxk@upenn.edu @luxk on Slack

Overview Background info How POP addresses the challenges Design of POP and an abbreviated feature list Current production use cases of POP Discussion of an operation before and after POP (and demo!)

Background: Me Intern in Penn Security Lab (SEAS) in 2001 while attending Drexel. Transitioned to full-time while earning a MSE in CIS. Moved to PSOM in 2006. Started building POP in late 2016.

Background: Group Member of SYS-WIN. Ultimately part of Systems. Work very closely with infrastructure.

Background: Operational Systems The operational groups of PMACS interact with a wide variety of systems on a daily basis. Most of these systems do not talk to each other. Penn Assignments

Background: Challenges Systems receives many tickets. Many account tasks are simple, but switching between them is expensive. Manual processes and repeatability are usually at odds.

Background: Challenges (cont.) Operational requests must be addressed in a timely manner. Little, if any, integration with University systems. Desire to push operational tasks down to application owners, if possible.

Background: Challenges (cont.) LSP tools are scattered among a variety of systems. Desire for unified and consistent logging. You know… normal challenges…

Solution: POP Web-based portal for people to create, track and execute tasks. Strives to tie together all operational systems under one main umbrella. Built to be agile. These are the core ideas of POP.

POP: Core Ideas Design a system to make common requests easy. Apply structure and context to data, backed by code. Designed with security awareness from the start. Integrate existing systems. These are the core ideas of POP.

POP: Ease of Use Users access POP via the web and are presented with simple forms to fill out. Most common operations are “POPable”; many are even self-service. Wizards make more complicated operations easier to accomplish. Each form is designed to do exactly one task well. Primary interface for POP is a website. POP tries to make requests easy for requestor and requestee.

POP: Structured Data The problem with tickets: they just contain text. Each field on a POP form has a specific meaning and can be validated to verify correctness. Forms can undergo extended validation (such as checking AD or executing a web service call). Text has no real value until it is interpreted. POP does not lose the value of data in fields; it is kept with the request.

POP: Security Two primary security levels on forms: execute and save. Authorization is determined by AD group membership or an arbitrary piece of code. Allows for very fine-grained permission control.

POP: Integration Each operational system is generally a silo. Many systems didn’t have any kind of API or it was incomplete. Integrated systems are used as ‘sources of truth’ – nearly all operations occur directly on the target system in real-time.

POP: Moving on… Software design and security considerations Feature List Use cases Sample operation Demo

POP: Software Design Basic design unit is the form. A new operation (user reset, new user, etc.) is a new form. Forms are split in to two components: data and code. Data file describes the form fields and permissions. Code file contains programming code. Programmer extends built-in hooks on the form to give functionality for different actions/events. One form, one operation.

POP: Software Design (cont.) Code is pulled in at the moment it is required to run. Slight loss in performance vs being able to update the system without restarts. Saved forms always receive the latest version of the code. The runtime environment automatically includes helper functions that abstracts common programmer tasks away. e.g. connecting to LDAP, databases, etc.

POP: Security Model Users in POP do not require elevated privileges anywhere. Users can be given very specific permissions that applications may not support. A common permission set (Active Directory) specifies user access – no need to have more permissions in each app.

POP: Security Model (cont.) Ability for POP to impersonate users is present. Users can store encrypted credentials that the server will use for certain tasks. POP runs at different access levels depending on where it is running and what it needs to do. POP rotates the passwords it uses.

POP: Logging Actions taken for each user account are logged; builds a transcript. Inputs and results are also logged. Logs can be sent to a syslog compatible server for analysis and archiving.

POP: Advanced Features Forms can call other forms leading to complex orchestrations called plans. The code is multi-platform. Built-in scheduler. SMTP processor. Forms can contain other forms as subforms.

POP: Advanced Features (cont.) Instant messaging integration (presently Jabber and Slack) to alert team members of new work. All forms are automatically available as a REST service to authorized users. Templated forms allow for perceived customization, while maintaining the same code base. Forms can have an execution host specified.

POP: Advanced Features (cont.) Batch processing of all forms. Forms in a plan can be executed in a multi-threaded fashion. Background processing of forms. Forms are duplicate checked.

POP: Use Case Exploration Use cases currently in production: Active Directory Management Server Data Mining Application Ownership Penn Community Others Future work Discussion/demo of a simple operation.

POP: Active Directory Management Suite of forms for management of users and computers. Customized reports. Integration of Macs in AD (e.g. FileVault). Provisioning of DFS home folders.

POP: Server Data Mining KACE is used for inventorying all servers and stores meta data about projects and ORGs. POP generates server lists and stores detailed server information in our wiki. Built a query language to allow managers to report resource usage for servers/projects to enable chargebacks.

POP: Application Ownership We support a variety of AD authenticated apps for users that aren’t necessarily desktop users. App users may not have LSPs. Do not want admins to be LSPs. Push application account management down to application owners. Complication: users are all over AD.

POP: Application Ownership (cont.) Designed standardized setup for applications. 3 groups: App Account Team – users who are application owners for ‘App’ App Managed Users – users who can be ‘managed’ by above App User Access – users who can access the app App owners can perform limited operations on the users they support, essentially acting as a LSP. Can be more or less restrictive depending on the application. Standard setup allows all applications to share code (using templates).

POP: Penn Community Diverse users: some are UPHS employees, some are University staff, some are students, etc. AD has been modified to include Penn Community affiliations on each user account. Affiliation data is maintained by POP. Automatically disables accounts due to affiliation changes.

POP: More Use Cases Pushing data to other systems (project lists to Kanbanize). DNS management (Assignments and Infoblox). Synchronizing listserv membership with AD group membership.

POP: Future Work Approvals: the ability for forms to be blocked pending user approvals (thought: data privilege). Billing: create a pipeline to enable automatic billing of server resources. Public-facing portal: system for end users to make requests and perform self-service operations for their accounts.

POP: Future Work (cont.) Automatic systems monitoring: use KACE data (installed software plus machine configuration) to generate tags for a server. Use those tags to automatically generate Icinga2 configurations.

POP: Sample AD Operation Compare / contrast pre-POP and post-POP workflow. Live demo follows (honest!).

POP: Sample AD Operation User Password Reset Changes a user’s password when they have forgotten it and notifies the user of the new password. Common operation (especially with as many users as we have). Privileged operation.

Pre-POP: User Reset LSP Team: SYS: LSP puts in a KACE ticket with username (ideally). SYS: Receive ticket. Dig out necessary details from ticket. Find user in AD. Create a new password, assign it to user. Take a boilerplate message, update it and SecureShare it to the user/LSP. Close the ticket in KACE.

POP: User Reset LSP Team: SYS: Fills out userreset form. POP verifies username, submits a ticket and then saves structured info. SYS: Loads saved form. Hits execute. Form does all work including updating Active Directory and KACE and sending a SecureShare in approximately 15 seconds. Bonus: All POP operations are logged, creating a detailed log for each user along with an action log.

POP: User Reset Evaluation User resets now happen exactly the same way each and every time. POP automatically generates tickets for requests that require them. The admin need only give the request the once over and simply hit “Execute”. The ease of processing allows admins to do it quickly and with minimal mental impact.

POP: User Reset Evaluation (cont.) Work can be performed by admins or designees. Ticket request language is standardized. The most obvious benefits are when you scale POP up: Different types of requests Dozens of LSPs Lots of users Etc.

POP: Outcomes 75% of all Systems tickets are now POP requests. Based on our numbers, POP is 19x more effective at processing user reset/new user requests than doing it manually. This translates into many weeks of productivity gained.

POP: Platform Programmed in Perl; 1 developer; ~ 35kloc. 275 forms programmed (not all in production). Main site runs under FastCGI on IIS Linux execute host is Apache.

POP: Demo User reset See what LSPs see and then what an admin sees.

POP: Q&A Thanks for your attention! Questions? Follow-up communication channels: luxk@upenn.edu @luxk on Slack http://kevinlux.info/