Condor: Firewall Mirroring UK Condor Week 2004

Outline Problem of Firewalls within a Condor Pool Options to alleviate these problems Our Solution

Firewalls within a Condor Pool Some resource owners have firewalls on their personal workstations Since Condor needs each submit node to be able to talk to every potential execute node, this does not scale well.

Slide based on one from the University of Wisconsin-Madison Job Startup Central Manager Negotiator Collector Submit Machine Execute Machine Schedd Startd Starter Job Steps 1. Startd sends collector ClassAd describing itself. (The Schedd does as well, but it has nothing interesting to say yet.) 2. The user calls condor_submit to submit a job. The job is handed off to the schedd and condor_submit returns. 3. The schedd alerts the collector that it now has a job waiting. 4. The negotiator asks the collector for a list machines able to run jobs and schedd queues with waiting jobs. 5. The negotiator contacts the schedd to learn about the waiting job. 6. The negotiator matches the waiting job with the waiting machine. 7. The negotiator alerts the schedd and the startd that there is a match. 8. The schedd contacts the startd to claim the match. 9. The schedd starts a shadow to monitor the job. 10. The startd starts a starter to start the job. 11. The starter and the shadow contact each other. 11. The starter starts the job. 12. If the job is using the Condor syscall library (typically through being condor_compiled), it will contact the shadow to access necessary files. Submit Shadow Condor Syscall Lib Slide based on one from the University of Wisconsin-Madison

What if the firewall is out-of-step? A Job may still match for the newly added machine to the firewalled resource. This job will not be able to run Parts of the system jam as a result. condor_q on submitting node The other parts of the submit script (maybe also parts of the central node)

A Related problem Similar “jams” occur if part of your pool (or flock of pools) is on a network that is unavailable to some of the other nodes How can we permit jobs from submit nodes that can access the private network to run on these nodes whilst preventing Condor sending jobs from other submit nodes there?

How can we get round this? Restrict the number of submit nodes Automatically update the firewall files Ensure everything is up-to-date Permit pool to evolve whilst persuading Condor to “avoid” going to nodes where the job can’t run Restrict the number of submit nodes Only these nodes need to be updated when new machines are added to the pool. User’s must all have accounts on at least one of the submit nodes. Automatically update the FW files Resource owners who are serious enough about security to have their own firewall are unlikely to want their firewall files messed with by a script which runs as root! Ensure everything is up-to-date Infeasible

Firewall Mirroring (1) Each machine with a firewall declares the fact in its ClassAds: HAS_FIREWALL = TRUE Also, which machines and/or subnets it permits to access its Condor ports (mirroring FW table settings): FW_ALLOWS_113 = TRUE FW_ALLOWS_rjavig6 = TRUE Finally, it needs to export these settings: STARTD_EXPRS = HAS_FIREWALL, FW_ALLOWS_113, \ FW_ALLOWS_rjavig6

Firewall Mirroring (2) To ensure that jobs can only go to resources they can reach. Ensure that they declare their subnet and hostname: MY_SUBNET = 113 MY_HOST = condor Use these value in the following macro which is added to all REQUIREMENTS for jobs from this machine: OK_FOR_THIS_MACHINE = ( \ (HAS_FIREWALL =!= TRUE) || \ (FW_ALLOWS_$(MY_HOST) == TRUE) || \ (FW_ALLOWS_$(MY_SUBNET) == TRUE) ) APPEND_REQUIREMENTS = $(OK_FOR_THIS_MACHINE)

And Private Networks? Same solution can be used for private networks by pretending they have a firewall and declaring which other nodes have access to that network

Conclusion While this solution does not solve the firewalled workstation problem, it does make it nicer to live in their presence!