1. Condor: Firewall Mirroring
UK Condor Week 2004

2. Outline
Problem of Firewalls within a Condor Pool
Options to alleviate these problems
Our Solution

3. Firewalls within a Condor Pool
Some resource owners have firewalls on their personal workstations
Since Condor needs each submit node to be able to talk to every potential execute node, this does not scale well.

4. Slide based on one from the University of Wisconsin-Madison
Job Startup
Central Manager
Negotiator
Collector
Submit Machine
Execute Machine
Schedd
Startd
Starter
Job
Steps
1. Startd sends collector ClassAd describing itself. (The Schedd does as well, but it has nothing interesting to say yet.)
2. The user calls condor_submit to submit a job. The job is handed off to the schedd and condor_submit returns.
3. The schedd alerts the collector that it now has a job waiting.
4. The negotiator asks the collector for a list machines able to run jobs and schedd queues with waiting jobs.
5. The negotiator contacts the schedd to learn about the waiting job.
6. The negotiator matches the waiting job with the waiting machine.
7. The negotiator alerts the schedd and the startd that there is a match.
8. The schedd contacts the startd to claim the match.
9. The schedd starts a shadow to monitor the job.
10. The startd starts a starter to start the job.
11. The starter and the shadow contact each other.
11. The starter starts the job.
12. If the job is using the Condor syscall library (typically through being condor_compiled), it will contact the shadow to access necessary files.
Submit
Shadow
Condor
Syscall Lib
Slide based on one from the University of Wisconsin-Madison

5. What if the firewall is out-of-step?
A Job may still match for the newly added machine to the firewalled resource.
This job will not be able to run
Parts of the system jam as a result.
condor_q on submitting node
The other parts of the submit script
(maybe also parts of the central node)

6. A Related problem
Similar “jams” occur if part of your pool (or flock of pools) is on a network that is unavailable to some of the other nodes
How can we permit jobs from submit nodes that can access the private network to run on these nodes whilst preventing Condor sending jobs from other submit nodes there?

7. How can we get round this?
Restrict the number of submit nodes
Automatically update the firewall files
Ensure everything is up-to-date
Permit pool to evolve whilst persuading Condor to “avoid” going to nodes where the job can’t run
Restrict the number of submit nodes
Only these nodes need to be updated when new machines are added to the pool.
User’s must all have accounts on at least one of the submit nodes.
Automatically update the FW files
Resource owners who are serious enough about security to have their own firewall are unlikely to want their firewall files messed with by a script which runs as root!
Ensure everything is up-to-date
Infeasible

8. Firewall Mirroring (1)
Each machine with a firewall declares the fact in its ClassAds:
HAS_FIREWALL = TRUE
Also, which machines and/or subnets it permits to access its Condor ports (mirroring FW table settings):
FW_ALLOWS_113 = TRUE
FW_ALLOWS_rjavig6 = TRUE
Finally, it needs to export these settings:
STARTD_EXPRS = HAS_FIREWALL, FW_ALLOWS_113, \ FW_ALLOWS_rjavig6

9. Firewall Mirroring (2)
To ensure that jobs can only go to resources they can reach.
Ensure that they declare their subnet and hostname:
MY_SUBNET = 113
MY_HOST = condor
Use these value in the following macro which is added to all REQUIREMENTS for jobs from this machine:
OK_FOR_THIS_MACHINE = ( \
(HAS_FIREWALL =!= TRUE) || \
(FW_ALLOWS_$(MY_HOST) == TRUE) || \
(FW_ALLOWS_$(MY_SUBNET) == TRUE) )
APPEND_REQUIREMENTS = $(OK_FOR_THIS_MACHINE)

10. And Private Networks?
Same solution can be used for private networks by pretending they have a firewall and declaring which other nodes have access to that network

11. Conclusion
While this solution does not solve the firewalled workstation problem, it does make it nicer to live in their presence!