22 nd Oct 2008Euro Condor Week 2008 Barcelona 1 Condor Gotchas III John Kewley STFC Daresbury Laboratory

22 nd Oct 2008Euro Condor Week 2008 Barcelona 2 Outline In this talk we will look at a few aspects of condor that frequently cause problems for new users. Custom attributes / ClassAds Arguments and Environment variables Connectivity (especially Firewalls)

22 nd Oct 2008Euro Condor Week 2008 Barcelona 3 What is a "gotcha" From Wikipedia: "A 'gotcha' is a feature of a system, a program or a programming language that works in the way it is documented, but is counter-intuitive and almost invites mistakes because it is both enticingly easy to invoke and completely unexpected and/or unreasonable in its outcome."

22 nd Oct 2008Euro Condor Week 2008 Barcelona 4 Custom attributes How do I declare my own attributes so my jobs can distinguish between Red Hat 9 and other Linuxes? On Red Hat 9 machines add the following to its configuration: OPSYS_FLAVOUR = RH9 OPSYS_FLAVOUR = "RH9" STARTD_EXPRS = OPSYS_FLAVOUR STARTD_ATTRS = OPSYS_FLAVOUR, $(STARTD_ATTRS) It can then be matched with a REQUIREMENTS sta tement: REQUIREMENTS = (OPSYS_FLAVOUR = RH9) REQUIREMENTS = (OPSYS_FLAVOUR == RH9) REQUIREMENTS = (OPSYS_FLAVOUR == "RH9") REQUIREMENTS = (OPSYS_FLAVOUR =?= "RH9") √

22 nd Oct 2008Euro Condor Week 2008 Barcelona 5 Custom Attributes 1.Declare the attribute and match against it 2.Always quote non-numerics 3."Export" the attribute by adding to STARTD_ATTRS, rather than STARTD_EXPRS 4.Remember the use of the meta operators: =?= and =!=

22 nd Oct 2008Euro Condor Week 2008 Barcelona 6 How do I pass ROOT=C:\ to Windows? ENVIRONMENT = ROOT=C:\ ARGUMENTS = first_param ENVIRONMENT = ROOT=C:\\\ ARGUMENTS = first_param ENVIRONMENT = ROOT=C:\\ ARGUMENTS = first_param ENVIRONMENT = ROOT=C:\ □ ARGUMENTS = first_param ENVIRONMENT = ROOT=C:\\ ARGUMENTS = first_param ENVIRONMENT = ROOT=C:\|DUMMY="" ARGUMENTS = first_param ROOT=C:ARGUMENTS = …ROOT=C:\ARGUMENTS = … ROOT=C:\\ARGUMENTS = … ROOT=C:ARGUMENTS = … √ √ ENVIRONMENT = ROOT="C:\" ARGUMENTS = first_param ? Environment variables

22 nd Oct 2008Euro Condor Week 2008 Barcelona 7 Environment variables How about passing more than one environment variable on Windows and/or Linux? E1=AB □E2=DE E1=AB E2=DE ENVIRONMENT = E1=AB|□E2=DEENVIRONMENT = E1=AB□|□E2=DE ENVIRONMENT = E1=AB□|E2=DEENVIRONMENT = E1=AB|E2=DE √√

22 nd Oct 2008Euro Condor Week 2008 Barcelona 8 Environment Variables What about using newlines for better formatting when passing lots of environment variables? E1=AB □E2=DE E1=AB E2=DE ENVIRONMENT = E1=AB;□\ E2=DE ENVIRONMENT = E1=AB□;\ E2=DE ENVIRONMENT = E1=AB\ ;E2=DE ENVIRONMENT = E1=AB;\ E2=DE √

22 nd Oct 2008Euro Condor Week 2008 Barcelona 9 Environment variables Remember a trailing backslash will escape the newline, even if you add trailing whitespace or escape it with another backslash, or both. Be careful about blanks and tabs in the middle of the environment line; they may render your declarations impotent. Probably better to avoid whitespace. OR, even better …

22 nd Oct 2008Euro Condor Week 2008 Barcelona 10 The "new" syntax [Paraphrased from condor_submit man pages] Put double quotes around the entire string. To insert a double quote, repeat it. Use whitespace to separate arguments. To include whitespace, surround by single quotes. To insert a single quote, repeat it inside a single-quoted section.

22 nd Oct 2008Euro Condor Week 2008 Barcelona 11 How are previous gotchas handled? The following all work as expected for passing environment variables: ENVIRONMENT = "ROOT=C:\" ENVIRONMENT = "E1=AB E2=AC" ENVIRONMENT = "E1=AB \ E2=AC"

22 nd Oct 2008Euro Condor Week 2008 Barcelona 12 What? No Gotchas? How do I setup an environment variables for a single quote or a double quote? DQUOTE="\"" DQUOTE='"' SQUOTE="'" # in old Syntax ENVIRONMENT = DQUOTE="\"" ENVIRONMENT = DQUOTE='"' ENVIRONMENT = SQUOTE="'"

22 nd Oct 2008Euro Condor Week 2008 Barcelona 13 DQUOTE # New Syntax # Trying to produce DQUOTE='"' ENVIRONMENT = "DQUOTE=''''""''''" # Trying to produce DQUOTE="\"" ENVIRONMENT = "DQUOTE=""\"""""

22 nd Oct 2008Euro Condor Week 2008 Barcelona 14 SQUOTE # New Syntax # Trying to produce SQUOTE="'" ENVIRONMENT = "SQUOTE=""''''"""

22 nd Oct 2008Euro Condor Week 2008 Barcelona 15 So … Use the "new syntax"

22 nd Oct 2008Euro Condor Week 2008 Barcelona 16 Condor + Connectivity Firewalls are becoming more commonplace; as well as site firewalls, many machines will have their own firewalls. Many grid resources or local clusters will have their worker nodes on private networks which are not directly addressable. Is this a problem?

22 nd Oct 2008Euro Condor Week 2008 Barcelona 17 Execute Machine Submit Machine Submit Schedd Starter Job Shadow Startd Central Manager CollectorNegotiator Slide based on one from the University of Wisconsin-Madison

22 nd Oct 2008Euro Condor Week 2008 Barcelona 18 Condor + Firewalls All nodes in the pool must be able to "see" the central node(s) Every submit node must be able to "see" all execute nodes on which its jobs are expected to run and vice versa ["see" means accessible for the relevant condor fixed ports as well as the ephemeral ports (defined using HIGHPORT and LOWPORT) for TCP and UDP ]

22 nd Oct 2008Euro Condor Week 2008 Barcelona 19 Condor + Grids How is Flocking affected? Your jobs will not run on remote execute nodes which you cannot "see" or can't "see" you.  How about Glidein? Your jobs will not run on remote execute nodes which you cannot "see" or can't "see" you.  Grid Universe (incl. Condor-C) Your job is effectively handed over to that grid so firewalls are less of a problem. 

22 nd Oct 2008Euro Condor Week 2008 Barcelona 20 Firewall solutions 1.Run the whole pool inside a single (site?) firewall so no problems. 2.Use a restricted set of submit nodes (possibly with portal front-ends) and open resources for these. 3.Mirror the firewall settings for each machine in its ClassAds (exercise for the user – but see me for hints) 4.Use the Grid universe 5.Use the Generic Connection Broker – see Todd!

22 nd Oct 2008Euro Condor Week 2008 Barcelona 21 In Summary If you declare your own attributes – avoid the above Gotchas Use the "new" syntax for environment and argument values in submit files Be aware/wary of Firewalls and design your pool accodingly For further info on Firewall mirroring:

22 nd Oct 2008Euro Condor Week 2008 Barcelona 22 Whitespace, or not We have already seen that \ escapes a newline, even with whitespace after it. What about the comment char? How about: ARGUMENTS = one two \ three ARGUMENTS = one two \ # Above 2 Args are important \ three ARGUMENTS = A B # 2 Args ?