Tutorial: Condor on Windows

Overview Simple job submission Accessing network shares Handling DLL dependencies Checkpointing in Vanilla universe Submitting interpreted scripts

Overview Simple job submission Accessing network shares Handling DLL dependencies Checkpointing in Vanilla universe Submitting interpreted scripts

Simple Job Submission

condor_store_cred Contacts local SchedD and asks it to securely store a user’s password Password is placed encrypted in a registry location myp4sswd y0urs C:\>condor_store_cred add Account: gquinn@CROW Enter password: Operation succeeded.

Job Execution: Submit Side SchedD Secure Password Store Submit myp4sswd y0urs Submit Shadow

Job Execution: Execute Side Starter Jobs run using a Condor-specific account with minimal privileges. Job condor-reuse-slot1

Overview Simple job submission Accessing network shares Handling DLL dependencies Checkpointing in Vanilla universe Submitting interpreted scripts

Accessing Files on a Network Share The condor-reuse-slotX accounts are local accounts without network privileges To access files on a network share, the job needs to run using an account that has access to the shared network resources Using a CredD, its possible to have jobs run with the credentials of the submitting user

The Windows CredD A centralized repository for user passwords CredD myp4sswd y0urs “store password” C:\>condor_store_cred add Account: gquinn@CROW Enter password: Operation succeeded. CredD <password>

The Windows CredD SchedD Shadow myp4sswd “fetch password” y0urs <password> Shadow Submit machines can use the CredD to impersonate the user in the shadow

The Windows CredD Starter Job “fetch password” myp4sswd y0urs <password> Job Execute machines can use the CredD to run jobs as the desired user

Securing the CredD NTSSPI can be used to authenticate to CredD and send the password encrypted over the network myp4sswd y0urs C:\>condor_store_cred add Account: gquinn@CROW Enter password: Operation succeeded. CredD “store password”

NTSSPI Authentication Uses Windows built-in authentication capability Used for user-to-daemon authentication Always works when client and server are on the same machine Can also be used over the network if the user account exists on both the client and server and the passwords match

Securing the CredD Starter Job “fetch password” myp4sswd y0urs Job Condor normally runs as SYSTEM, and therefore can’t use NTSSPI

Securing the CredD Options for securing password fetch operations Kerberos SSL Password

Password Authentication Daemon-to-daemon communication is secured using a shared secret (the “pool password”) Pool password must be distributed to all machines in the pool before enabling Pool password is stored using the “-c” option to condor_store_cred Password can be set remotely using “-n” option (make sure connection is authenticated!)

Setting the Pool Password Enable CONFIG access from the machine that you will run condor_store_cred: If setting the password remotely, ensure the user account that runs the condor_store_cred command exists on every machine and has a common password Set the pool password on every machine: HOSTALLOW_CONFIG = $(HOSTALLOW_ADMINISTRATOR) C:\> condor_store_cred add –c –n node01.cs.wisc.edu

Configuring the CredD Host Once pool password has been distributed, the CredD can be configured: # Set security settings so that full security to the credd is required CREDD.SEC_DEFAULT_AUTHENTICATION = REQUIRED CREDD.SEC_DEFAULT_ENCRYPTION = REQUIRED CREDD.SEC_DEFAULT_INTEGRITY = REQUIRED CREDD.SEC_DEFAULT_NEGOTIATION = REQUIRED # Require NTSSPI auth for password storing CREDD.SEC_WRITE_AUTHENTICATION_METHODS = NTSSPI # Require PASSWORD auth for password fetching CREDD.SEC_DAEMON_AUTHENTICATION_METHODS = PASSWORD

Configuring the CredD Host Use the condor_config.local.credd example file: Include this file as one of the local configuration files: C:\Condor> copy etc\condor_config.local.credd .\ LOCAL_CONFIG_FILE = $(LOCAL_DIR)/condor_config.local \ $(LOCAL_DIR)/condor_config.local.credd

Running Jobs as Submitting User Submit and execute machines need to be configured to use the CredD: CREDD_HOST = vault.cs.wisc.edu STARTER_ALLOW_RUNAS_OWNER = True SEC_CLIENT_AUTHENTICATION_METHODS = \ NTSSPI, PASSWORD CREDD_CACHE_LOCALLY = True

Running Jobs as Submitting User Example submit file: universe = vanilla executable = whoami.exe log = whoami.log output = whoami.out run_as_owner = true queue

Overview Simple job submission Accessing network shares Handling DLL dependencies Checkpointing in Vanilla universe Submitting interpreted scripts

Jobs with DLL Dependencies

Jobs with DLL Dependencies Two ways to deal with DLL dependencies: Ship the DLLs to the execute machine with the job Make sure the job runs on an execute machines with the needed DLLs, and knows where to find them

Shipping DLLs with the Job universe = vanilla initial_dir = c:\cygwin executable = sleep.exe log = sleep.log transfer_input_files = cygwin1.dll,cygintl-8.dll,cygiconv-2.dll queue

Using DLLs on the Execute Side Windows searches directories in the PATH environment variable for needed DLLs C:\> set PATH=%PATH%;c:\cygwin\bin universe = vanilla executable = c:\cygwin\sleep.exe log = sleep.log getenv = true queue

Overview Simple job submission Accessing network shares Handling DLL dependencies Checkpointing in Vanilla universe Submitting interpreted scripts

Checkpointing in Vanilla Universe No Standard universe on Windows Some jobs can do their own checkpointing Condor will “warn” your job before eviction by sending the WM_CLOSE window message For console applications, this translates to a CTRL_CLOSE_EVENT

Checkpointing in Vanilla Universe Catching a “eviction notice” in a C-based console app: Need to let Condor know that partial results are valuable in the submit file: SetConsoleCtrlHandler(ckpt_function, TRUE) when_to_transfer_output = ON_EXIT_OR_EVICT

Overview Simple job submission Accessing network shares Handling DLL dependencies Checkpointing in Vanilla universe Submitting interpreted scripts

Submitting Scripts Can just submit the interpreter as the executable and the script as an input file executable = c:\perl\bin\perl.exe arguments = script.pl args for script transfer_executable = false

Submitting Scripts If UNIX-style “#!” behavior is desired, use a USER_JOB_WRAPPER Useful for using a single submit file that works on all platforms executable = script.pl arguments = args for script

USER_JOB_WRAPPER # open the actual executable so we can look for interpreters # if (open(EXECUTABLE, $ARGV[0])) { # see if the first two bytes are #! my $two_bytes; if ((read(EXECUTABLE, $two_bytes, 2) == 2) && ($two_bytes eq "#!")) { # look for the /usr/bin/perl interpreter if (<EXECUTABLE> =~ /\/usr\/bin\/perl/) { unshift(@ARGV, "c:\\cygwin\\bin\\perl.exe"); } system(@ARGV);

Questions? Windows BOF: Thursday, 11:00 – 12:00 Room 221