Moving CHTC from RHEL 6 to RHEL 7 Greg Thain HTCondor Week 2017

Migration without Migraines

Actually from SL  CentOS, not RHEL Need ecryptfs for encrypted sandbox (Actually CentOSplus…) executable = calculate.exe encrypt_execute_directory = true queue

“EL”

Today’s CHTC Users OS %age of CHTC users Require EL 7 5% Either EL 6 or 7 90% ???? Require EL 6 5% ????

Who are EL 7 only users?

Who are EL 6 only users? Standard universe jobs that have started

Tomorrow’s CHTC Users OS %age of CHTC users Require EL 7 Was 5%, going ↑ Either EL 6 or 7 Was 90%, going ? Require EL 6 Was 5%, going ↓

Observation OS %age of CHTC users Require EL 7 Was 5%, going ↑ Either EL 6 or 7 Was 90%, going ? Require EL 6 Was 5%, going ↓

Moving CHTC machines -> RHEL 7

Moving CHTC machines -> CentOS7

Easy – not the focus of this talk Moving CHTC machines -> CentOS7 Easy – not the focus of this talk

Moving CHTC machines -> CentOS7 jobs

Moving CHTC jobs -> CentOS7 users

I know, I’ll deploy containers! Containers as an Infrastructure, or as jobs? CaaI or CaJ? Containers as job require user work to setup Tricky for GPU jobs GOAL: minimize user work!

The Time is now! CHTC formed a team in Nov 2016 Condor + Ops + RCF + OSG Controlled transition with minimum user pain

Plan on a slide Before: Every job gets EL 6 Phase 1: Flag Day! Roll out some # of EL 7 machines, but Default to EL6, users can opt into EL7 or BOTH Communicate to EVERY user Flag Day! Default to EL7, users can opt into EL6 or BOTH

CHTC Pool CHTC execute machines jobs CHTC submit machines foreign execute machines (flocking, OSG glidein, etc.) Foreign submit machines

CHTC Pool CHTC execute machines Want all configuration changes in our Schedds – it’s the point we control the most! jobs CHTC submit machines foreign execute machines (flocking, OSG glidein, etc.) Foreign submit machines

Attributes: Useful and Otherwise Find the EL attribute? How to force a job to land here? OpSys = "LINUX" OpSysAndVer = "CentOS7" OpSysLongName = "CentOS Linux release 7.3.1611 (Core)" OpSysMajorVer = 7 OpSysName = "CentOS" OpSysShortName = "CentOS" OpSysVer = 703

Forcing a job to EL 7 Requirements = OpSysMajorVer == 7 Submit file OpSys = "LINUX" OpSysAndVer = "CentOS7" OpSysLongName = "CentOS Linux release 7.3.1611 (Core)" OpSysMajorVer = 7 OpSysName = "CentOS" OpSysShortName = "CentOS" OpSysVer = 703 Requirements = OpSysMajorVer == 7 Submit file Works with no startd changes – existing glideins, etc.!

Forcing a job to either Requirements = (OpSysMajorVer == 6) OpSys = "LINUX" OpSysAndVer = "CentOS7" OpSysLongName = "CentOS Linux release 7.3.1611 (Core)" OpSysMajorVer = 7 OpSysName = "CentOS" OpSysShortName = "CentOS" OpSysVer = 703 Requirements = (OpSysMajorVer == 6) || (OpSysMajorVer == 7) Submit file

Pop Quiz: What about default? OpSys = "LINUX" OpSysAndVer = "CentOS7" OpSysLongName = "CentOS Linux release 7.3.1611 (Core)" OpSysMajorVer = 7 OpSysName = "CentOS" OpSysShortName = "CentOS" OpSysVer = 703 Requirements = nothing said Submit file

Hint: APPEND_REQUIREMENTS won’t work OpSys = "LINUX" OpSysAndVer = "CentOS7" OpSysLongName = "CentOS Linux release 7.3.1611 (Core)" OpSysMajorVer = 7 OpSysName = "CentOS" OpSysShortName = "CentOS" OpSysVer = 703 APPEND_REQUIREMENTS is unconditional!

Schedd xforms to the Rescue! # https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=OsMigrationHints JOB_TRANSFORM_NAMES = EL, EL_VER JOB_TRANSFORM_EL @= end REQUIREMENTS JobUniverse == 5 && Regexp("OpSysMajorVer",UnParse(Requirements),"i") =?= false && Regexp("\"WINDOWS\"", UnParse(Requirements)) =?= false SET Requirements (Target.OpSysMajorVer == 6) && $(MY.Requirements) @end JOB_TRANSFORM_EL_VER @= end [ eval_set_WantELVer = isError(int(Regexps("OpSysMajorVer[[:space:]]*=[?]?=[[:space:]]*([0-9])", UnParse(Requirements), "\\\\1", "i"))) ? 0 : int(Regexps("OpSysMajorVer[[:space:]]*=[?]?=[[:space:]]*([0-9])", UnParse(Requirements), "\\\\1", "i")); ]

Regexp("OpSysMajorVer",UnParse(Requirements),"i")

Engineering vs. Science Requirements = blah blah blah Requirements = (Target.OpSysMajorVer == 6) && blah blah blah

Engineering vs. Science Requirements = OpSysMajorVer == 7 && blah Requirements = OpSysMajorVer == 7 && blah

Current Status ~ 10 % of CHTC pool running CentOS 7 Some users invited to try CentOS 7 Every one successfully running EL 6 binaries on CentOS 7 with no changes! No transform surprises!

Future work What about OSG? Keep working with users How to ask for EL6 vs EL7? Keep working with users

Summary Users need RHEL 7 sooner than you may think With some work, transition may be less painful for them than you may think

Avoid User Migraines, not ours!