Glidein Factory Operations glideinWMS training Glidein Factory Operations i.e. How we spend our time? by Igor Sfiligoi (UCSD) glideinWMS training G.Factroy Operations

G. Factory Operation Categories Factory node operations Serving VO Frontend Admin requests Keeping up with changes in the Grid Debugging Grid problems glideinWMS training G.Factroy Operations

G. Factory Operation Ongoing Costs Factory node operations Pretty much runs itself, unexpected <1day/month Serving VO Frontend Admin requests Highly variable, average a few hours/week Keeping up with changes in the Grid Variable, currently O(10 hours)/week Debugging Grid problems More than we have effort for! Better tools could drastically reduce this glideinWMS training G.Factroy Operations

Factory node ops O(hours) / month The factory mostly just runs Occasional upgrade of SW needed, but typically fast and painless Most effort going into investigating unexpected behavior, e.g. High load Weird problems after a reboot/OS upgrade Of course, installing a new node can take significant time But a very rare event O(hours) / month glideinWMS training G.Factroy Operations

VO FE Admin requests O(hours) / week Adding a new VO FE can be expensive Apart from config changes, to help them start running However, relatively rare to have new VOs In steady state, VOs may request New sites New attributes g.Factory operators also must assist with debugging FE config changes Error logs come only to GF (currently) O(hours) / week glideinWMS training G.Factroy Operations

Following changes in the Grid G.Factory operational principle is trust-but-verify G.Factory admins must approve any change in the G.Factory config Grid a very dynamic place At least one site makes a change every single day Mostly complaint driven, have no good tools to automate change discovery G.Factory admins thus must change the G.Factory config often Currently mostly a manual process Better tools would be welcome O(10 hours) / week glideinWMS training G.Factroy Operations

Grid debugging 1/2 With O(50k) glideins running at any time, we always find something broken somewhere Full spectrum of errors Broken worker nodes (validation errors) Broken CEs (authentication/startup/monitor errors) Network problems (glideins not registering) Mostly cannot directly solve the problem(s) i.e. have to notify remote Admins But we have to discover the root cause to get it solved glideinWMS training G.Factroy Operations

Many FTEs DC, if we had them Grid debugging 2/2 Grid a difficult place to debug Most sites are black boxes for us Luckily, glideins provide lots of info in the logs When we get them... a broken site may not return anything useful, or anything at all Prodding the black box often needed Which is hard! And some problems may be VO specific, too Many FTEs DC, if we had them glideinWMS training G.Factroy Operations

What else we do? In order to make our life easier, we also Host a test glideinWMS instance Develop new helper tools The test glideinWMS instance allows us to discover problems early, thus both Increasing user satisfaction Reducing the time needed in debugging errors We create helper tools to suit our needs And anything major we contribute back to glideinWMS glideinWMS training G.Factroy Operations

The test glideinWMS Instance The test glideinWMS instance contains both a G.Factory and a VO Frontend This allows us end-to-end testing Major focus on the G.Factory, to test before deploying in production New SW releases New sites New services on existing sites glideinWMS training G.Factroy Operations

Summary Operating a G.Factory is much more than keeping the G.Factory service alive Indeed, this part takes almost a negligible amount of time Most effort going into debugging Grid-related problems At O(50k) CPUs, something is always broken somewhere Finally, providing expertise to help VO FE Admins also an essential part of the job glideinWMS training G.Factroy Operations

Acknowledgments This document was sponsored by grants from the US NSF and US DOE, and by the UC system glideinWMS training G.Factroy Operations