Spring Cleaning the Software Repositories Matthias Schröder Topics: Introduction The Issue What can be done Conclusion 27-March-2008 Spring Cleaning Repositories

Spring Cleaning Repositories Introduction Software Repositories hold all packages used on any quattor managed node Organised by ARCH_OS We keep all versions of packages ever put into the repositories To be sure that we can reinstall a machine if needed Number of packages grows with time… ARCH_OS i386_slc3 x86_64_slc3 i386_slc4 x86_64_slc4 Number of packages 22271 5389 15389 16369 27-March-2008 Spring Cleaning Repositories

Spring Cleaning Repositories The Issue The number of packages has reached a critical level Going through the listings to find packages takes longer and longer Panc compile times directly effected Impossible to upload the repository listings to CDBs with little memory Ok, 1GB is not a lot, but still… Impossible to compile templates for enclosures with many nodes Running out of memory even on CDBs with lots of installed memory… We have to clean up the repositories now! 27-March-2008 Spring Cleaning Repositories

Possible Ways to Clean Up Prepare a positive list of most frequent packages, keep all others List of versions to keep is difficult to obtain We can’t be sure that we will keep all versions required …andwe can not count on panc to warn us We might only discover that packages are missing when we have to reinstall a node Prepare a list of packages to remove Might be difficult to obtain for all packages Same risk as above Tedious We probably keep many versions we don’t use any more => too little gain Make list of versions still in use from xml profile of all nodes Safest But also have to keep newer versions not yet in use Highest gain Needs a new tool to extract list of packages and versions from xml profiles 27-March-2008 Spring Cleaning Repositories

Spring Cleaning Repositories Tests Done With a positive list for castor and CA’s for i386_slc3 But castor list now known to be too small… Results very encouraging Have to keep more castor versions Can gain more on other packages Full repo listing Svelte repo listing # packages 22271 14868 Compile time 54 s 35 s Memory usage 3.8 GB 1.7 GB 27-March-2008 Spring Cleaning Repositories

Spring Cleaning Repositories How to Continue We have to be very careful with removal of versions We have to reduce the repo listings now Some profiles do not compile any more! My proposal: two step approach Step 1: Remove known obsolete castor and CA versions, keep all the rest These are the packages for which the biggest gain can be achieved with little work Step 2: Make a listing of all used versions from XML profiles Maximum gain, safe method, but takes longer to prepare Have to rethink the ways how we handle kernels (i386_slc3: 1122 “kernel” packages!) Only keep these kernels that are configured on nodes? 27-March-2008 Spring Cleaning Repositories

What to do With Obsolete Versions We should not delete them Disk space is not an issue Copy them to another place on SWREP Then remove them via swrep-soap-client 27-March-2008 Spring Cleaning Repositories

Spring Cleaning Repositories Conclusion We have to reduce the size of the repository listings This is a delicate operation, we have to ensure that all used versions are kept Can we agree on two step approach? 27-March-2008 Spring Cleaning Repositories