Software repository replication using the ASIS Local Copy Manager IT/DIS/OSE, CERN ASIS Team Presented by: German Cancio

Presentation Plan u Introduction to ASIS u ASIS structure u Transactional model u Repository replication using ASISlcm u Project status

Software Distribution: Goals and Requirements u Centralized management of SW applications necessary for: u standard and homogeneous user environment u better support (less diversity!) u Equally accessible by all supported platforms u same availability u same configuration u User nodes to see ‘ready to use SW’ trough a distributed file system

Software Distribution at CERN: ASIS u ASIS: stands for Application Software Installation Service, offering: u A multiplatform and multiversion SW repository with ready-to- use applications u A toolsuite for u assisted application compilation and configuration u accessing, manipulating and replicating the repository u Contents: > 700 packages for ~8 platforms u GNU software (compilers, system tools, editors, desktops) u LaTeX u X11 and contributions u CERN program library u Others, like Perl, Tcl/Tk, Java, licensed software

Platforms currently supported ASIS (AFS) name Platform i386_redhat51 RedHat Linux 5.1 sun4x_56 Sun Solaris 2.6 hp_ux102 HP_UX ACE3 alpha_dux40 Digital UNIX 4.0b rs_aix41 IBM AIX rs_aix43 IBM AIX sgi_64 SGI IRIX 6.5

Software Distribution at CERN: ASIS (II) u Main tools: u ASIShappi: concurrent software compilation and configuration on multiple platforms u ASISwsm: client workstation access to the repository u ASIStm: checks and validates repository modification requests u ASISlcm: repository update and replication u ASIStar: generate tarfiles from ASIS packages

Logical organization u Packages are classified into Families, Products and Versions u A family groups together products with similar origin or functionality (i.e. MAIL, KDE, GNU.EDIT,CERN.LIB) u A product may contain one or several versions (emacs-19.34, emacs-20.2) u Each version is usually available for all supported platforms u ‘share’ platform: contains common files (documentation, fonts, scripts)

Logical organization (II) u A model has been defined for the software processing from the sources release until delivery to users. u Based on a state-transition diagram, 2 phases: generation and deployment u A deployed package may be on one of the following states: u In Production: default version, formally supported u Certified: proved to work in the proper environment u Under Certification: for evaluation by beta users u ASIS stores InProduction packages in a common area for improved efficiency

Client access to the repository u ASISwsm manages the links from the repository to the workstation (in /usr/local or /cern) u The workstation administrator can customize the access to the packages (i.e. default version, specific, ignore) u Packages can be stored as ‘local copies’ ASIS  ASISwsm ASIS repository

Transaction Model In order to group together related repository modifications, a transactional system is used u A transaction is a sequence of operations... … either u a modification of the state of a package u a ‘patch’ operation (modifying a package) u The atomicity of the execution of the transaction is guaranteed.

Transaction: example Description: update emacs & related packages RemoveFromProduction GNU.EDIT/emacs (sun4x_55,alpha_osf32,hp_ux102) RemoveFromProduction GNU.EDIT/vm-5.96beta (share) IntroduceInProduction GNU.EDIT/emacs-20.2 (sun4x_55,alpha_osf32,hp_ux102,alpha_dux40) IntroduceInProduction GNU.EDIT/vm-6.32 (share)

Transaction System Based on the cooperation between ASIStm (ASIS transaction manager) and ASISlcm (ASIS local copy manager) u A transaction is submitted to ASIStm, which checks its validity and coherence. þ The transaction is queued if accepted. u Repository modifications are carried out by ASISlcm

ASISlcm: Repository Replication ASISlcm... è Reads transactions which are new since last run è Determines what has to be done according to local configuration è Performs locally all selected transactions. u Only new packages are copied u ASISlcm can be run asynchronously, (i.e. at nighttime)

ASISlcm (II) u Fault tolerance: ASISlcm rolls back the currently executed transaction in case of failure (i.e. when network problems arise) þ Transaction atomicity is always guaranteed. u Star scheme: modifications are propagated to the replicas (master/slave) u ASISlcm runs locally on the replicated site and only accesses the master when necessary u Replication costs proportional to the size of changes

Replication: example ASIS ASISlcm CERN AFS master repository ASIS ASISlcm ASIStm CERN NFS replica Replica 1 on site XXX Replica 2 on site XXX WAN file system (AFS) LAN file system (NFS) LAN file system (NFS)

Full repository replication ASISlcm also offers a ‘full’ replication mode u State of all packages is checked against the master and updated accordingly u When to use one mode or the other? Transactional (or incremental) replication: u frequent repository updates, i.e local replicas for load balancing Full replication: u initial repository cloning u independent replicas (occasional updates) u recovery after crashes u the full replication does not offer atomic properties

ASISlcm: configuration ASISlcm can be configured to match your site’s needs: u Filtering by architecture u list of platforms to be replicated u Filtering by family / product: u take all product versions u skip a family / product / version u ‘freeze’ a family / product / version u based on the product’s state (Certified, In Production, Under Certification)

ASISlcm: configuration example  Architectures: sun4c_411 sun4x_55 hp700_ux90 hp_ux102 rs_aix41 alpha_osf32 sgi_52 i386_linux2 Products:  / InProduction # Lightweight copy GNU.LANG/ None GNU.LANG/cvs I nProduction GNU.LANG/rcs All GNU.MISC/ Frozen GNU.MISC/gzip InProduction MISC/ None MISC/a2ps InProduction MISC/tcsh 6.06 # Specific version nr MISC/arc Frozen

How to set up an ASIS local copy? 1st. Get the ASIS software u by mirroring from the CERN AFS tree or u by downloading it from our ftp server 2nd. Configure and run ASISlcm (full) on your server(s) 3rd. Configure and run ASISwsm on your workstations 4th. Decide frequency and update type (full, incremental) Please contact us for assistance!

ASIS project status u ASIS runs at CERN as part of the standard setup: u > 2000 nodes, > Users u ~ 700 packages on ~ 20 GB u ASISlcm: u in production for the CERN NFS server for 18 months u on other sites: ??? u EPFL (Swiss Federal Institute of Technology, Lausanne): acts as co-developer and owns a private repository

Further documentation and contacts ASIS has an URL on the web: with links to: u Guides (not up to date…) u Product Documentation u Site Usage Statistics u Download area Contacts: ASIS team at CERN: