1. EGEE to EGI Transition Meeting: Middleware Repository
Kostas Koumantaros
Ioannis Liabotis
Marios Chatziangelou
Antonis Zissimos
Christos Kanellopoulos
EGEE-III INFSO-RI

2. EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010
Outline
EGI Software Repository
Requirements
Components
Contents
Infrastructure
Implementation
Use Cases
Discussion on software providers requirements
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

3. Provision of a software repository
Part of EGI-InSPIRE TSA2.4: Provision of a software repository and support tools
Requirements:
highly available source of software components available for inclusion in the UMD Release or for direct use by NGIs
Provide a repo
with real components (source and binary components located in the hardware provided for this activity)
with virtual components (links to source and binary components located in the software providers servers)
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

4. Components of the Software Repository
Unified Middleware Distribution (UMD Components)
Fully supported (via SLD) i.e. EMI provided components – UMD part of gLite, ARC, UNICORE
Community supported – actively used and supported by the community
Associated Components
Software that works well with UMD, but EGI does not provide support for it (i.e. current RESPECT software)
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

5. Contents of the Repository
Versioned binaries uploaded by software providers
Multiple supported platforms
RPM, deb, etc
Source Code uploaded by software providers if required
As RPMS, Source Deb, tarballs
Links to binaries or source code to external repositories
Allow for state assignment and transition
i.e. contributed, under evaluation, in staged rollout, in production, rejected, and deprecated
Support yum, apt, etc
Web front end – reference point for all end users of the repo
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

6. Operations on the repository
Download the software packages directly from the repository using yum/apt.
Sync repository contents using rsync.
Add/remove users
Add/remove groups
Add/remove software components
Add/Edit software component metadata
Create/delete yum/apt sub-repositories
Approve/reject software components
Digitally sign software packages (to be decided)

7. EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010
Infrastructure
Installed in 3 sites in the HellasGrid – GRNET NGI Infrastructure
ICCS, IASA, AUTH
Mirrored repositories
Good network connection to GEANT through GRNET
Monitoring of services via Nagios
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

8. Proposed Implementation and architecture (to be discussed)
Basic assumptions
Binary Repository (coordinated by IASA)
Web Front end (coordinated by ICCS)
Source code repository (coordinated by AUTH)
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

9. EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010
Assumptions
Enabling Grids for E-sciencE
Assumes
Software components are registered to a central registry
Each SW component has an owner and members combined in a user registry as a group
Owner has the option to request repository service for the SW component
Owner/members have IGTF certificates
EGEE-III INFSO-RI
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010 9 9

10. Repository Contents For End Users For Repository Administrators
Binary and Source RPMs (RPM, SRPM)
Delivery Methods: YUM, APT, http, rsync
Tarballs
Delivery Methods: http, rsync
Binary and Source DEBs
Delivery Methods: APT, http, rsync
For Repository Administrators
able to perform restricted operations on the contents of the repository, such as:
checks whether the software were downloaded correctly or not,
physically moves the data between the repositories (based on the metadata set and upon the state change as this is set by the Evaluators/Verifiers),
create YUM/APT sub-repositories (based on the metadata)
e.t.c.

11. Use case: Fully supported SW (i.e. EMI)
Software provider (submitter)
Metadata container
Inserts metadata
Upload Area
Moves data from the meta-URL to the UMD upload area
UM Verification Group
Scratch
Moves data to the /scratch repo
UMD repo team
Rejected
Moves data to the /stage-rollout repo
Stage RollOut
Moves data to the /production repo
Rejected
Production
Move
Depreciated
Rejected
Notify
Verify
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

12. Fully supported SW - State transitions
For the fully supported software the following states should be addressed and the corresponding transitions should be satisfied, for every release:

13. Use case: Community/Associate SW
Software provider (submitter)
Metadata container
Inserts metadata
Upload Area
Moves data from the meta-URL to the UMD upload area
UM Verification Group
Moves data to the /scratch repo
Scratch
UMD repo team
Rejected
Moves data to the /production repo
Production
Move
Depreciated
Rejected
Notify
Verify
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010 13

14. Community/Associate SW - State transitions
For the associate and the community software the following states should be addressed and the corresponding transitions should be also satisfied, for every release:

15. Use case: Downloads from end users
Downloading packages from the UMD can be achieved in three ways:
1.Using the repositories through yum/apt
2.Downloading the packages from the UMD portal.
3.rsync will be supported for repository mirroring
In the first case, the user adds the repository and certificates to its repository management system. This ensures timely updates and automates the procedure. Some users might find it handy to download certain packages by hand from the portal itself. Such an approach may be used by end users to revert their installation to a previous version or to get a particular deprecated version due to compatibility concerns.

16. Workflow Support State transitions support
A software release can be moved only forward, meaning:
Submitted -> NotVerified -> [ Verified-InStageRollout ] -> Verified-InProduction -> Depreciated
Regardless release’s current state, it could be moved into “Rejected” state by the Evaluators/Verifiers. The “Rejected” state it’s a permanent/final state and when the software provider “fixes” the problem, the software release has to be re-submitted into the UMD repo (a new revision number will be assigned to it).
YUM/APT related metadata provision
The corresponding metadata should be provided by the software provider for the creation of the YUM/APT related sub-repositories.
Do we need this software product to contain YUM and/or APT sub-repositories? (e.g Service nodes CE)
In case of YUM;
Do we get metapackage based YUM or group-based (group based is currently the case of glite-UI and glite-WN in glite-3.2)?
In case of metapackage-based YUM;
Are all the required metapackages contained in the submitted release?
Do we get a list of the identified metapackages?
In case of group-based YUM;
Is there corresponding list (possibly in an xml format – i.e. a comps.xml file) that contains all the indentified groups? Do we get it ?
In case of APT;
Did the software provider specified under which directory he wants the APT sub-repositories to be created?
Interaction with the tools that supports the verification process
There is a need for the UMD repo to access the state of each release as this is set by the Evaluators/Verifiers, thus an interface to the tool(s) that covers the verification process should be provided.

17. EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010
Technical Features
Repositories
Service monitored by the HellasGrid Nagios Monitoring system.
The repositories are replicated across the three HellasGrid Sites.
Package uploading area/repository
Users will be authenticated using their Grid certificates ??
Synchronize account information from VOMS ??
Command-line client interface
One master site and two sites in a failover-standby state, ready to take over in case of a “failure” of the master
Continuous replication from the master to the failover-standby sites/services
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

18. Web front end - General Description
The web frontend will be the reference point for all the end-users and the software providers
It will provide information about all the registered components in the repository
Common metadata for all the registered components
Each software provider will be responsible for registering their components and maintaining their metadata
Main input method through web
Explore the possibility of an API for automating the registration procedure, especially for the large software providers
Provide public search functionality to the metadata
Users will be authenticated using their Grid certificates
Authentication based on VOMS?
EGEE-III INFSO-RI
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

19. Metadata All Software Components Real Software Components
Virtual Software Components

20. All Software Components
Static (Rarely Change)
Software_Provider
Software_Component
Creator
Contact
Description
SLD_Type (fully supported, community supported, associated components)
License
Keywords
supported platforms
architecture
documentation links

21. Real Software Components
Dynamic (Change at every x.y. release)
Date
Version
Release_Notes_Url
Services_Affected {set}
Change_Log
Rsync_Url (host and path from which we pull the data)
Distribution_method {yum,apt,tar)
Dependencies
Service_Groups
Documentation_Url
category based on middleware functionality

22. Virtual Software Components
Dynamic (Change at every x.y. release)
Date
Version
Release_Notes_Url
Change_Log_Url
Repository_Url
(URL from where the software packages can be downloaded)
Documentation_Url

23. User Registry
Take advantage of existing authentication services available in EGI
User accounts should be able to be members in Groups and be assigned specific roles
One group for the people belonging to the Middleware Verification Unit
One group for the Repository Administrators
One group for each of the Software Providers (i.e. EMI, lcg-CA etc) that will be registered in the UMD repository
In order to ease the administration burden there should be the role of Group Admin for each group in order for the person who is assigned this roles to be able to perform Group administration tasks (e.g. adding or removing users) internal to the group.

24. User Registry Certificate based access EGI SSO access
Most of the people collaborating with EGI have or are able to acquire X509 certificates from an IGTF accredited certification authority.
All users requiring RW access to the repository will be members to a specific VO (e.g. umd-repo.vo.egi.eu) and will be members to the Group they belong.
EGI SSO access
CESNET will be offering an EGI wide SSO service.
All users requiring RW access to the repository will be members to at least the Group they belong. This means that the EGI SSO should allow users to be grouped into groups.
According to our current understanding the existing EGI SSO is not based on one of the commonly used Federation protocols, but instead it will be an LDAP holding all the user accounts. Is this true?

25. Users and Roles in the Repository
End Users:
read-only access to the Repository services
Able to browse all public web pages and the production and staged-rollout package repositories.
End users are not authenticated and considered as anonymous users to the system.
Software Providers:
should have RW access to specific parts of the UMD Repository Services.
Able to edit only information regarding the software they "own".
Access to the programmatic interface
Middleware Verification Unit:
should have RW access to specific parts of the UMD repository services.
They can edit the information on any software package
Repository Administrators:
full administrator access to the repository in order to perform the repository maintenance tasks.

26. EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010
User Registry
Enabling Grids for E-sciencE
1 Virtual Organization for software developers
1 VO Group per software provider
1 Owner per group
Members of the group will have read/write access to the repository
EGEE-III INFSO-RI
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

27. Monitoring
All the UMD Repository Services will be monitored by the HellasGrid Nagios monitoring infrastructure
Currently the following UMD Repository Service components have been identified:
The yum/apt/tarball repositories (HIGH CRITICALITY )
The web front end for the end users. (MEDIUM CRITICALITY)
The web front end and programmatic access interface for the software providers. (MEDIUM CRITICALITY)
The web front end from the Middleware Verification Unit (LOW CRITICALITY)
The backend database (HIGH CRITICALITY)

28. Monitoring
The package repositories, the web front ends and the databases will be replicated across three HellasGrid sites.
round robin (RR) DNS entries for package repositories and web front ends.
monitoring the availability of all the services and if any one fails it will be automatically removed from the RR.
Proven mechanism used for than 3 years on the HellasGrid (WMS/Top BDII HA)
In the case of the database, if the master instance experiences a failure, one of the slave instances will be promoted to master.

29. Thoughts: Package Signing
Both RPM and DEB package formats support digital signatures using GnuPG .
The reason for signing a package is to provide authentication.
Both DEB & RPM package formats support more than one signature per package. This way we can provide a means of documenting the path of ownership from the package builder to the end-user.

30. Thoughts: Package Signing
Private Key protection
The private key used for the digital signing should be properly protected and not reside on a publicly accessible system.
Life time of the key
What happens to the packages when the key that have been signed by expires?
Key distribution
In order for the users to be able to verify the digital signatures, it is necessary that the public key is distributed before hand to them in a secure way
Software Packages
the package contents change in order to include also the signature(s). we will have to keep both the original package as it was uploaded by the software provider and the final signed package? Recalculate checksums for all signed packages

31. EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010
Other Thoughts
Enabling Grids for E-sciencE
Possibility to have read-only replica (hidden from the public) at CESNET if needed (e.g. for TRAC or similar tool)
Possibility to host repository mirrors if groups decide to have their own private source code repositories, but they want also to have their code available through the EGI repositories
What is the structure of an EMI release?
Is the a possible one?
It should be a structured set of files that contains all previous updates + the newly added updates and it will be managed and finally populated by the UMD repo “as it is” (only yum/apt related modifications will be allowed to the product). Correct?
EGEE-III INFSO-RI
EGEE to EGI Transition Meeting – Amsterdam, 4 March 2010

32. More Thoughts …
Is the term “UMD release” correct? If yes, please specify what is a UMD release?
How shall we handle the middleware releases in the first few months (after May 1st)?
Do we have to assemble the software we distribute? If yes, please clarify, in which level?
The state transition of the components will be the responsibility of the verifiers (not of the admins of the repo). Correct?
The versioning schema that will be followed, it will be in the format of “project-X-Y-Z”, X:Major, Y:Minor, Z:Revision (i.e. emi-3.2.8). Correct?