WP3 R-GMA: Likely status New Years Eve Steve Fisher / RAL 24/2/2003

WP3 Steve Fisher/RAL - 24/2/2003R-GMA2 Outline Where we are “now” – J Day New things for J+27 Likely status by end of EDG project What we probably will not do by Dec 31

WP3 Steve Fisher/RAL - 24/2/2003R-GMA3 Current Producers (J) DataBaseProducer –Relatively slow –Information not lost –Clean up strategy needed (will be there for J+27) –No streaming (though could be defined in principle) –Supports joins CircularBufferProducer –Fast –Uses an SQL parser – no RDBMs involved –Holds data in memory –Does not support joins –Deprecated for J+27

WP3 Steve Fisher/RAL - 24/2/2003R-GMA4 Displays (J) Pulse –A simple Java client –Could be improved a bit Command Line –Looks rather like MySQL command line –Interactive or one command and quit –It could be also be improved a bit – though nothing planned for J+27 BrowserServlet –JSP application –some fixed common queries –or compose your own –Ergonomics and functionality could be improved – though nothing planned for J+27

WP3 Steve Fisher/RAL - 24/2/2003R-GMA5 Producer Inheritance – J+27 This is not visible to the user DataBaseProducer Cleanable Insertable Declarable APIBase Concrete Class (an example – some classes inherit from lower down) Supports clean up mechanism Allows rows to be inserted Allows tables to be declared Methods shared by all our APIs - e.g. disconnect()

WP3 Steve Fisher/RAL - 24/2/2003R-GMA6 API Features: APIBase disconnect/ reconnect() –For APIs that are used infrequently – their machines can now be switched off! setTerminationInterval() and showSignOfLife() –The API must send heart-beats to its servlet in order to stay registered (GRRP-like protocols). setTupleChecking()

WP3 Steve Fisher/RAL - 24/2/2003R-GMA7 API Features: Declarable, Insertable, Cleanable Declarable: declare/ undeclareTable() –Declarables are publishers that register their views. Insertable: insert() –Now a vector of tuples may be inserted at a go: if the method returns, the servlet received them safely. Cleanable: declareTable(…, cleanUpPedicate, cleanUpInterval) –Cleanables store tuples locally using a database – –The servlet starts a thread to clean the database periodically according to the policy.

WP3 Steve Fisher/RAL - 24/2/2003R-GMA8 API Features: Consumers Three query types are supported: history, continuous and “latest snapshot” –API makes it explicit

WP3 Steve Fisher/RAL - 24/2/2003R-GMA9 Producers for J+27 StreamProducer –Intended to replace the CircularBufferProducer –Can define minimum retention period ResilientProducer –Like the StreamProducer but won’t lose data if system crashes –So slightly slower LatestProducer –Just holds the latest information for any “primaryish” key –Supports joins CanonicalProducer –Offers anything as relations

WP3 Steve Fisher/RAL - 24/2/2003R-GMA10 ResilientStreamProducer The servlet keeps a log of changes made to a producer’s state (by serializing a Command object). Periodically snapshots are taken, and the InstanceTracker (a hash map of producers kept by the servlet) is serialized and stored on disk. Recovering from Failure: –When the servlet restarts, the last snapshot of the InstanceTracker is retrieved, and state changes re-applied. –Then the registry is consulted, and any producers that had timed-out are re-registered.

WP3 Steve Fisher/RAL - 24/2/2003R-GMA11 LatestProducer Supports “latest snapshot” queries –offers up-to-date values for each primary key (previously, R-GMA tables had no primary keys). Implementation –When declareTable() is called, the servlet creates a new mysql database containing that table. –When a tuple is received, the servlet first tries to update the table. If this fails, the tuple is inserted. –Snapshot queries are simply passed on to the database.

WP3 Steve Fisher/RAL - 24/2/2003R-GMA12 Canonical Producer (J+27) Allows user defined code to be invoked to respond to SQL query Developed in collaboration with CrossGrid CP API User Code Canonical Producer Servlet Files CreateTable, Port, Protocol, Security, SQL Support, Multiple Query Support Security Insert Query Port Register

WP3 Steve Fisher/RAL - 24/2/2003R-GMA13 Archiver (Re-publisher) It is a combined Consumer-Producer You just have to tell it what to collect and it does so on your behalf For J Day re-publishes to a DataBaseProducer For J+27 will re-publish to any kind of “Insertable”

WP3 Steve Fisher/RAL - 24/2/2003R-GMA14 Use of components (J+27) BS Archiver of O-Z StreamProducer Archiver of I-N StreamProducer Archiver of A-H StreamProducer Smith (Wants to be told of each change of state of his job) Fitzwilliam (Wants to look at current state of all his jobs) Archiver of A-H LatestProducer Archiver of I-N LatestProducer Archiver of O-Z LatestProducer Consumer Each Bookkeeping Server publishes to a StreamProducer Archiver has a where clause to collect jobs belonging to a subset of users Most queries will be satisfied by one Archiver

WP3 Steve Fisher/RAL - 24/2/2003R-GMA15 A user application: CMS BOSS for job tracking on local farm –It currently forks the executable and parses stdout to publish info directly to an SQL DB –They publish to one table per job type and one table which is common to all job types They can now publish via R-GMA instead –Providing a scaleable Grid solution –Included in EU review demo –Works with J components but they want J+27

WP3 Steve Fisher/RAL - 24/2/2003R-GMA16 Programme till Dec 31 Functionality –Not much to do here Resilience and Scalability –Significant work to keep everyone happy Security –Essential for some areas of use Performance Optimisation –Nothing done here yet OGSA Compliant Implementation –Essential for long life

WP3 Steve Fisher/RAL - 24/2/2003R-GMA17 Functionality - Nagios Looking to include Nagios as a presentation tool. Will write a Nagios plug-in to instantiate an Archiver and use that information to populate Nagios displays Can also benefit from Nagios alert mechanism Will have different configurations for Site, Country, whole Grid etc. Expect to see this completed by Dec 31

WP3 Steve Fisher/RAL - 24/2/2003R-GMA18 Functionality - mediator Queries posed against a virtual data base The Mediator must: –find the right Producers –combine information from them Can now merge information from several producers The final mediator will take “any” SQL statement and do the right thing –How far can we get by end of project?

WP3 Steve Fisher/RAL - 24/2/2003R-GMA19 Resilience - Registry Will have one logical registry and schema per VO Each logical registry will have multiple physical “copies” Each entry in registry has 3 possible states Transmit new records and deleted records and checksum after records deleted locally Self healing even supports new registry instances Consumer uses any instance Fail over mechanism not yet implemented Schema more tricky Producer1 Producer2 Registry2 Info mastered by Registry2 Copy of info from Registry1 Copy of info from Registry3 Registry3 Info mastered by Registry3 Copy of info from Registry1 Copy of info from Registry2 Registry1 Info mastered by Registry1 Copy of info from Registry2 Copy of info from Registry3

WP3 Steve Fisher/RAL - 24/2/2003R-GMA20 Scalability testing We are trying many dummy CEs and SEs scattered around Europe along with our set- up for the EU review demo. Will use this to ensure that we can support a realistic number of components.

WP3 Steve Fisher/RAL - 24/2/2003R-GMA21 Resilience Testing Taking 7 components –Schema –2 registry instances –Producer API –Consumer API –Producer Servlet with other APIs –Consumer Servlet with other APIs Consider each component in turn –Break the network and bring it back –Close the component down and bring it back –Crash the component and bring it back Will also consider real life scenarii

WP3 Steve Fisher/RAL - 24/2/2003R-GMA22 Security Adding edg-security for authentication –Gives secure socket factory for https –Should be in for J+27 Plan to use VOMS Need to be independent of authentication implementation

WP3 Steve Fisher/RAL - 24/2/2003R-GMA23 Performance By design: –Very flexible - to avoid bottlenecks –Powerful queries allow a single query to be made We need to use the right tools to measure and improve performance –NetLogger to get an idea of time delays through system –Java profiling tools to understand the details

WP3 Steve Fisher/RAL - 24/2/2003R-GMA24 OGSIfication Have recently started the migration to web and grid services –Apache axis –WSDL generated APIs –Will provide a wrapper for backwards compatibility

WP3 Steve Fisher/RAL - 24/2/2003R-GMA25 All Grid Services OGSA Factories, GSH, GSR Registry includes HandleMapper SQL as Service Data Element Query Language Consumer Factory Producer Instance OGSIfied R-GMA Sensor Producer API Application Consumer API Schema Registry Consumer Instance Producer Factory

WP3 Steve Fisher/RAL - 24/2/2003R-GMA26 OGSIfication issues Consider XML as internal representation of service data elements –Depends on other developments Consider Xquery as service data elements query language –Depends on how Xquery develops X-GMA ??

WP3 Steve Fisher/RAL - 24/2/2003R-GMA27 Programme till Dec 31 Functionality –Enhanced Mediator –Nagios Integration Resilience and Scalability –Consider all failure modes –Complete Registry and Schema replication Security –Authorisation Performance Optimisation OGSA Compliant Implementation –GT3 inter-operability –Wrappers for backwards compatibility

WP3 Steve Fisher/RAL - 24/2/2003R-GMA28 What we don’t expect to do/complete Port to other flavours of Unix Port to Windows Complete the mediator to cope with any SQL query –This is a BIG job Complete the OGSIfication –OGSA will be changing under our feet Pluggable authentication Complete authorisation scheme