DDM Central Catalogs and Central Database Pedro Salgado

Central Catalogs 0.2 Central Catalogs 0.3 Plans

Central Catalogs 0.2

Separation of location and subscription catalog MySQL implementations of repository, location and subscription catalog User-input validation Improved database transaction handling Improved communication between client and server pure python output usage of the HTTP status codes Testcases introduction of tests to validate the system Improved user support and documentation mailing-lists, tutorials and installation instructions and savannah.

Central Catalogs 0.2 Code reorganization (CVS) common components per catalog components

Central Catalogs 0.2

Central Catalogs 0.3

New content catalog implementation (MySQL/Oracle) New location, repository and subscription catalog in Oracle migration scripts 0.2 => 0.3 already done (MySQL only) Some changes to subscription catalog to cope with new site services requirements for 0.3 Communication between clients and catalogs moved to HTTP/RPC (single URL) same principles as before but easier to maintain Redesigned client and server architecture using aspect-oriented principles improves code reuse and maintainability validation, transaction, error handling, logging

Central Catalogs 0.3 Improved user input validation (client and server side) A unique transaction id (tuid) for the whole system improve tracking of bugs complete trace of events on each DQ2 call propagated down to the site services track initial subscription from user to final file registration on replica catalog Error handling is improved and connected to the transaction (tuid) error message presents this information to the user Error messages are now “pickled” Added logging to client and server side configurable and available in a file or database implementation

Central Catalogs 0.3 Implemented a pure REST interface Implemented a dataset RSS endpoint Repository catalog added dataset closed and frozen date Subscription catalog reset subscription => the same as deleting and creating a subscription cancel subscription => notify site services getInfo => retrieve more information regarding the subscription Content catalog no more duplicate file information between dataset versions

Central Catalogs 0.3 More testcases ran against both MySQL and Oracle implementations validate the system simulate previous bugs MySQL production instance maintained by DDM operations team MySQL 0.2 and 0.3 development instances maintained by DQ2 team intention to make a 0.3 “pre-production” instance Oracle 0.3 development instance maintained by Atlas Oracle admins and DQ2 team

Plans

Profiling system for DQ2 performance measurements content catalog first performance test against reveal same performance as 0.2… but 0.3 has more functionality! Implementation of performance tests on all DQ2 calls check impact of database schema changes Support of multiple client versions on the same central catalog make DDM server backward compatible API version number is now included in the URL Regional catalogs Cross-publishing

Plans Make some DQ2 calls transactional registerNewDataset registerFilesInDataset updateVersion Support for result paging Support for multiple output formats