Interoperability and Prototype Project: Lessons Learned Jesús Salgado and the interoperability project members Planetary Science Archive (PSA) European Space Astronomy Centre (ESAC) ESA

Project Setup Initially as working group and as an activity between PDS and PSA (spring 2005) With the invocation of the IPDA, activity was moved over to an IPDA Project (autumn 2006) Project Leader: J. Salgado Project Members: Sean Kelly, Steve Hughes, Reta Beebe, Pedro Osuna, Ed Guinness, Susie Slavney, Joe Zender And with the support of M. Castro, T. Stein, and more Project closure: 7/2007 Project status: use cases successfully finished

Planetary Data Access Protocol (PDAP) One of the main objectives of the IPDA (not the only one) is the interoperability General standard protocols for planetary data would provide easy access to data from other IPDA members and the creation of general/light clients Main existing outcome from this project:: PDAP (Planetary Data Access Protocol) Concept was born at the ESA/PSA and NASA/PDS Technical Interoperability Meeting, held January 10-12, 2006 in Madrid Spain Current version v0.4 working draft (July 2007) Protocol to access Datasets, products and images Two servers implementations already in place (PDS and PSA) Two client implementations, different flavors PSA Mars Map client: Geometrical searches for PDS/PSA products PDS dataset/product browser: PSA datasets/products access through PDAP

Planetary Data Access Protocol PDAP is a two steps protocol: Metadata Access: Software Clients search for available data that match certain criteria. The matching criteria includes specific metadata and PDS keywords Data Retrieval: Software client retrieve through a synchronous HTTP GET/POST request using a reference URL returned from first step Any PDAP server service implementation should be registered. Registration allows service discovery and get access to publisher/curation information Interchange default format is VOTable (XML). This format can be easily parsed by a client and displayed in different ways

First Use Case: Access to Data Sets and Products using PDAP PDAP server created on all the PSA public data accessible form the standard PSA archive PDAP server created for PDS GRS data. This server was extended for the next use case (including map information) Client developed by the PDS project members to access PSA remote data through PDAP Similar client aspect and almost transparent access to both local and remote data

PDS OODT/PDAP Client

Conclusions of the first use case Probe of concept quite successful Easy and fast to implement It allows agencies to maintain curation and control of the distribution in an efficient way Duplication of storage and synchronization problems prevented Negative aspects: Not all the metadata present in the PDAP records (more Dublin core and characterization metadata to be included in protocol) Remote server-client metadata interchange; less performance

Second Use Case: Map based queries using PDAP

Second use case: Implementations needed This use case is a very complex one. It implies the following implementations: Two PDAP servers (one at PDS and one at PSA) should be implemented. Both servers must allow clients to query using geometrical constraints and output must contain footprint information PDS PDAP server developed on GRS data PSA PDAP server developed on Mars Express data At least, one map based PDAP client should be implemented Client developed by the PSA team @ ESTEC using IDL

Map based queries PDAP servers selection Region of interest

Map based queries (II) PDS Server find 8 products and PSA server 43 products (difficult to visualize as GRS footprints are small)

Query construction Both servers receive same query and return VOTable responses http://psa.esac.esa.int/aio/jsp/metadata.jsp?RESOURCE_CLASS=IMAGE&TARGET_NAME=MARS&MINI MUM_LATITUDE>19.0&MAXIMUM_LATITUDE<19.1&MINIMUM_LONGITUDE>225.8&MAXIMUM_LONGIT UDE<225.9 http://www.planetarydata.org/grs/pdap.jsp?RESOURCE_CLASS=IMAGE&TARGET_NAME=MARS&MINI MUM_LATITUDE>19.0&MAXIMUM_LATITUDE<19.1&MINIMUM_LONGITUDE>225.8&MAXIMUM_LONGIT UDE<225.9 Client uses “<“ “>” operators to constraint the region of interest (note this has been deprecated to prevent the use of these symbols and to define ranges in a more flexible way). Recommended: http://psa.esac.esa.int/aio/jsp/metadata.jsp? RESOURCE_CLASS=IMAGE& RETURN_TYPE=VOTABLE& TARGET_NAME=MARS&LATITUDE=19.0/19.1&LONGITUDE=225.8/225.9 Scheme Query Type Response return format And Query!!!

Conclusions of the second use case: Server side More problems were found to implement this complex use case Footprint information is contained in index files (not in labels). Not defined format in PDS. PDAP defines one format through polygons, circles, etc There were some problems to access this information for the PDS PDAP footprint implementation. That produced delay PSA had already a good support for it Performance problems: Queries can take long. Reasons: Number of products can be huge: Pagination Searches needs to be optimized in the server side (e.g. by using database geometrical indexing modules like healpix, pgshere, HTM,.. But: Not all the PDAP servers should implement RESOURCE_CLASS=IMAGE server,.. Try to implement it for DATASETS and PRODUCTS!!!!

Conclusions of the second use case: Client side Lack of support for VOTable parsers for IDL (excellent support for JAVA and C). Small IDL parser developed from scratch and problems with VOTable versioning and fields not in correct order Due to not IPDA related problems, unfortunately Map client implementation and support was stopped for some months Plugins to manipulate data? (only browse products can be opened in present version)

Items Found and conclusions PDS is quite open and it allows GROUP (multiple values to the same keyword) Geometrical information is in INDEX files Keywords from the PDS Standard are often “overloaded” It is necessary to create a Query Data Model so the PDAP can work effectively Query and retrieval integrated into existing interfaces – Mostly achieved Implementation only depends on the draft protocol standard – Successful Cheap and Quick solution – Longer than expected Lessons learned!

Comments to PDAP

Relation with VO for astronomy The protocol has been designed close to SIAP protocol from the IVOA, but maintaining planetary data spirit: PDS keywords for the metadata PDS format for the data We consider the protocol in the correct balance Future planetary applications would be able to query IPDA and IVOA resources in a quite easy/transparent way PDAP authors have received comments in both directions Should the protocol be based more on “proprietary” planetary formats? Is the present approach the correct one? Steering Committee decision

Query refining PDAP needs to evolve to a better query definition. The query parameters are only described in the document through the parameter=value paradigm. The query uses a default “AND” operator to do the query. This has been done to maintain, at the beginning, the protocol as simpler as possible It could be nice to have a better query mechanism, allowing not only the use of “AND” operators but “OR”, “NOT”, etc. This could be done (PDS has experience in this field) in the following way: http://<service_URL>/<service>? QUERY=“(param1=value1&&param2=value2)||(param3!=value3)”

Refining the result PDAP needs to evolve to a better response definition too In the current document, the response contains some fixed- compulsory fields per class type (data set, product, image,…) These metadata is the one that, in principle, are necessary to fully characterize the element However, there is not way to pre-select in the client what the client wants in the response (only by post-filtering) This could also be done (PDS has experience in this field) in the following way: http://<service_URL>/<service>?QUERY=“<query>”& RETURN=“class.field1,class.field2,motherclass.field1,motherclass.field2 Easy for free fields, problem for compulsory?

Proprietary data A HTTP protocol like the PDAP can present some problems to allow access to proprietary data For the time being, only access to public data is done for PDAP servers This could be a reason to change the protocol to something more complex, or to extend to https https implementation in servers is trivial https implementation in clients is more complex, but if the client is developed in language with a good community support, e.g. Java, this problem is already solved Not clear if we have a real need to interchange proprietary data through PDAP services in current stage, but the service should be flexible enough to support this mode Login/password

Pagination The number of products or even data sets that fulfill a certain query can be very large There is a need to incorporate a way to establish pagination in the protocol Of course, this pagination should be done in the server (the client can paginate the results, but the problem found is really in the metadata interchange) First, the problem is in which input parameters/result fields can be used for pagination. Some approaches include: TOP, e.g., TOP=10: Return the top first 10 records that fulfill the query (however, not clear at all how to decide in the server the “top ten” records) RECORDS and TOKEN: RECORDS can be used to define how many records the client admit in one page. A TOKEN field will be returned in the response

Other issues UNICODE versus plain ASCII Not need for character code at this stage (including JAXA) 1 byte ASCII code is enough for the time being It could be needed in a later stage, so we should try to get some expertise into the authors members Irregular bodies Can we extend the simple longitude/latitude queries to irregular bodies? If not, how to query different servers that contains information for the same planetary object? Is this a problem to be solved in the protocol or we need some data model support to define the queries in the protocol?

Questions?