1. Interoperability and Prototype
Project: Lessons Learned
Jesús Salgado
and the interoperability project members
Planetary Science Archive (PSA)
European Space Astronomy Centre (ESAC)
ESA

2. Project Setup
Initially as working group and as an activity between PDS and PSA (spring 2005)
With the invocation of the IPDA, activity was moved over to an IPDA Project (autumn 2006)
Project Leader: J. Salgado
Project Members: Sean Kelly, Steve Hughes, Reta Beebe, Pedro Osuna, Ed Guinness, Susie Slavney, Joe Zender
And with the support of M. Castro, T. Stein, and more
Project closure: 7/2007
Project status: use cases successfully finished

3. Planetary Data Access Protocol (PDAP)
One of the main objectives of the IPDA (not the only one) is the interoperability
General standard protocols for planetary data would provide easy access to data from other IPDA members and the creation of general/light clients
Main existing outcome from this project::
PDAP (Planetary Data Access Protocol)
Concept was born at the ESA/PSA and NASA/PDS Technical Interoperability Meeting, held January 10-12, 2006 in Madrid Spain
Current version v0.4 working draft (July 2007)
Protocol to access Datasets, products and images
Two servers implementations already in place (PDS and PSA)
Two client implementations, different flavors
PSA Mars Map client: Geometrical searches for PDS/PSA products
PDS dataset/product browser: PSA datasets/products access through PDAP

4. Planetary Data Access Protocol
PDAP is a two steps protocol:
Metadata Access: Software Clients search for available data that match certain criteria. The matching criteria includes specific metadata and PDS keywords
Data Retrieval: Software client retrieve through a synchronous HTTP GET/POST request using a reference URL returned from first step
Any PDAP server service implementation should be registered. Registration allows service discovery and get access to publisher/curation information
Interchange default format is VOTable (XML). This format can be easily parsed by a client and displayed in different ways

5. First Use Case: Access to Data Sets and Products using PDAP
PDAP server created on all the PSA public data accessible form the standard PSA archive
PDAP server created for PDS GRS data. This server was extended for the next use case (including map information)
Client developed by the PDS project members to access PSA remote data through PDAP
Similar client aspect and almost transparent access to both local and remote data

6. PDS OODT/PDAP Client

7. Conclusions of the first use case
Probe of concept quite successful
Easy and fast to implement
It allows agencies to maintain curation and control of the distribution in an efficient way
Duplication of storage and synchronization problems prevented
Negative aspects:
Not all the metadata present in the PDAP records (more Dublin core and characterization metadata to be included in protocol)
Remote server-client metadata interchange; less performance

8. Second Use Case: Map based queries using PDAP

9. Second use case: Implementations needed
This use case is a very complex one. It implies the following implementations:
Two PDAP servers (one at PDS and one at PSA) should be implemented. Both servers must allow clients to query using geometrical constraints and output must contain footprint information
PDS PDAP server developed on GRS data
PSA PDAP server developed on Mars Express data
At least, one map based PDAP client should be implemented
Client developed by the PSA ESTEC using IDL

10. Map based queries
PDAP servers selection
Region of interest

11. Map based queries (II)
PDS Server find 8 products and PSA server 43 products
(difficult to visualize as GRS footprints are small)

13. Query construction
Both servers receive same query and return VOTable responses
MUM_LATITUDE>19.0&MAXIMUM_LATITUDE<19.1&MINIMUM_LONGITUDE>225.8&MAXIMUM_LONGIT UDE<225.9
MUM_LATITUDE>19.0&MAXIMUM_LATITUDE<19.1&MINIMUM_LONGITUDE>225.8&MAXIMUM_LONGIT UDE<225.9
Client uses “<“ “>” operators to constraint the region of interest (note this has been deprecated to prevent the use of these symbols and to define ranges in a more flexible way). Recommended:
RESOURCE_CLASS=IMAGE&
RETURN_TYPE=VOTABLE&
TARGET_NAME=MARS&LATITUDE=19.0/19.1&LONGITUDE=225.8/225.9
Scheme
Query Type
Response return format
And Query!!!

14. Conclusions of the second use case: Server side
More problems were found to implement this complex use case
Footprint information is contained in index files (not in labels). Not defined format in PDS. PDAP defines one format through polygons, circles, etc
There were some problems to access this information for the PDS PDAP footprint implementation. That produced delay
PSA had already a good support for it
Performance problems: Queries can take long. Reasons:
Number of products can be huge: Pagination
Searches needs to be optimized in the server side (e.g. by using database geometrical indexing modules like healpix, pgshere, HTM,..
But: Not all the PDAP servers should implement RESOURCE_CLASS=IMAGE server,.. Try to implement it for DATASETS and PRODUCTS!!!!

15. Conclusions of the second use case: Client side
Lack of support for VOTable parsers for IDL (excellent support for JAVA and C). Small IDL parser developed from scratch and problems with VOTable versioning and fields not in correct order
Due to not IPDA related problems, unfortunately Map client implementation and support was stopped for some months
Plugins to manipulate data? (only browse products can be opened in present version)

16. Items Found and conclusions
PDS is quite open and it allows GROUP (multiple values to the same keyword)
Geometrical information is in INDEX files
Keywords from the PDS Standard are often “overloaded”
It is necessary to create a Query Data Model so the PDAP can work effectively
Query and retrieval integrated into existing interfaces – Mostly achieved
Implementation only depends on the draft protocol standard – Successful
Cheap and Quick solution – Longer than expected
Lessons learned!

17. Comments to PDAP

18. Relation with VO for astronomy
The protocol has been designed close to SIAP protocol from the IVOA, but maintaining planetary data spirit:
PDS keywords for the metadata
PDS format for the data
We consider the protocol in the correct balance
Future planetary applications would be able to query IPDA and IVOA resources in a quite easy/transparent way
PDAP authors have received comments in both directions
Should the protocol be based more on “proprietary” planetary formats? Is the present approach the correct one? Steering Committee decision

19. Query refining PDAP needs to evolve to a better query definition.
The query parameters are only described in the document through the parameter=value paradigm. The query uses a default “AND” operator to do the query. This has been done to maintain, at the beginning, the protocol as simpler as possible
It could be nice to have a better query mechanism, allowing not only the use of “AND” operators but “OR”, “NOT”, etc.
This could be done (PDS has experience in this field) in the following way:
QUERY=“(param1=value1&&param2=value2)||(param3!=value3)”

20. Refining the result
PDAP needs to evolve to a better response definition too
In the current document, the response contains some fixed- compulsory fields per class type (data set, product, image,…)
These metadata is the one that, in principle, are necessary to fully characterize the element
However, there is not way to pre-select in the client what the client wants in the response (only by post-filtering)
This could also be done (PDS has experience in this field) in the following way:
RETURN=“class.field1,class.field2,motherclass.field1,motherclass.field2
Easy for free fields, problem for compulsory?

21. Proprietary data
A HTTP protocol like the PDAP can present some problems to allow access to proprietary data
For the time being, only access to public data is done for PDAP servers
This could be a reason to change the protocol to something more complex, or to extend to https
https implementation in servers is trivial
https implementation in clients is more complex, but if the client is developed in language with a good community support, e.g. Java, this problem is already solved
Not clear if we have a real need to interchange proprietary data through PDAP services in current stage, but the service should be flexible enough to support this mode
Login/password

22. Pagination
The number of products or even data sets that fulfill a certain query can be very large
There is a need to incorporate a way to establish pagination in the protocol
Of course, this pagination should be done in the server (the client can paginate the results, but the problem found is really in the metadata interchange)
First, the problem is in which input parameters/result fields can be used for pagination. Some approaches include:
TOP, e.g., TOP=10: Return the top first 10 records that fulfill the query (however, not clear at all how to decide in the server the “top ten” records)
RECORDS and TOKEN: RECORDS can be used to define how many records the client admit in one page. A TOKEN field will be returned in the response

23. Other issues UNICODE versus plain ASCII
Not need for character code at this stage (including JAXA)
1 byte ASCII code is enough for the time being
It could be needed in a later stage, so we should try to get some expertise into the authors members
Irregular bodies
Can we extend the simple longitude/latitude queries to irregular bodies?
If not, how to query different servers that contains information for the same planetary object?
Is this a problem to be solved in the protocol or we need some data model support to define the queries in the protocol?

24. Questions?