Sept NVO Summer School1 The NVO DataScope: Internals Tom McGlynn NASA/GSFC T HE US N ATIONAL V IRTUAL O BSERVATORY

Sept NVO Summer School2 Summary DataScope as an example VO application Issues in building DataScope

Sept NVO Summer School3 Overall Organization CacheManagerDataScope VO Registry VO Resources User Sends request to DataScope Is this request cached? Yes: Just send results back to user No: find resources to query RequestManager Queue the requests and start them off Start telling the user about requests that have returned Cache the results as they come in

Sept NVO Summer School4 Perl VO Objects: Services Service URLAggregater Scanner Cone SIAP PositionalService Tarrer OASIS Aladin HeasarcResolver Resolver JHURegistryRegistry HeasarcScanner Services Not easily usable in other applications Portable object

Sept NVO Summer School5 Perl VO Library: Display Display Criterion State VOTableRendererServiceFilter

Sept NVO Summer School6 Perl VO Library: Control CacheManagerRequestManagerCacheableRequest

Sept NVO Summer School7 Simple Galaxy Checker Look for all resources with subject of galaxy (or galaxies) and see if there are any matches at a given position. Run service at:

Fix bug in SOAP… This is where VO library was placed Going to use these Convert user position to standard decimal J2000 RA/Dec Errors and defaults Create objects Query registry Loop over matches Make a cone search service and indicate where to search Display the results. Note how we use eval which allows us to recover from a parsing error if the VOTable is malformed. Query the cone search and get the VOTable We only look for galaxies at the beginning of the subject. We might use %Galax% to match anywhere.

Sept NVO Summer School9 How does DataScope use these objects? Main DataScope page is basically a HeasarcScanner which gets the data and a ServiceFilter which shows the results. Cone Search and SIAP results are shown using a VOTableRenderer –A few custom filters which have knowledge not in the VO registries are used to make links to ADS, MAST and HEASARC archives The URLAggregater handles most of the details of getting information to OASIS and Aladin Single file services (VOPlot, VOStat) currently use custom code at CGI script level.

Sept NVO Summer School10 DataScope: Registry Issues Registry is the most rapidly evolving element of the VO SOAP interface to registry –SOAP::Lite used in Perl One local fix to library to retrieve multiple string fields properly. –Setup may be non-trivial How do we select resources of interest? –Restrictions on ServiceType Cone or SIAP –Restrictions on ContentLevel –Should we use positional information about service? Not currently. The data is too ill-formed. Do we need to cache registry queries? –Takes several seconds, so not needed for DataScope Organizing resources –Use ServiceType, Subject and Content fields DataScope organization is just one of many possible ideas –Redundant and superseded resources Currently minimal information on these –Priority of resources Currently no information –More work needed if we are going to handle 10 3 to 10 4 resources No information on analysis services in registry –Hardwired into top level DataScope routines

Sept NVO Summer School11 DataScope: Cone and SIAP issues Search radius: –SIAP implicitly handles overlapping regions but Cone search does not. May need to increase size of region by ServiceSize to get all relevant rows from cone search services. Failed queries: –Should we retry? Overloading provider sites –Dont want to spit hundreds of queries to the same site simultaneously DataScope sorts resources by destination URL and only sends one query to a given host at a time. Non-compliant services –Cone searches returning multiple tables –Cone searches without position columns –SIAP services with improper UCD columns –But these are useful sites! (ADS, VizieR, XMM-Newton) Need to cache SIAP results before they expire How do we handle deferred SIAP results? Dont have any now. SIAP can have both quicklook and FITS files but the relationship between them is not defined. A registry entry may describe only some of the results returned by a SIAP request (using ID_MAIN column == ServiceName convention) –Look for multiple registry entries with identical ServiceURLs When will these services be available as Web services?

Sept NVO Summer School12 DataScope: Analysis software issues Currently two types of `analysis tools, some handle only one file at a time (VOPlot, TopCat, DS9,…) others take a set of URLs (Aladin, OASIS). URL Aggregating services may run over GET limits or limits on number of URLs that can be used. Single file tools may have more capabilities in non-applet modes Tools are not described in registry so knowledge of them is built into DataScope itself.

Sept NVO Summer School13 DataScope caching issues Not necessarily a VO issue Limits loads on data providers Enhances response time Data may go out of date. –Failed queries may now be available SIAP results may expire if left on provider Some services (e.g., Aladin) cannot currently handle FTP URLs (though this should change with latest versions of Java). SIAP (and SSAP) URLs may need to be cached to be available to such services.

Sept NVO Summer School14 Miscellaneous SkyNodes not accessible through DataScope. –Cone search probably provides appropriate functionality for DataScope. –Are all positional SkyNode tables available through cone search?

Sept NVO Summer School15 HEASARC Perl library distribution Very preliminary distribution with limited documentation and testing for the non- VOTable modules. Available at: – or volib.0.1.ziphttp://skyview.gsfc.nasa.gov/vo/volib.0.1.tar or volib.0.1.zip –Installation documentation at

Simple command line script to invoke an SIAP service and download all of the referenced files.