1. Prototype Web Services Using SDSS DR1
Alex Szalay, Tamas Budavari, Sam Carlisle, Jim Gray, Vivek Haridas, Nolan Li, Tanu Malik, Maria Nieto-Santisteban, Wil O’Mullane, Ani Thakar

2. NVO: How Will It Work? Define commonly used ‘core’ services
Build higher level toolboxes/portals on top
We do not build ‘everything for everybody’
Use the rule:
Define the standards and interfaces
Build the framework
Build the 10% of services that are used by 90%
Let the users build the rest from the components

3. Using SDSS DR1 SDSS DR1 (Data Release1) is now publicly available
About 1TB of catalog data
Using MS SQL Server 2000
Complex schema (72 Tables)
About 80 million photometric objects
Two versions (TARGET/BEST)
Automated documentation
Raw data at FNAL file server with URL access

4. Loading DR1 Automated table driven workflow system for loading
Included lots of verification code
Over 16K lines of SQL code
Loading process was extremely painful
Lack of systems engineering for the pipelines
Poor testing (lots of foreign key mismatch)
Detected data bugs even a month ago
Most of the time spent on scrubbing data
Fixing corrupted files (RAID5 disk errors)
Once data was clean, everything loaded in 3 days
Neighbors calculation took about 10 hours
Reorganization of data took about 1 week of experiments in partitioning/layouts

5. Reorganization Introduced partitions and filegroups
Photo, Tag, Neighbors, Spectro, Frame, Other, Profiles
Keep partitions under 100GB
Vertical partitioning – tried and abandoned
Both partitioning and index build now table driven
Stored procedures to create/drop indices at various granularities
Tremendous improvement in performance when doing this on a large memory machine (24GB)
Also much better performance afterwards

6. Spatial Features Precomputed Neighbors Boundaries, Masks and Outlines
All objects within 30”
Boundaries, Masks and Outlines
Stored as spatial polygons
Time Domain:
Precomputed Match
All objects with 1”, observed at different times
Found duplicates due to telescope tracking errors
Manual fix, recorded in the database
MatchHead
The first observation of the linked list used as unique id to chain of observations of the same object

7. Spatial Algorithms Updated HTM library Zones
Automated depth for HTM_Cover
Output vertices
Simplify polygon
Boolean operations on regions
Part of VO data model (A. Rots)
Zones
Much better performance for bulk neighbors at a fixed radius
Footprint service in progress
Bool Contains(point)
Region Intersect(region)

8. Web Services in Progress
Registry
Harvesting and querying
Data Delivery
Query driven Queue management
Graphics and visualization
Query driven vs interactive
Show spatial objects (Chart/Navi/List)
Footprint/intersect
It is a “fractal”
Cross-matching
SkyQuery and SkyNode
Ferris-wheel
Distributed vs parallel

9. Registry: Easy Clients
Just use SOAP toolkit (T. McGlynn & J. Lee have done Perl client).
Easy in Java
java org.apache.axis.wsdl.WSDL2Java "
Gives set of Classes for accessing the service
Gives Classes for the XML which is returned (i.e. SimpleResource)
Still need to write client like
RegistryLocator loc = new RegistryLocator();
RegistrySoap reg = loc.getRegistrySoap();
ArrayOfSimpleResource reses = null;
reses = reg.queryRegistry(args[0]);

10. Generic Catalog Access
After 2 years of SDSS EDR and 6 months of DR1 usage, access patterns start to emerge
Lots of small users, requiring instant response
1/f distribution of request sizes (tail of the lognormal)
How to make everybody happy?
No clear business model…
We need a separate interactive and batch server
We also need access to full SQL with extensions
Users want to access services via browsers
Other services will need SOAP access

11. Data Formats Different data formats requested:
HTML, CSV, FITS binary, VOTABLE, XML, graphics
Quick browsing and exploration
Small requests, need to be nicely rendered
Needs good random access performance
Also simple 2D scatter plots or density plots required
Heavy duty statistical use
Aggregate functions on complex joins, lots of scans but small output, mostly want CSV
Successive Data Filter
Multi-step non-indexed filtering of the whole database, mostly want FITS binary

12. Data Delivery Small requests (<100MB) Medium requests (<1GB)
Putting data on the stream
Medium requests (<1GB)
Use DIME attachments to SOAP messages
Large requests (>1GB)
Save data in scratch area and use asynch delivery
Only practical for large/long queries
Iterative requests
Save data in temp tables in user space
Let user manipulate via web browser
Paradox: if we use web browser to submit, users want immediate response from batch-size queries

13. How To Provide a UserDB
Goal: through several search/filter operations reduce data transfer to manageable sizes (1-100MB)
Today: people download tens of millions of rows, and then do their next filtering on client side, using F77
Could be much better done in the database
But: users need to create/manage temporary tables
DOS attacks, fragmentation, who pays for it
Security, who can see my data (group access)?
Follow progress of long jobs
Who does the cleanup?

14. Query Managament Service
Enable fast, anonymous access to small requests
Enable large queries, with ability to manage
Enable creation of temporary tables in user space
Create multiple ways to get query output
Needs to support multiple mirrors/load balancing
Do all this without logging in to Windows
Need also support of machine clients
Web Service:
Two request categories:
Quick
Batch

15. Queue Management Need to register batch ‘power users’
Query output goes to ‘MyDB’
Can be joined with source database
Results are materialized from MyDB upon request
Users can do:
Insert, Drop, Create, Select Into, Functions, Procedures
Publish their tables to a group area
Data delivery via the CASService (C# WS)

16. Graphics Tools
Simple xy plots
Density plot
Chart/Navi/List
Can be built into various applications

17. Archive Footprint Footprint is a ‘fractal’ Result depends on context
all sky, degree scale, pixel scale
Translate to web services
Footprint() returns single region that contains the archive
Intersection(region, tolerance) feed a region and returns the intersection with archive footprint
Contains(point) returns yes/no (maybe fuzzy) if point is inside archive footprint

18. Cross-Matching SkyQuery – SkyNode
Currently lots of proprietary features
Data transmitted via .NET DataSet => VOTable
Query plan written in MS T-SQL => ADQL
Spatial operator restricted to a cone =>VORegion
Made up metadata delivery => VORegistry
Data delivery in XML/HTML => VOTable
Catalogs in the near future
SDSS DR1, FIRST, 2MASS, INT
POSS-1, GSC-2, HST, ROSAT, 2dF
GALEX, IRAS, PSCZ

19. Spatial Cross-Match
For small area HTM is close to optimal, but needs more speed
For all-sky surveys the zone algorithm is best
Current heuristic is a linear chain of all nodes
Easy to generalize to include precomputed neighbors
But, for all sky queries very large number of random reads instead of sequential

20. Ferris-Wheel Sky split into buckets/zones All archives scan in sync
Queries enter at bottom
Results come back after full circle
Only sequential access => buckets get into cache, then queries processed
Portal
SDSS

21. Utilitites
FITSLIB C# library around the CFITSIO package
MIRAGE Java wrapper around Mirage, can directly access the VORegistry, and ConeSearch
HTM2.0 Updated HTM library, conforming to the new Region specification
ADQL Prototype service to convert back and forth between ADQL and SQL
SDSSQA Java application, emulating MS Query Analyzer

22. Summary Web Services have been remarkably easy to use
Now different platforms are interoperable
We have invested a lot of energy to develop various interface libraries (FITS, VOTable)
Integrating graphics into web services was very easy
Next:
Parallel queries
Finish query queue management
Upgrade SkyQuery
Bring in more archives
Ferris-Wheel experiment
On-demand database creation
100TB parallel data access layer