1. Data Challenges I'm Struggling With Jim Gray, Microsoft Research 1.Sneakernet is probably the best way to moving WAN data at 1GBps File transfer efforts are currently 550MBps via Internet2. How to manage the multi-petybyte file repository we are about to generate. 2.The TerraServer has evolved from a mainframe to a bunch of bricks. The new design has been operating for a year and we are quite pleased with it. But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?. 3.The SkyServer website is built using database technology and web services. Now moving the web services inside the database. Others are working to design a scale-out version of the server. There are several interesting data challenges in these changes. 4.Using relational tuples to represent spatial volumes as constraints. Point-in-polygon and polygon-overlap queries can then be quickly evaluated. I will briefly describe this idea.

2. How Do You Move A Terabyte? 14 minutes6172001,920,0009600OC 1922.2 hours1000Gbps 1 day100100 Mpbs 14 hours97631649,000155OC3 2 days2,01065128,00043T3 2 months2,4698001,2001.5T1 5 months360117 50 0.6Home DSL 6 years3,0861,000400.04 Home phone Time/TB $/TB Sent $/Mbps Rent $/month Speed Mbps Context Source: TeraScale Sneakernet, Source: TeraScale Sneakernet, Microsoft Technical Report May 2002, MSR-TR-2002-54 http://research.microsoft.com/research/pubs/view.aspx?tr_id=569

3. Moving Data Bricks WAN costs >> 100$/Mbps/month >> 1$/GB Beowulf networking 10,000x cheaper than WAN factors of 10 5 matter. The cheapest and fastest way to move a Terabyte cross country is sneakernet. 24 hours = 4 MB/s 50$ shipping vs 1,000$ wan cost.

4. Giga Byte Per Second File Mover CERN to Pasadena –Windows TCP/IP stack improvements –Opteron demo –Disk-to-Disk at 550MBps now (~2 TB/Hour) What we learned: –Linux tcp stack is good/better at high perf we are catching up. –NTFS is better than various Linux FS –Near the PCI-X limit –Good way to engage the community. GOAL: 1GBps disk-to-disk. OC192 = 9.9 Gbps PCI -X limit tcp limit

5. But then what? Managing Petabytes CERN files are 30MB They produce 1 B files/year. How name them? How manage them? Depends on workload: how use them. Its a DB problem.

6. Data Challenges I'm Struggling With Jim Gray, Microsoft Research 1.Sneakernet is probably the best way to moving WAN data at 1GBps File transfer efforts are currently 550MBps via Internet2. How to manage the multi-petybyte file repository we are about to generate. 2.The TerraServer has evolved from a mainframe to a bunch of bricks. The new design has been operating for a year and we are quite pleased with it. But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?. 3.The SkyServer website is built using database technology and web services. Now moving the web services inside the database. Others are working to design a scale-out version of the server. There are several interesting data challenges in these changes. 4.Using relational tuples to represent spatial volumes as constraints. Point-in-polygon and polygon-overlap queries can then be quickly evaluated. I will briefly describe this idea.

7. KVM / IP TerraServer – Whats new Web Service and Web Server New ~1 ft 2 /pixel full color image of 120 urban areas Storage Bricks –Commodity servers –4 TB raw / 2 TB Raid1 SATA storage –Dual 2 GHz + 4GB RAM –3 Bricks = TerraServer data –Data partitioned –Moving to Yukon –Working on low TCO auto-manage Low Cost Availability Pair & Spare –RAID1 Mirroring –Mirrored Bunches (Yukon log ship?) –Spare Brick –Web Application Load balances mirrors Uses surviving database on failure

8. TerraServer Challenges Best Geoplex strategy? Moving Web Services into the DB? Managing bunches (lower TCO).

9. Data Challenges I'm Struggling With Jim Gray, Microsoft Research 1.Sneakernet is probably the best way to moving WAN data at 1GBps File transfer efforts are currently 550MBps via Internet2. How to manage the multi-petybyte file repository we are about to generate. 2.The TerraServer has evolved from a mainframe to a bunch of bricks. The new design has been operating for a year and we are quite pleased with it. But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?. 3.The SkyServer website is built using database technology and web services. Now moving the web services inside the database. Others are working to design a scale-out version of the server. There are several interesting data challenges in these changes. 4.Using relational tuples to represent spatial volumes as constraints. Point-in-polygon and polygon-overlap queries can then be quickly evaluated. I will briefly describe this idea.

10. Skyserver Personal DB

11. Quick Overview (Services) SkyServer (skyserver.sdss.org) –Web site delivers Sloan Digital Sky Survey data –Also has education –1,000x less popular than Terraserver, but HUGE for a science website. A Batch Job System with Personal DBs –Lets users run jobs http://casjobs.sdss.org/CasJobs/ http://casjobs.sdss.org/CasJobs/ –Parameters & Answers to & from Personal DB –Simple batch job scheduler. Web Services: http://www.voservices.org/ http://www.voservices.org/ –Photographic objects –Spectrographic objects –Transformation functions –7 out of the 8 are.NET.

12. Quick Overview (Integration) SkyQuery ( http://skyquery.net/) Distributed Query portal using web services http://skyquery.net/ Archives: Pasadena, Chicago, Baltimore, Cambridge (England ) Has grown from 4 to 20 archives, ½ day to add new one (adapt DB and register with Portal) becoming international standard 2MASS INT SDSS FIRST SkyQuery Portal Image Cutout Each SkyNode publishes –Schema Web Service –Database Web Service Portal –Plans Query (2 phase) –Integrates answers –Is itself a web service

13. Challenges Parallel data search (data pump). How to partition? How manage load Moving web services to DB What is the right approach? Move objects into DB Spatial access methods Data analysis in the DB.

14. Data Challenges I'm Struggling With Jim Gray, Microsoft Research 1.Sneakernet is probably the best way to moving WAN data at 1GBps File transfer efforts are currently 550MBps via Internet2. How to manage the multi-petybyte file repository we are about to generate. 2.The TerraServer has evolved from a mainframe to a bunch of bricks. The new design has been operating for a year and we are quite pleased with it. But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?. 3.The SkyServer website is built using database technology and web services. Now moving the web services inside the database. Others are working to design a scale-out version of the server. There are several interesting data challenges in these changes. 4.Using relational tuples to represent spatial volumes as constraints. Point-in-polygon and polygon-overlap queries can then be quickly evaluated. I will briefly describe this idea.

15. A Detail: 3 Ways We Do Spatial? Hierarchical mesh (extension to SQL) –Uses table valued stored procedures –Acts as a new spatial access method –Porting to Yukon CLR for a 10x speedup. Zones: fits SQL like a glove –Amazingly simple, amazingly good. Constraints: a really novel idea –Lets us do algebra on regions. Paper: There Goes the Neighborhood: Relational Algebra for Spatial Data Search There Goes the Neighborhood: Relational Algebra for Spatial Data Search Idea in backup slides.

16. Equations Define Subspaces For (x,y) above the line ax+by > c Reverse the space by -ax + -by > -c Intersect a 3 volumes: a 1 x + b 1 y > c 1 a 2 x + b 2 y > c 2 a 3 x + b 3 y > c 3 x y x=c/a y=c/b ax + by = c x y

17. Domain is Union of Convex Hulls Simple volumes are unions of convex hulls. Higher order curves also work Complex volumes have holes and their holes have holes. (that is harder). Not a convex hull +

18. Now in Relational Terms create table HalfSpace ( domainID int not null -- domain name foreign key references Domain(domainID), convexID int not null,-- grouping a set of ½ spaces halfSpaceID int identity(),-- a particular ½ space x float not null, -- the (a,b,..) parameters y float not null, -- defining the ½ space z float not null, cfloat not null, -- the constant (c above) primary key (domainID, convexID, halfSpaceID) (x,y,z) inside a convex if it is inside all lines of the convex (x,y,z) inside a convex if it is NOT OUTSIDE ANY line of the convex select convexID-- return the convex hulls from HalfSpace-- from the constraints where @x * x + @y * y + @x * z < l -- point outside the line? group by all convexID-- consider all the lines of a convexID having count(*) = 0 -- count outside == 0

19. The Algebra is Simple (Boolean) @domainID = spDomainNew (@type varchar(16), @comment varchar(8000)) @convexID = spDomainNewConvex (@domainID int) @halfSpaceID = spDomainNewConvexConstraint (@domainID int, @convexID int, @x float, @y float, @z float, @l float) @returnCode = spDomainDrop(@domainID) select * from fDomainsContainPoint(@x float, @y float, @z float) Once constructed they can be manipulated with the Boolean operations. @domainID = spDomainOr (@domainID1 int, @domainID2 int, @type varchar(16), @comment varchar(8000)) @domainID = spDomainAnd (@domainID1 int, @domainID2 int, @type varchar(16), @comment varchar(8000)) @domainID = spDomainNot (@domainID1 int, @type varchar(16), @comment varchar(8000))

20. What! No Bounding Box? Bounding box limits search. A subset of the convex hulls. If query runs at 3M halfspace/sec then no need for bounding box, unless you have more than 10,000 lines. But, if you have a lot of half-spaces then bounding box is good.

21. HTM Approach Table-valued function find points near a point –Select * from fGetNearbyEq(ra,dec,r) Use Hierarchical Triangular Mesh www.sdss.jhu.edu/htm/ www.sdss.jhu.edu/htm/ –Space filling curve, bounding triangles… –Standard approach 13 ms/call… So 70 objects/second. Too slow, so precompute neighbors: Materialized view. At 70 objects/sec it takes 6 months to compute a billion objects.

22. Zone Approach Divide space into zones Key points by Zone, offset (on the sphere this need wrap-around margin.) Point search look in a few zones at a limited offset: ra ± r a bounding box that has 1-π/4 false positives All inside the relational engine Avoids impedance mismatch Can batch all-all comparisons 33x faster and parallel 6 days, not 6 months! r ra-zoneMax (r 2 +(ra-zoneMax) 2 ) cos(radians(zoneMax)) zoneMax x Ra ± x

23. In SQL select o1.objID -- find objects from zone o1 -- in the zoned table where o1.zoneID between -- where zone # floor((@dec-@r)/@zoneHeight) and-- overlaps the circle floor((@dec+@r)/@zoneHeight) and o1.ra between @ra - @r and @ra + @r-- quick filter on ra and o1.dec between @dec-@r and @dec+@r -- quick filter on dec and ( (sqrt( power(o1.cx-@cx,2)+power(o1.cy-@cy,2)+power(o1.cz-@cz,2)))) < @r -- careful filter on distance Eliminates the ~ 21% = 1-π/4 False positives Bounding box

24. Summary SQL is a set oriented language You can express constraints as rows Then You –Can evaluate LOTS of predicates per second –Can do set algebra on the predicates. Benefits from SQL parallelism SQL == Prolog?