Microsoft Large Databases and Grid Computing

Microsoft Large Databases and Grid Computing
Jim Gray Microsoft Research Presentation to Kaiser Information Management Briefing 21 May 2003

About me in Microsoft research (located in San Francisco)
A database researcher IBM, Tandem, DEC, Microsoft Work on Scalable Systems Building supercomputers from commodity components. Do academic/government things too PITAC, GriPhyn TAB, NSF/CISE, Library of Congress, … For the last 4 years, been working with the astronomy community to build the World Wide Telescope.

Agenda TerraServer SkyServer / WWT Grid Computing What it is
What we learned What we are doing now. SkyServer / WWT What we are doing now Grid Computing General comments Build a web service

TerraServer TerraService.net
A photo of the United States 1 meter resolution (photographic/topographic) USGS data Some demographic data (BestPlaces.net) Home sales data Linked to Encarta Encyclopedia 15 TB raw, 6 TB cooked (grows 10GB/w) Point, Pan, zoom interface Among top 1,000 websites 40k visitors/day 4M queries/day 3 B page views (in 5 years) All in an SQL database

TerraServer Statistics
June ‘98 Jan ‘99 Jan ‘00 May ‘00 Sept ’01 Dec ‘02 Unique Users Page Views Image Tiles Db Queries Bytes Xfered Daily Average 40,011 1,266,838 3,735,789 4,484,089 70 gb Peak Day 277,292 12,388,104 10,475,674 163 gb 2,401,209 June Oct, 2002 63,656,904 2,015,539,605 5,943,641,024 7,134,186,170 108tb 900 m Rows 755m Rows SQL TB Db SQL TB Db SQL TB Db SQL Server 1.5 TB Db SQL Server .8 TB Db 298 m Rows SQL TB Db 231 m Rows SQL TB Db SQL TB Db 217 m Rows SQL TB Db SQL TB Db 173 m Rows SQL TB Db SQL TB Db 1 Server / Win NT 4.0 EE 2nd Server / Win 2k DataCenter 4 Node / Win2k Datacenter Failover Cluster

8 Compaq DL360 “Photon” Web Servers 4 Compaq ProLiant 8500 Db Servers
TerraServer Cluster 8 Compaq DL360 “Photon” Web Servers One SQL database per rack Each rack contains 4.5 TB 1 rack not in picture 18.0 TB total 2200 Fiber SAN Switches E J O Meta Data Stored on 101 GB “Fast, Small Disks” (18 x 18.2 GB) SQL\Inst1 F G L K P Q Imagery Data Stored on GB “Slow, Big Disks” (15 x 73.8 GB) This is a picture of the original TerraServer database server located at Microsoft’s Internet Data Center. This system was the primary database server from when we launched the TerraServer web site in June 1998 until the system was retired in October I claim that is the world’s largest PC, checking in at 24.5 feet and approximately 7 tons. I have not been challenged on this point The white cabinet on the right is the DEC AlphaServer It has 8 440mhz processors and 10GB of RAM. We are running NT Server V4.0 Enterprise Edition which only uses 2 GB of RAM. But we were well prepared to migrate to Windows 2000 Data Center edition. There are seven slate blue cabinets to the left of the processor. These house the Compaq StorageWorks disk arrays -- containing the UltraSCSI disk drives and HSZ70 Raid controllers. There are a total of GB drives per cabinet. Each drive is connected to two HSZ70 controllers that operate in a hot-standby, failsafe mode. The HSZ70 binds 11 drives into a single RAID-5 control set. 2 Drives per cabinet are used as “hot spares” and are transparently swapped in in case there is a disk failure. Each HSZ70 pair is connected to a separate KPSB controller on in the DEC Alpha 8400 CPU cabinet. The last white box on the far left is our StorageTek TimberWolf 9710 tape robot. It contains 10 Quantum DLT7000 tape drives. A pair of DLT7000 drives are connected on a Fast Wide Differential SCSI connection to the DEC AlphaServer The StorageTek tape robot takes commands from Legato Networker to backup or restore database files to/from the tape drives. Legato Networker is able to move data to/from all 10 tape drives in parallel. Using file backup, this configuration can backup the entire 1.0 TB SQL Server database files in 7 hours and 15 minutes. Also note, this picture does not include the web servers. We started with 4 Compaq ProLiant x200 mhz systems. They were approximately 18” high, rack mounted. We had our systems “scattered” around the data center, thus we don’t have a picture of them together. They would consume two racks with monitors. Oh and the attractive woman standing in front our the hardware is not a model but LeeAnn Stivers, our Compaq partner on the TerraServer project. My point in showing you the original configuration is to show the incredible rate of change in the hardware business. This system was “state of the art” when we first received it in early Back then, a single system with over 2.4 TB of disk space was hard to imagine in the PC world. The next slide is the current TerraServer “cluster” of processors. <Click or press ENTER NOW> Here is the current TerraServer. Instead of 10 racks of equipment, we only have 5 racks! On the right are two racks, with one half empty, containing the processors. The second rack contains the 3 active database servers, 1 spare database server, and the Storage Area [fiber] Network hubs. The SAN connects to our 13.7 TB of Compaq StorageWorks RAID 10 disk arrays. This configuration is highly available. Instead of using RAID-5 technology as we have in the past, we have opted for mirroring physical volumes. Instead of dual- mirror, we have tripled mirrored each disk. That is one “logical” disk is comprised of three physical disks. We were one of the first to receive 73 GB (72.8 GB actually) hard drives in August These disks spin at 10k RPM and store all our imagery. We also have smaller, 18GB 15k RPM “fast drives”. These drives store all the meta-data about TerraServer imagery, the search system, and the Gazetteer. Note, most “difficult/complicated” queries execute against these drives. On the right rack at the top are eight web servers. Each server is 1 “rack unit”, i.e. “1U” tall, and contains mhz processors, 734kb of RAM, 2 18 GB hard drives, and 2 100mb/1gb network cards. Wow. Since we’ve taken this picture, an additional rack containing 4.5 TB of 73 GB drives has been added on the left. Also, we are replacing our StorageTek 9710 robot, (who’s lower panel can be seen through the bottom of the right most rack in the back), with a Scalar 1000 ADIC robot. While we very happy with StorageTek, the DLT drives and media have been less than satisfactory for us. This year, we have worked with ADIC on testing LTO technology. We have found it superior in raw performance, space usage, and reliability when compared to DLT media. The ADIC Scalar 1000 will be put into the production environment during the summer of 2001. SQL\Inst2 I H M N R S SQL\Inst3 Added GB Disks in Feb 2001 to create 18 TB SAN Spare 4 Compaq ProLiant 8500 Db Servers

Cluster Configuration
Internet Cisco 12000 Internet Router TerraServer SAN Gigibit Ethernet 1 Compaq StorageWorks Database 100-Mbps Ethernet Compaq DL360 (10) Cluster MA8000/HSG80 Controllers (3) 2 Summit 7i Switch (2) Compaq DL360 (6) (Windows 2000 Web Servers) TerraServer.microsoft.com One of the main goals of TerraServer is to demonstrate how to build a highly available and scalable configuration using commercially available, “off the shelf”, commodity hardware running on Microsoft software. In TerraServer’s particular case, we are showing high-end, or “enterprise” customers how we recommend they configure a complete, highly scalable, highly reliable system using off-the-shelf, commodity hardware and software. This particular diagram shows off the key elements of our high availability story. TerraServer is located in Microsoft’s Tukwila Data Center. This is a location near the SEATAC airport south of Seattle. Microsoft’s Internal Technology Group (ITG) provides a high speed link to the internet. We host our own network environment running on Extreme Data Networks gigabit over copper technology. We have on fiber to the internet. This is the “green” line that separates out web servers, “TkTerra#” from the internet. The Extreme network hardware implements a “Virtual IP”, a.k.a. “a VIP”, address, for and The Extreme network equipment routes requests to TkTerra1 thru TkTerra TkTerra7, TkTerra8, TkTerra9, and TkTerra10 service a second VIP for More on that web site later. A second, “private network”, is created by our Extreme networking gear to connect the web servers with the four-node Windows 2000 Data Center cluster. This network is shown in pink. It is a 1 gigabit over copper network that only the web servers and database servers are connected too. The orange lines represent a third network. It is the connection to the Microsoft “corporate LAN” in Redmond Washington. This enables us to load data from our systems located on the Redmond campus “over the wire” to Tukwila. It also lets us remotely log into the database servers with Terminal Server and manage the systems. For security reasons, we cannot directly access the web servers from the corporate Lan. We can reach these systems by first logging into a TerraServer database server. The key attribute of any highly available configuration is no single point of failure. I’ve talked at length about the network so far. Having 10 web servers with 50% wired to one network switch and the other 50% wired to the second network switch ensures that no single web server’s failure or network switch failure can take TerraServer off-line. Yes, we have roughly double the web server capacity that we really use on a normal day. Actually, we have 3-to-1 the “normal” capacity, such that if 50% of our web servers are un-available, we still have excess capacity. On a peak day, which typically are 3-to-1 a normal day, we would run in a degraded mode if we lost all the web servers on a single network switch. The red boxes in the center represent the four Compaq 8500 processors that comprise our database cluster of processors. They too are split between Extreme network switches for network performance balancing purposes and for availability. To the left, the lines depict the redundant paths to the Compaq StorageWorks disk arrays of triple mirrored disks. We use Brocade “Silkworm” 16 port fiber hubs, sold and distributed by Compaq to implement our SAN network. The Silkworm hubs are configured such that there are 3 separate SAN “fabrics”. The 1st fabric shown at the top, implements two redundant paths between each Compaq processor and the 1st fabric, and two separate paths between each of the Compaq StorageWorks disk controllers, which happen to be MA8000 units with HSG80 dual redundant controllers. The 2nd fabric, shown in the middle, also connects the Compaq 8500 processors in a dual redundant manner to the all the Compaq HSG80 storage controllers. There is a separate HBA controller on the Compaq 8500 processors for each fabric. When we added the additional rack of disks, we also added a third Brocade Silkworm switch to fabric #1 and #2. This gave us the required ports we needed to manage all 18 TB, in a fully redundant manner on the two SAN switch. When we added the third Brocade Silkworm to each fabric, we also installed a 3rd fabric managed by another Brocade Silkworm switch, and includes a 3rd HBA installed in each Compaq The 3rd fabric exists to connect our ADIC Scalar 1000 to the Compaq SAN. Thus, we never lose connectivity between a processor and disks due to a hardware failure. Compaq SAN switch by Brocade Communications Internet Extreme Networks Summit 48 Switch ADIC LTO Tape Library 3 Compaq ProLiant 8500 (4) Microsoft Corporate LAN

TerraServer Becomes a Web Service TerraServer. net -> TerraService
TerraServer Becomes a Web Service TerraServer.net -> TerraService.Net Web server is for people. Web Service is for programs The end of screen scraping No faking a URL: pass real parameters. No parsing the answer: data formatted into your address space. Hundreds of users but a specific example: US Department of Agriculture

And now.. 4 slides from the “customer” who built a portal using TerraService

Data Gateway Functional Overview
ITC - Fort Collins, Colorado NCGC - Fort Worth, Texas Customer Orders Data Terra Service Billing Services Soil Data Viewer Navigation Service Rimage CD Service XML XML XML FTP Services ASP Catalog Service Ship Service <<Requests Products>> Package Service Send order info Order Placer validate (dtd) Insert into SQL / GUID to client return est time raise OrderMgr.event Product Catalog Updates Geospatial Data Order Database Data Services Called by anyone Logger XML Request for data rasies to stats svc' Selects from Item Broker Listen for OrderPlacer Raised Event Select sequenced Item Output XML Acknowledges item ready for delivery rasie event : stats.delivery start

Custom End Product Soil Interpretation Map Web Soil Data Viewer
XML Soil Report

Business Rules National Soils Data Geospatial Data Terraserver
Web Server - COM+ Applications ArcIMS Connector WebSDV IMSNavigator Image Retriever Connects to ArcIMS; communication is done through ArcIMS XML (AXL) Retrieves and processes Soils Data from the NASIS relational Database Generates maps (JPGs) using ArcIMS Retrieves imagery from the Microsoft TerraServer Database Server - ESRI Spatial Data Server ESRI Spatial Data Engine Database Server - Microsoft SQL Server Business Rules National Soils Data Geospatial Data Microsoft Terraserver Terraserver

Brief tour of TerraService
Show map service Show some methods See TerraService.NET: An Introduction to Web Services Tom Barclay; Jim Gray; Eric Strand; Steve Ekblad; Jeffrey Richter, MSR TR , pp 13, June 2002

What We Learned You can build and manage a very popular website with relatively little effort (if you do it right and have Tom Barclay) Loading 20 TB takes a lot of energy And you get to do it many times -- automate Tape and tape software are problematic Triplex and snap-shot disks works (we have never had to use it, but..) The internet gives you 2-9’s Servers can run at 4 9’s easily, 5 9’s with effort.

What we are doing now. Building with 3K$ 2TB bricks
4 bricks = 1 backend Triplexing systems Duplexing sites. 4*3*2 = 24k$ for Geoplex Very simple operations model See: “TeraScale SneakerNet: Using Inexpensive Disks for Backup, Archiving, and Data Exchange,” Jim Gray; Wyman Chong; Tom Barclay; Alex Szalay; Jan Vandenberg, pp. 1-8, May 2002

SkyServer SkyServer.SDSS.org
Like the TerraServer, but looking the other way: a picture of ¼ of the universe Pixels + Data Mining Astronomers get about 400 attributes for each “object” Get Spectrograms for 1% of the objects

Why Astronomy Data? It has no commercial value
IRAS 25m It has no commercial value No privacy concerns Can freely share results with others Great for experimenting with algorithms It is real and well documented High-dimensional data (with confidence intervals) Spatial data Temporal data Many different instruments from many different places and many different times Federation is a goal The questions are interesting How did the universe form? There is a lot of it (petabytes) 2MASS 2m DSS Optical IRAS 100m WENSS 92cm NVSS 20cm GB 6cm ROSAT ~keV

Demo of SkyServer Shows standard web server Pixel/image data
Point and click Explore one object Explore sets of objects (data mining)

Virtual Observatory http://www.astro.caltech.edu/nvoconf/ http://www.voforum.org/
Premise: Most data is (or could be online) So, the Internet is the world’s best telescope: It has data on every part of the sky In every measured spectral band: optical, x-ray, radio.. As deep as the best instruments (2 years ago). It is up when you are up. The “seeing” is always great (no working at night, no clouds no moons no..). It’s a smart telescope: links objects and data to literature on them.

Time and Spectral Dimensions The Multiwavelength Crab Nebulae
Crab star 1053 AD X-ray, optical, infrared, and radio views of the nearby Crab Nebula, which is now in a state of chaotic expansion after a supernova explosion first sighted in 1054 A.D. by Chinese Astronomers. Slide courtesy of Robert CalTech.

Data Federations of Web Services
Massive datasets live near their owners: Near the instrument’s software pipeline Near the applications Near data knowledge and curation Super Computer centers become Super Data Centers Each Archive publishes a web service Schema: documents the data Methods on objects (queries) Scientists get “personalized” extracts Uniform access to multiple Archives A common global schema Federation

Grid and Web Services Synergy
I believe the Grid will be many web services share data (computrons are free) IETF standards Provide Naming Authorization / Security / Privacy Distributed Objects Discovery, Definition, Invocation, Object Model Higher level services: workflow, transactions, DB,.. Synergy: commercial Internet & Grid tools

Web Services: The Key? Internet-scale distributed computing
Web SERVER: Given a url + parameters Returns a web page (often dynamic) Web SERVICE: Given a XML document (soap msg) Returns an XML document Tools make this look like an RPC. F(x,y,z) returns (u, v, w) Distributed objects for the web. + naming, discovery, security,.. Internet-scale distributed computing Your program Web Server http Web page Your program Web Service soap Data In your address space object in xml

SkyQuery: a prototype Defining Astronomy Objects and Methods.
Federated 3 Web Services (fermilab/sdss, jhu/first, Cal Tech/dposs) multi-survey cross-match Distributed query optimization (T. Malik, T. Budavari, Alex JHU) My first web service (cutout + annotated SDSS images) online WWT is a great Web Services (.Net) application Federating heterogeneous data sources. Cooperating organizations An Information At Your Fingertips challenge.

Demo of Image Cutout Service
Shows image cutout Show project and debugging project Show hello World Show “theAnswer” method

SkyQuery (http://skyquery.net/)
Distributed Query tool using a set of services Feasibility study, built in 6 weeks from scratch Tanu Malik (JHU CS grad student) Tamas Budavari (JHU astro postdoc) Implemented in C# and .NET Allows queries like: SELECT o.objId, o.r, o.type, t.objId FROM SDSS:PhotoPrimary o, TWOMASS:PhotoPrimary t WHERE XMATCH(o,t)<3.5 AND AREA(181.3,-0.76,6.5) AND o.type=3 and (o.I - t.m_j)>2

SkyNode Basic Web Services
Metadata information about resources Waveband Sky coverage Translation of names to universal dictionary (UCD) Simple search patterns on the resources Cone Search Image mosaic Unit conversions Simple filtering, counting, histogramming On-the-fly recalibrations

Portals: Higher Level Services
Built on Atomic Services Perform more complex tasks Examples Automated resource discovery Cross-identifications Photometric redshifts Outlier detections Visualization facilities Goal: Build custom portals in days from existing building blocks (like today in IRAF or IDL)

Architecture Image cutout SkyNode First Web Page SkyQuery
SkyNode 2Mass SkyNode SDSS

Summary So Far Some real web services deployed today
Easy to build & deploy Services publish data, Portals unify it Tools really work! I’m using C# and foundation classes of VisualStudio, a great! Tool A nice book explaining the ideas: (.Net Framework Essentials, Thai, Lam isbn )

Possible Relevance to You
This web service stuff is REAL If you have a class, It is a way to publish data: Internet Intranet It is a way to find data data comes with schema no more screen scraping/parsing Business model unclear Your ideas go here. Your program Data In your address space Web Service soap object in xml

What We Learned Web services really are a breakthrough.
Data mining worked beautifully. See Data Mining the SDSS SkyServer Database,” J. Gray, D. Slutz, A. Szalay, A. Thakar, P. Kuntz, C. Stoughton, MSR TR , pp1-40, 2002. You can operate a system in Chicago from San Francisco – Terminal Server is wonderful. The Internet gives you 2 9’s of availability TeraScale SneakerNet works well

What we are doing now. Loading more data (next data release)
Preparing for the next generation Building the WWT Web Services for the Virtual Observatory, Alexander S. Szalay, Tamás Budavária, Tanu Malika, Jim Gray, and Ani Thakar, SPIE Astronomy Telescopes and Instruments, August 2002, Waikoloa, Hawaii, Petabyte Scale Data Mining: Dream or Reality?, Alexander S. Szalay; Jim Gray; Jan vandenBerg, SIPE Astronomy Telescopes and Instruments, August 2002, Waikoloa, Hawaii, Online Scientific Data Curation, Publication, and Archiving Jim Gray; Alexander S. Szalay; Ani R. Thakar; Christopher Stoughton; Jan vandenBerg, SPIE Astronomy Telescopes and Instruments, August 2002, Waikoloa, Hawaii,

The Grid Computation Grid: harvest Internet cpus.
Data Grid: Share files Application Grid: Web services Access Grid: teleconferencing

The Microsoft View Web Services will subsume the Grid
The Grid will be data and services not renting cycles OGSA: evolution of Globus Toolkit to Web services concepts and technologies… Lots of encouragement from Microsoft, IBM, Oracle, Sun GGF as forum for discussion

Engagement with Grid Community
Goal: GXA as infrastructure for Grids Working with Globus & GGF Funding work at Argonne National Lab (Globus) Globus Toolkit 3, and CondorG on Windows (we sponsored this) OGSA for .NET (prototyping) Also OGSI.NET at U. VA is very interesting GGF Active membershp HPC .net kit – see Part of .net server scale out development Includes MPI-CH 1.2.4, distributed job scheduler,… Thomas Sterling, Beowulf on Windows, MIT Press 2001

What’s Microsoft Doing
Mostly .NET, W3C standards, web services, … I think SkyQuery is the best web service (grid app) in GriPhyN today. My stuff is grid computing But… Globus (GT3), OGSA, and CondorG ported to Windows (we sponsored it) We have a HPC toolkit: MPI-CH 1.2.4 See for many useful links

I Can Talk About Computing on Demand But… Best to read
Distributed Computing Economics, Jim Gray, MSR-TR , March 2003 The slides that follow are based on that paper.

Distributed Computing Economics
Why is a great idea Why is Napster a great deal? Why is the Computational Grid uneconomic When does computing on demand work? What is the “right” level of abstraction Is the Access Grid the real killer app?

Computing is Free Computers cost 1k$ (if you shop right)
So 1 cpu day == 1$ If you pay the phone bill (and I do) Internet bandwidth costs 50 … 500$/mbps/m (not including routers and management). So 1GB costs 1$ to send and 1$ to receive

Why is Seti@Home a Good Deal?
Send 300 KB for costs 3e-4$ User computes for ½ day: benefit .5e-1$ ROI: 1500:1

Why is Napster a Good Deal?
Send 5 MB costs 5e-3$ ½ a penny per song Both sender and receiver can afford it. Same logic powers web sites (Yahoo!...): 1e-3$/page view advertising revenue 1e-5$/page view cost of serving web page 100:1 ROI

The Cost of Computing: Computers are NOT free!
Capital Cost of a TpcC system is mostly storage and storage software (database) IBM 32 cpu, 512 GB ram 2,500 disks, 43 TB (680, $/tpmc available 11/08/03) A 7.5M$ super-computer Total Data Center Cost: 40% capital &facilities 60% staff (includes app development)

Computing Equivalents 1 $ buys
1 day of cpu time 4 GB ram for a day 1 GB of network bandwidth 1 GB of disk storage 10 M database accesses 10 TB of disk access (sequential) 10 TB of LAN bandwidth (bulk)

Some consequences Beowulf networking is 10,000x cheaper than WAN networking factors of 105 matter. The cheapest and fastest way to move a Terabyte cross country is sneakernet. 24 hours = 4 MB/s 50$ shipping vs 1,000$ wan cost. Sending 10PB CERN data via network is silly: buy disk bricks in Geneva, fill them, ship them – one way. TeraScale SneakerNet: Using Inexpensive Disks for Backup, Archiving, and Data Exchange Jim Gray; Wyman Chong; Tom Barclay; Alex Szalay; Jan vandenBerg Microsoft Technical Report may 2002, MSR-TR

How Do You Move A Terabyte?
Time/TB $/TB Sent $/Mbps Rent $/month Speed Mbps Context 6 years 3,086 1,000 40 0.04 Home phone 5 months 360 117 70 0.6 Home DSL 2 months 2,469 800 1,200 1.5 T1 2 days 2,010 651 28,000 43 T3 14 hours 976 316 49,000 155 OC3 14 minutes 617 200 1,920,000 9600 OC 192 1 day 100 100 Mpbs 2.2 hours 1000 Gbps

Computational Grid Economics
To the extent that computational grid is like or ZetaNet or or… it is a great thing The extent that the computational grid is MPI or data analysis, it fails on economic grounds: move the programs to the data, not the data to the programs. The Internet is NOT the cpu backplane. The USG should not hide this economic fact from the academic/scientific research community.

Computing on Demand Was called outsourcing / service bureaus in my youth. CSC and IBM did it. Payroll is standard outsource. Now we have Hotmail, Salesforce.com, Oracle.com,…. Works for standard apps. Airlines outsource reservations. Banks outsource ATMs. But Amazon, Amex, Wal-Mart, ... Can’t outsource their core competence. So, COD works for commoditized services. It is not a new way of doing things: think payroll.

What’s the right abstraction level for Internet Scale Distributed Computing?
Disk block? No too low. File? No too low. Database? No too low. Application? Yes, of course. Blast search Google search Send/Get Portals that federate astronomy archives ( Web Services (.NET, EJB, OGSA) give this abstraction level.

Access Grid Q: What comes after the telephone? A: eMail?
A: Instant messaging? Both seem retro technology: text & emotons. Access Grid could revolutionize human communication. But, it needs a new idea.

Distributed Computing Economics
Why is a great idea? Why is Napster a great deal? Why is the Computational Grid uneconomic When does computing on demand work? What is the “right” level of abstraction? Is the Access Grid the real killer app? Based on: Distributed Computing Economics, Jim Gray, Microsoft Tech report, March 2003, MSR-TR

Microsoft Large Databases and Grid Computing

Similar presentations

Presentation on theme: "Microsoft Large Databases and Grid Computing"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Microsoft Large Databases and Grid Computing

Similar presentations

Presentation on theme: "Microsoft Large Databases and Grid Computing"— Presentation transcript:

Similar presentations

About project

Feedback