1. Gray @ IBM STL 1 April 1999 Scaleable Computing Jim Gray Microsoft Research Gray@Microsoft.com http://research.Microsoft.com/~Gray Gray@Microsoft.com Outline –TerraServer (Barclay, Slutz, Gray) –TPC –Sort

2. Gray @ IBM STL 1 April 1999 Terra-Server Requirements BIG —1 TB of data including catalog, temporary space, etc. PUBLIC — available on the world wide web INTERESTING — to a wide audience ACCESSIBLE — using standard browsers (IE, Netscape) REAL — a LOB application (users can buy imagery) FREE —cannot require NDA or money to a user to access FAST — usable on low-speed (56kbps) and high speeds(T-1+) EASY — small group to develop, deploy, maintain the app

3. Gray @ IBM STL 1 April 1999 Image Data USGS “DOQ” 4 TB 6TB Coming DRG 50,000 Topo Maps adding now Spin-2 1 TB WorldWide New Data Coming

4. Gray @ IBM STL 1 April 1999 Database & App UI Coverage: Range from 70ºN to 70ºS today: 35% U.S., 1% outside U.S. Source Imagery: –4 TB 1sq meter/pixel Aerial (USGS - 60,000 46Mb B&W- 151Mb Color IR files) –1 TB 1.56 meter/pixel Satellite (Spin-2 - 2400 300 Mb B&W) Display Imagery: 200x200 pixel images, subsample to build image pyramid Nav Tools: –1.5 m place names –“Click-on” Coverage map –Expedia & Virtual Globe map 1.6x 1.6 km “city view”.8 x.8 km 8m thumbnail,4 x,4 km browse 200x200 m tile Concept: User navigates an ‘almost seamless’ image of earth

5. Gray @ IBM STL 1 April 1999 TerraServer Grid System 180º 0º -90º 90º 0º -180º 0 17280 0 Image pyramid based on 200x200 pixel tiles Resolutions from 1 meter up to 1 km Resolutions down to millimeter Images sub-sampled to form image pyramid Image tiles stitched together on web page ImgCutter to Tile large images ODBC-BCP Off-Line Load Program Active Server Page Load Mgmt Sys. ODBC On-Line Load Program Legato Networker integrated into Load Mgmt System StorageTek 9714 maintains full-resolution image archive Code App/DB Design

6. Gray @ IBM STL 1 April 1999 IE 3…5 Netscape 3…4 HTML Java Viewer The Internet Web Client Image Delivery Application SQL Server SPIN-2/USGS Store Active Server Pages Microsoft Site Serve EE 3.0 Image Commerce Site(s) SQL Server 7.0 Terra-Server DB Terra-Server Stored Procedures Internet Information Server 4.0 Terra-Server Active Server Pages Active Data Object ODBC Terra-Server Web Site 19 24 13 39 (14 Img) (8 Place) Software Architecture

7. Gray @ IBM STL 1 April 1999 Lookup by UGrid or ZGrid ID plus resolution Lookups are fast. Indices are in DRAM (auto-magically by SQL) SQL manages all the tiles and indices Images are brought in on demand Gazetteer Index on image, place, type image, state, type image, state, country, type image, place, state, type image, place, country, type all lookups are fast Logical Schema Country Name State Name Place Name PlaceType Feature Type Where Am I Img Meta Tile Meta Jump Img Browse Img Tile Img Theme Meta Information Spin Frame Meta Thumb Img Image Data & Meta Data

8. Gray @ IBM STL 1 April 1999 TerraServer Administrator Web Site Accessible by Microsoft, SPIN-2, and USGS Web browser forms to: –Edit Famous Places list –Modify Image Status fields –Define new TerraServer Administrators

9. Gray @ IBM STL 1 April 1999 USGS Store –Built by USGS –MSCS V2.0 Based –Standard Shopping Basket approach –Purchase Digital Ortho Quads used by MS to build TerraServer –Pricing subject to quantity –Image you were viewing given away for free (public domain data) Spin-2 Store –Microsoft SP built –MSCS V2.0 Based –Buy Small, Medium, Large Digital image –Can get Photographic print thru SPIN-2 relationship with Kodak –Digital images are “sized” to make photographic prints look good Microsoft does not collect or share in the revenues generated by TerraServer image sales! Electronic Commerce

10. Gray @ IBM STL 1 April 1999 Backup and Recovery –Using Legato Networker integrated with SQL Backup/Restore Utility –Fast, incremental, differential, online Restore –Fast, incremental (file oriented), not online. SQL Server Enterprise Manager –DBA Maintenance –SQL Performance Monitor Management & Maintenance

11. Gray @ IBM STL 1 April 1999 TerraServer Backup Factoids Offline Backup

12. Gray @ IBM STL 1 April 1999 Site Configuration 9710 TimberWolf Enterprise Storage Array 9 HSZ70 Ultra-SCSI Dual redundant Controllers 324 9 GB Seagate Disks Compaq 5500 4x200mhz Web Servers To the Web Compaq 5500 4x200mhz Web Servers Compaq 5500 4x200mhz Web Servers Compaq 5500 4x200mhz Web Servers Compaq 5500 4x200mhz Web Servers Compaq 5500 4x200mhz Web Servers

13. Gray @ IBM STL 1 April 1999 The Microsoft TerraServer Hardware Compaq AlphaServer 8400Compaq AlphaServer 8400 8x400Mhz Alpha cpus8x400Mhz Alpha cpus 10 GB DRAM10 GB DRAM 324 9.2 GB StorageWorks Disks324 9.2 GB StorageWorks Disks –3 TB raw, 2.4 TB of RAID5 STK 9710 tape robot (4 TB)STK 9710 tape robot (4 TB) WindowsNT 4 EE, SQL Server 7.0WindowsNT 4 EE, SQL Server 7.0

14. Gray @ IBM STL 1 April 1999 Use StorageWorks to form 28 RAID5 sets Each raid set has 11 disks (16 spare drives) Use NTFS to form 4 595GB NT volumes Each striped over 7 Raid sets on 7 controllers Create 26 20,000MB files on F:, 27 on G: DB is File Group of 53 files (1.011TB) F: G: H: I: File System Config

15. Gray @ IBM STL 1 April 1999 TerraServer: Lots of Web Hits Today: –1.7 billion web hits –1 TB, largest SQL DB on the Web –100 qps average, 1,000 Qps peak –1.5 B SQL queries so far SummaryTotal Max Unique Users17 M 150 k Sessions24 M 172 k Hits 1.7 B 29 M Page Views274 M 1.1 M 6.6 M DB Queries 1.5 B 18 M Image Xfers 1.3 B Average 69 k 94 k 6.8 M 5.8 M 5.0 M 15 M As of Feb 28, 1999

16. Gray @ IBM STL 1 April 1999 How Images are Found Coverage Map 19% Expedia Map 22% Name Search 40% Famous Places 18% Geo Coordinate 1%

17. Gray @ IBM STL 1 April 1999 SQL 7 TerraServer Availability Operating for 9 months : 6400 hrs Unscheduled outage: 36.5 minutes: 99.9905% scheduled up Scheduled outage: 60 minutes Availability: 99.96% overall up No NT failures (ever) One SQL7 Beta2 bug No failures in July, Aug, Oct, Dec, Jan, Feb, Mar

18. Gray @ IBM STL 1 April 1999 1 Degree Latitude DOQQ Origin Point 1 Degree Longitude USGS DOQ Editing Process 1 Quadrangle (7.5’ x 7.5’) 1 “DOQ QUAD” DOQQ Photo (3.75’ x 3.75’) 48MB - 150 MB DOQQ cut into 1800m x 1200m TIF rectangles Merge Pixels 200x200 Subsampled Jpeg UTM Projection Summary 1800x1200 images Seamless mosaic 8 level pyramid Jpeg format 2:1 4:1 8:1 16:1 32:1 64:1 128:1

19. Gray @ IBM STL 1 April 1999 USGS Grid System USGS DOQ ortho-rectified to UTM projection World divided into 60 “zones” Zone logically is 1,000,000 meters by 10,000,000 meters USGS tiles are uniform size 1800 X 1200 meters relative to zone meridian USGS X = int((Easting + 400) / 1800) + (Zone * 512) USGS Y = int(Northing / 1200)

20. Gray @ IBM STL 1 April 1999 USGS DOQQ Data File Meridian 500,000 1,000,000 (max) 0 (min) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

21. Gray @ IBM STL 1 April 1999 Spin-2 Image Editing Break into 200x200 m grid Image aligned to left corner of grid system Non-image squares (all white) are discarded Cut Images are extracted SubSample Jump Tiles are cut scrambled output Jpeg 200x200 2:1 4:1 8:1 16:1 32:1 64:1 128:1

22. Gray @ IBM STL 1 April 1999 Spin-2 Meta Data  File name (of image)  City 1  State 1  Country  Number of Rows  Number of Columns  Shooting Height  Height of Sun  Date of survey (mm/dd/yyyy)  Time of survey (GMT) (hr:mn:ss)  Upper Left Latitude  Upper Left Longitude  Lower Right Latitude  Lower Right Longitude  Camera System 1  Pixel size 1  Copyright 1 1 Field is not required, if not present, then a blank field is present

23. Gray @ IBM STL 1 April 1999 Image Load and Update ODBC Tx TerraLoader ODBC TX TerraServer SQL DBMS DLT Tape “tar” Metadata Load DB Active Server Pages Cut & Load Scheduling System Staging Disk JPEG tiles Image Cutter Merge ODBC Tx Dither Image Pyramid From base

24. Gray @ IBM STL 1 April 1999 Things we did right... Simple X, Y Z-Grid navigation system Used ImgStatus to control logical “presence” of the image in the app “Stitching tiles together” from multiple input images to form seamless mosaic Offering two forms of seamless -- time based (SPIN-2) and theme based (DOQ) Using a fixed tile size Can dynamically load data into tables while viewing application is active

25. Gray @ IBM STL 1 April 1999 TS 3: Things are changing... Square Tiles, power of 2 size (200x200) Power of 2 zoom levels (2:1, 4:1, 8:1, etc.) so uniform tile size on each zoom (variable ground size per tile) Indexing system independent of tile size Digital Raster Graphics (Topo maps) Layered Maps (Topo merge with DOQ) Integrate with other applications and services Later: –Digital Elevation Models (DEMs) –Other foreign data sources (EU, etc.)

26. Gray @ IBM STL 1 April 1999 Future Plans Add additional data sets –Digital Raster Graphics (Topo maps) –Digital Elevation Models (DEMs) –Other foreign data sources (EU, etc.) Layered Maps –Allow Topo maps to overlay DOQ, DEM, etc. data Integrate with other applications and services

27. Gray @ IBM STL 1 April 1999 Thank You! SPIN-2 Tom Barclay did most of this app, Slutz and Gray helped.

28. Gray @ IBM STL 1 April 1999 Outline TerraServer TPC Sort

29. Scaleability Scale Up and Scale Out SMP Super Server Departmental Server Personal System Grow Up with SMP 4xP6 is now standard Grow Out with Cluster Cluster has inexpensive parts Cluster of PCs

30. Gray @ IBM STL 1 April 1999 Billions Of Clients Need Millions Of Servers Mobile clients Fixed clients Server Superserver Clients Servers  All clients networked to servers  May be nomadic or on-demand  Fast clients want faster servers  Servers provide  Shared Data  Control  Coordination  Communication Trillions Billions

31. Gray @ IBM STL 1 April 1999 Thesis Many little beat few big  Smoking, hairy golf ball  How to connect the many little parts?  How to program the many little parts?  Fault tolerance & Management? $1 million $100 K $10 K Mainframe Mini Micro Nano 14" 9" 5.25" 3.5" 2.5" 1.8" 1 M SPECmarks, 1TFLOP 10 6 clocks to bulk ram Event-horizon on chip VM reincarnated Multi-program cache, On-Chip SMP 10 microsecond ram 10 millisecond disc 10 second tape archive 10 nano-second ram Pico Processor 10 pico-second ram 1 MM 3 100 TB 1 TB 10 GB 1 MB 100 MB

32. Gray @ IBM STL 1 April 1999 1988: IBM DB2 + CICS Mainframe 65 tps IBM 4391 Simulated network of 800 clients 2m$ computer Staff of 6 to do benchmark 2 x 3725 network controllers 16 GB disk farm 4 x 8 x.5GB Refrigerator-sized CPU

33. Gray @ IBM STL 1 April 1999 1987: Tandem Mini @ 256 tps 14 M$ computer (Tandem) A dozen people (1.8M$/y) False floor, 2 rooms of machines Simulate 25,600 clients 32 node processor array 40 GB disk array (80 drives) OS expert Network expert DB expert Performance expert Hardware experts Admin expert Auditor Manager

34. Gray @ IBM STL 1 April 1999 1997: 9 years later 1 Person and 1 box = 1250 tps 1 Breadbox ~ 5x 1987 machine room 23 GB is hand-held One person does all the work Cost/tps is 100,000x less 5 micro dollars per transaction 4x200 Mhz cpu 1/2 GB DRAM 12 x 4GB disk Hardware expert OS expert Net expert DB expert App expert 3 x7 x 4GB disk arrays

35. Gray @ IBM STL 1 April 1999 mainframe mini micro time price What Happened? Where did the 100,000x come from? Moore’s law: 100X (at most) Software improvements: 10X (at most) Commodity Pricing: 100X (at least) Total 100,000X 100x from commodity –(DBMS was 100K$ to start: now 1k$ to start –IBM 390 MIPS is 7.5K$ today –Intel MIPS is 10$ today –Commodity disk is 50$/GB vs 1,500$/GB –...

36. Gray @ IBM STL 1 April 1999 I think there is a world market for maybe five computers. “ ” Thomas Watson Senior, Chairman of IBM, 1943

37. Gray @ IBM STL 1 April 1999 SGI O2KUE10KDELL 6350Cray T3EIBM SP2PoPC per sqft cpus2.14.77.04.75.013.3 specint29.060.5132.779.372.3253.3 ram4.14.77.00.65.06.8gb disks1.30.55.20.02.513.3 Standard package, full height, fully populated, 3.5” disks HP, DELL, Compaq are trading places wrt rack mount lead PoPC – Celeron NLX shoeboxes – 1000 nodes in 48 (24x2) sq ft. $650K from Arrow (3yr warrantee!) on chip at speed L2 Web & server farms, server consolidation / sqft http://www.exodus.com (charges by mbps times sqft)

38. Gray @ IBM STL 1 April 1999 Peta Scale Computing Peta scale w/ traditional balance 20002010 1 PIPS processors 10 15 ips 10 6 cpus @10 9 ips 10 4 cpus @10 11 ips 10 PB of DRAM10 8 chips @10 7 bytes 10 6 chips @10 9 bytes 10 PBps memory bandwidth 1 PBps IO bandwidth10 8 disks 10 7 Bps 10 7 disks 10 8 Bps 100 PB of disk storage 10 5 disks 10 10 B10 3 disks 10 12 B 10 EB of tape storage 10 7 tapes 10 10 B10 5 tapes 10 12 B 10x every 5 years, 100x every 10 (1000x in 20 if SC) Except --- memory & IO bandwidth

39. Gray @ IBM STL 1 April 1999 Microsoft.com: ~150x4 nodes: a crowd (3)

40. Gray @ IBM STL 1 April 1999 HotMail: ~400 Computers Crowd

41. Gray @ IBM STL 1 April 1999 DB Clusters (crowds) 16-node Cluster –64 cpus –2 TB of disk –Decision support 45-node Cluster –140 cpus –14 GB DRAM –4 TB RAID disk –OLTP (Debit Credit) 1 B tpd (14 k tps)

42. Gray @ IBM STL 1 April 1999 Windows NT Versus UNIX Best Results on an SMP: SemiLog plot shows 3x (2 year) lead by UNIX All these numbers are off-scale huge (80,000 active users?)

43. Gray @ IBM STL 1 April 1999 TPC C Improvements (MS SQL) 250%/year on Price, 100%/year performance bottleneck is 3GB address space 40% hardware, 100% software, 100% PC Technology

44. Gray @ IBM STL 1 April 1999 UNIX (dis) Economy Of Scale

45. Gray @ IBM STL 1 April 1999 Disks are expensive Oracle is expensive

46. Gray @ IBM STL 1 April 1999 TPC-D My data is stale, not updated it. Materialized views have “spoiled” the benchmark. Overhaul is in progress (H & R) versions: –Ad Hoc (no materialized views) and –Reporting (cubes and pre-computed views allowed) Oracle did Q5 in 71 sec (1M$ bet that SQL was >> 7,100 seconds (1.5 hrs). IBM and Microsoft have done Q5 in 1 sec!

47. Gray @ IBM STL 1 April 1999 Outline TerraServer TPC Sort

48. Gray @ IBM STL 1 April 1999 A Short History of Sort April Fools 1995: Datamation Sort –Sort 1M 100 B records –An IO benchmark: 15-min to 1 hr! 1993:{Minute | Penny}x{Daytona | Indy} 1998: TeraByte Sort Web site: http://research.Microsoft.com/barc/SortBenchmark/

49. Gray @ IBM STL 1 April 1999 Penny Sort Ground Rules http://research.microsoft.com/barc/SortBenchmark How much can you sort for a penny. –Hardware and Software cost –Depreciated over 3 years –1M$ system gets about 1 second, –1K$ system gets about 1,000 seconds. – Time (seconds) = SystemPrice ($) / 946,080 Input and output are disk resident Input is –100-byte records (random data) –key is first 10 bytes. Must create output file and fill with sorted version of input file. Daytona (product) and Indy (special) categories

50. Gray @ IBM STL 1 April 1999 Bottleneck Analysis Drawn to linear scale Theoretical Bus Bandwidth 422MBps = 66 Mhz x 64 bits Memory Read/Write ~150 MBps MemCopy ~50 MBps Disk R/W ~15MBps

51. Gray @ IBM STL 1 April 1999 Bottleneck Analysis NTFS Read/Write 18 Ultra 3 SCSI on 4 strings (2x4 and 2x5) 3 PCI 64 ~ 155 MBps Unbuffered read (175 raw) ~ 95 MBps Unbuffered write Recently: SQL Server on Xeon: 190MBps scan. Good, but 10x down from S390/SGI/UE10k Memory Read/Write ~250 MBps PCI ~110 MBps Adapter ~70 MBps PCI Adapter 155 MBps

52. Gray @ IBM STL 1 April 1999 PennySort Hardware –266 Mhz Intel PPro –64 MB SDRAM (10ns) –Dual Fujitsu DMA 3.2GB EIDE disks Software –NT workstation 4.3 –NT 5 sort Performance –sort 15 M 100-byte records (~1.5 GB) –Disk to disk –elapsed time 820 sec cpu time = 404 sec

53. Gray @ IBM STL 1 April 1999 How Good is NT5 Sort? CPU and IO not overlapped. System should be able to sort 2x more RAM has spare capacity Disk is space saturated (1.5GB in, 1.5GB out on 3GB drive.) Need an extra 3GB drive or a >6GB drive CPU Disk Fixedram

54. Gray @ IBM STL 1 April 1999 Recent Results NOW Sort: 9 GB on a cluster of 100 UltraSparcs in 1 minute MilleniumSort: 16x Dell NT cluster: 100 MB in 1.18 Sec (Datamation) Tandem/Sandia Sort: 68 CPU ServerNet 1 TB in 47 minutes IBM SPsort 408 nodes, 1952 cpu 2168 disks 17.6 minutes = 1057sec (all for 1/3 of 94M$, slice price is 64k$ for 4cpu, 2GB ram, 6 9GB disks + interconnect

55. Gray @ IBM STL 1 April 1999 Sandia/Compaq/ServerNet/NT Sort Sort 1.1 Terabyte (13 Billion records) in 47 minutes 68 nodes (dual 450 Mhz processors) 543 disks, 1.5 M$ 1.2 GB ps network rap (2.8 GBps pap) 5.2 GB ps of disk rap (same as pap) (rap=real application performance, pap= peak advertised performance )

56. Gray @ IBM STL 1 April 1999 SP sort 2 – 4 GBps!

57. Gray @ IBM STL 1 April 1999 Progress on Sorting Speedup comes from Moore’s law 40%/year Processor/Disk/Network arrays: 60%/year (this is a software speedup).

58. Gray @ IBM STL 1 April 1999 Data Gravity Processing Moves to Transducers Move Processing to data sources Move to where the power (and sheet metal) is Processor in –Modem –Display –Microphones (speech recognition) & cameras (vision) –Storage: Data storage and analysis System is “distributed” (a cluster/mob)

59. Gray @ IBM STL 1 April 1999 Gbps SAN: 110 MBps SAN: Standard Interconnect PCI: 70 MBps UW Scsi: 40 MBps FW scsi: 20 MBps scsi: 5 MBps LAN faster than memory bus? 1 G B ps links in lab. 100$ port cost soon Port is computer Winsock: 110 MBps (10% cpu utilization at each end) RIP FDDI RIP ATM RIP SCI RIP SCSI RIP FC RIP ?

60. Gray @ IBM STL 1 April 1999 Disk = Node has magnetic storage (100 GB?) has processor & DRAM has SAN attachment has execution environment OS Kernel SAN driverDisk driver File SystemRPC,... ServicesDBMS Applications

61. Gray @ IBM STL 1 April 1999 Outline TerraServer TPC Sort

62. Gray @ IBM STL 1 April 1999 end

63. Kinds of Parallel Execution Pipeline Partition outputs split N ways inputs merge M ways Any Sequential Program Any Sequential Program Sequential Any Sequential Program Any Sequential Program

64. Gray @ IBM STL 1 April 1999 Why Parallel Access To Data? At 10 MB/s 1.2 days to scan 1,000 x parallel 100 second SCAN. Parallelism: divide a big problem into many smaller ones to be solved in parallel. BANDWIDTH

65. Gray @ IBM STL 1 April 1999 Why are Relational Operators Successful for Parallelism? Relational data model uniform operators on uniform data stream Closed under composition Each operator consumes 1 or 2 input streams Each stream is a uniform collection of data Sequential data in and out: Pure dataflow partitioning some operators (e.g. aggregates, non-equi-join, sort,..) requires innovation AUTOMATIC PARALLELISM

66. Gray @ IBM STL 1 April 1999 Database Systems “Hide” Parallelism Automate system management via tools –data placement –data organization (indexing) –periodic tasks (dump / recover / reorganize) Automatic fault tolerance –duplex & failover –transactions Automatic parallelism –among transactions (locking) –within a transaction (parallel execution)

67. Gray @ IBM STL 1 April 1999 SQL a Non-Procedural Programming Language SQL: functional programming language describes answer set. Optimizer picks best execution plan –Picks data flow web (pipeline), –degree of parallelism (partitioning) –other execution parameters (process placement, memory,...) GUI Schema Plan Monitor Optimizer Execution Planning Rivers Executors

68. Gray @ IBM STL 1 April 1999 Partitioned Execution Spreads computation and IO among processors Partitioned data gives NATURAL parallelism

69. Gray @ IBM STL 1 April 1999 N x M way Parallelism N inputs, M outputs, no bottlenecks. Partitioned Data Partitioned and Pipelined Data Flows

70. Gray @ IBM STL 1 April 1999 Automatic Parallel Object Relational DB Select image from landsat where date between 1970 and 1990 and overlaps(location, :Rockies) and snow_cover(image) >.7; Temporal Spatial Image datelocimage Landsat 1/2/72.... 4/8/95 33N 120W. 34N 120W Assign one process per processor/disk: find images with right data & location analyze image, if 70% snow, return it image Answer date, location, & image tests

71. Gray @ IBM STL 1 April 1999 Data Rivers: Split + Merge Streams Producers add records to the river, Consumers consume records from the river Purely sequential programming. River does flow control and buffering does partition and merge of data records River = Split/Merge in Gamma = Exchange operator in Volcano /SQL Server. River M Consumers N producers N X M Data Streams

72. Gray @ IBM STL 1 April 1999 Generalization: Object-oriented Rivers Rivers transport sub-class of record-set (= stream of objects) –record type and partitioning are part of subclass Node transformers are data pumps –an object with river inputs and outputs –do late-binding to record-type Programming becomes data flow programming –specify the pipelines Compiler/Scheduler does data partitioning and “transformer” placement

73. Gray @ IBM STL 1 April 1999 NT Cluster Sort as a Prototype Using –data generation and –sort as a prototypical app “Hello world” of distributed processing goal: easy install & execute

74. Gray @ IBM STL 1 April 1999 Remote Install RegConnectRegistry() RegCreateKeyEx() Add Registry entry to each remote node.

75. Gray @ IBM STL 1 April 1999 Cluster StartupExecution MULT_QI COSERVERINFO Setup : MULTI_QI struct COSERVERINFO struct CoCreateInstanceEx() Retrieve remote object handle from MULTI_QI struct Invoke methods as usual HANDLE Sort()

76. Gray @ IBM STL 1 April 1999 Cluster Sort Conceptual Model Multiple Data Sources Multiple Data Destinations Multiple nodes Disks -> Sockets -> Disk -> Disk B AAA BBB CCC A AAA BBB CCC C AAA BBB CCC BBB AAA CCC BBB AAA CCC

77. Gray @ IBM STL 1 April 1999 How Do They Talk to Each Other? Each node has an OS Each node has local resources: A federation. Each node does not completely trust the others. Nodes use RPC to talk to each other –CORBA? DCOM? IIOP? RMI? –One or all of the above. Huge leverage in high-level interfaces. Same old distributed system story. Wire(s) h streams datagrams RPC? Applications VIAL/VIPL streams datagrams RPC? Applications