Distributed computing November

Administrivia No lab this week Lab 6 (Visual Basic 2) next week Happy Thanksgiving!!

The big picture We built a computer We built an operating system to control the computer We attached the computer to a network We wrote a compiler to make programming the computer easier Next, we want to be part of a larger system –Shared CPU’s across machines - distributed computing –Shared data across machines – distributed data bases

Distributed computing Key idea –Buying 1000 machines of speed x is significantly cheaper than buying one machine of speed 1000x Key challenge –Making the machines work together to get effective speedup. Communication between machines is the challenge. Approaches –Find problems that can be distributed easily

Distributed problems Problems that can use decentralized computing –Weather prediction Weather in a location is most affected by weather nearby –Movie generation Individual frames can be generated separately –Google search engine 10,000 PC’s. all of them cheap, many of them identical Can answer 170,000,000 queries per day in ½ second or less –Looking for the origin of the universe Can be localized like weather prediction –Looking for extra terrestrials

Distributed computers Scales of distributed computing –Beowulf clusterhundreds of machines All dedicated to the task –NOW (Network of Workstations)thousands of machines Using spare cycles –SETI clustermillions of machines Screen saver situation

Beowulf cluster Machines are dedicated to the network All machines run similar software Problem is divided into pieces –Each piece is assigned to a machine in the cluster Problem pieces should be loosely linked –Computation is faster than communication

NoW (Network of Workstations) Loosely coupled on a LAN Workstations do other things at times When free cycles are available, they’re used Many more machines than Beowulf but less of each machine

NoW at Google Front end 100 machines called Searching machines Retrieving machines Fit machines in a 7’x2’x3’ rack

SETI Telescope at Arecibo collects data Data is processed real time by fast machines But, no one looks for weak signals –Too costly project built to do this

Receive data from Arecibo –35 Gbytes per day by snail mail Break into Work Units –.25 Mbyte each, so 140,000 WU’s per day WU takes 20 hours to process Need 115,000 machines to process a day

Get individual users to download software Machine idle and screen saver runs software –Download WU –Compute –When finished send back result Database at Berkeley reassembles results Progress to date --

Curing cancer at Intel Curing cancer 486 million hours is all machines at Princeton for 6 months

Distributed data bases Data spread across machines in different ways –Web pages E.g. HTTP –MP3 collections E.g. Napster, Gnutella E.g Morpheus, Kazaa, Music City –Auction items E.g. EBay

Client-Server Model Central server Clients store and retrieve data from server –File manager –HTTP:

Napster Model Server is only used to make connections Communication is peer-to-peer (P2P) –Client to client transfer without a real server Sits on top of TCP/IP Uses proprietary protocol

How Napster works Initial registration –name, password, local directory for files, … When client connects (TCP/IP connection) –Client provides list of files it will share –Napster updates its central index of available files When client asks for a file –Napster gives client with list of online clients with file

How Napster works II When client asks for download –Napster asks supplier to accept a request –Napster tells client how to contact supplier (IP address and port) –Client opens port and fetches file from supplier –Supplier and client report progress/completion status to Napster –Download is done according to a protocol Napster server directory is continually updated Client ranks potential servers by bandwidth and ping time

Napster Model Server is only used to make connections Communication is peer-to-peer (P2P) –Client to client transfer without a real server Can we do this without a central server???

Gnutella Gnutella design has no central server Built on 6 DOS/Milgram experiment Every machine is both client and server To connect you need –One IP address

How Gnutella works To connect to the network, –you only need to know the IP address and port of any servant that is already connected. –Your servant does announces your presence to all of the servants it is already connected to, and so on until the message propagates throughout the entire network. –Each of these servants then responds to this message with a bit of information about itself: how many files it is sharing, how many KBs of space they take up, etc. By connecting, you immediately know how much is available on the network to search through.

How Gnutella works - II To search –You send out a search request, it is propagated through the network, and each servant that has matching terms passes back its result set. –Each servant handles the search query in its own way. –To save on bandwidth, a servant does not have to respond to a query if it has no matching items. The servant also has the option of returning only a limited result set.

How Gnutella works - III For file sharing, each servant acts as a miniature HTTP web server. Just like TCP/IP packets, gnutella packets have a TTL (Time To Live). –The TTL starts off at some low number, like 5. –Each time a packet is routed through a servant, the servant lowers the TTL by 1. –Once the TTL hits 0 the packet is no longer forwarded. This helps to keep packets from circling the network forever..

Where does the bandwidth go? Chronicle of Higher Education 9/28/01

Chronicle of Higher Education 9/28/01

MusicCity.com MusicCity.com is a "media" community that includes a powerful web search engine that allows users to search for all types of media. To use MusicCity.com, you need a client -- Morpheus or Kaaza Clieant allows you to –search more accurately –organize and enjoy your personal media files, –share your media files with the rest of the world –…

Morpheus When Morpheus launched on April 21, 2001, it was brand new. 0 users and 0 downloads. By the first week we had as many as 50,000 downloads of Morpheus and a few thousand people using the MusicCity network simultaneously. Now –Over 9 million downloads of Morpheus in 90 days-now averaging over 1 million downloads per week! –Morpheus has become the #1 most popular download on the Internet –Morpheus is one of the most searched for topics on Lycos, Google and others. –Morpheus is the 4th most misspelled word on Google –More than 600,000 people access the MusicCity network simultaneously –More than 70 million files being shared by users-over 300 terabytes of data.

What is KaZaA KaZaA Media Desktop 17 Million downloads of KaZaA so far! and 2nd place on C|Net Download.com. Why don't you join the world's largest online media community right now. 91% o users are recommending it. What are you waiting for download it now -it's free. KaZaA Media Desktop is a full featured peer-to-peer file sharing application. You can search, download, organise and play your media files - audio, video, images and documents with it. It has a powerful search engine where you can search on 'meta data' such as categories, artist etc.

Comparisons FastTrack GnutellaNapster style ServersNo Yes Scalability (online users)virtually unlimited< per sever Protocol Stack APIYesNo User authenticationYesNoYes Self organizing networkYesNo Meta data searchYesNo Intelligent downloadYesNo Interconnected networksYesNo

What about Copyrights –Who owns this stuff? –Why is this not like going to Sam Goody’s and stealing? Breaches of trust –Can download Friends and get pornography –Can download serious viruses PU Bandwidth –This distracts from real work of the university?