Towards a Distributed WMS Cache... The Problem: NASA's World Wind offers a case in point that delivering multiple terabytes of free map data to the public.

Slides:

Advertisements

Similar presentations

Peer to Peer and Distributed Hash Tables

Advertisements

Technische Universität Chemnitz Kurt Tutschku Vertretung - Professur Rechner- netze und verteilte Systeme Chord - A Distributed Hash Table Yimei Liao.

1 1 Chord: A scalable Peer-to-peer Lookup Service for Internet Applications Dariotaki Roula

Project in Computer Security Integrating TOR’s attacks into the I2P darknet Chen Avnery Amihay Vinter.

Measurement, Modeling, and Analysis of a Peer-2-Peer File-Sharing Workload Presented For Cs294-4 Fall 2003 By Jon Hess.

SIGMOD 2006University of Alberta1 Approximately Detecting Duplicates for Streaming Data using Stable Bloom Filters Presented by Fan Deng Joint work with.

Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 ECSE-6600: Internet Protocols Informal Quiz #13: P2P and Sensor Networks Shivkumar Kalyanaraman:

Web Caching Schemes1 A Survey of Web Caching Schemes for the Internet Jia Wang.

Internet Networking Spring 2006 Tutorial 12 Web Caching Protocols ICP, CARP.

Topics in Reliable Distributed Systems Lecture 2, Fall Dr. Idit Keidar.

Chord: A Scalable Peer-to-Peer Lookup Protocol for Internet Applications Stoica et al. Presented by Tam Chantem March 30, 2007.

CS 104 Introduction to Computer Science and Graphics Problems

Efficient Content Location Using Interest-based Locality in Peer-to-Peer Systems Presented by: Lin Wing Kai.

1 Spring Semester 2007, Dept. of Computer Science, Technion Internet Networking recitation #13 Web Caching Protocols ICP, CARP.

Exploiting Content Localities for Efficient Search in P2P Systems Lei Guo 1 Song Jiang 2 Li Xiao 3 and Xiaodong Zhang 1 1 College of William and Mary,

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Principles of Reliable Distributed Systems Lecture 2: Peer-to-Peer.

Internet Networking Spring 2002 Tutorial 13 Web Caching Protocols ICP, CARP.

Chord-over-Chord Overlay Sudhindra Rao Ph.D Qualifier Exam Department of ECECS.

Topics in Reliable Distributed Systems Fall Dr. Idit Keidar.

Peer-to-Peer Networks Slides largely adopted from Ion Stoica’s lecture at UCB.

ICDE A Peer-to-peer Framework for Caching Range Queries Ozgur D. Sahin Abhishek Gupta Divyakant Agrawal Amr El Abbadi Department of Computer Science.

1CS 6401 Peer-to-Peer Networks Outline Overview Gnutella Structured Overlays BitTorrent.

BTREE Indices A little context information What’s the purpose of an index? Example of web search engines Queries do not directly search the WWW for data;

Proxy Design Pattern Source: Design Patterns – Elements of Reusable Object- Oriented Software; Gamma, et. al.

Basics of Operating Systems March 4, 2001 Adapted from Operating Systems Lecture Notes, Copyright 1997 Martin C. Rinard.

Hashing it Out in Public Common Failure Modes of DHT-based Anonymity Schemes Andrew Tran, Nicholas Hopper, Yongdae Kim Presenter: Josh Colvin, Fall 2011.

Roger ZimmermannCOMPSAC 2004, September 30 Spatial Data Query Support in Peer-to-Peer Systems Roger Zimmermann, Wei-Shinn Ku, and Haojun Wang Computer.

Introduction Widespread unstructured P2P network

Consistent Hashing: Load Balancing in a Changing World

Network Address Translation (NAT) CS-480b Dick Steflik.

By Shobana Padmanabhan Sep 12, 2007 CSE 473 Class #4: P2P Section 2.6 of textbook (some pictures here are from the book)

Application Layer – Peer-to-peer UIUC CS438: Communication Networks Summer 2014 Fred Douglas Slides: Fred, Kurose&Ross (sometimes edited)

BitTorrent How it applies to networking. What is BitTorrent P2P file sharing protocol Allows users to distribute large amounts of data without placing.

Thesis Proposal Data Consistency in DHTs. Background Peer-to-peer systems have become increasingly popular Lots of P2P applications around us –File sharing,

Smart Reference Proxy Provides additional actions whenever an object is referenced (e.g., counting the number of references to the object) Firewall Proxy.

M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.

Introduction of P2P systems

2: Application Layer1 Chapter 2: Application layer r 2.1 Principles of network applications r 2.2 Web and HTTP r 2.3 FTP r 2.4 Electronic Mail  SMTP,

Bit Torrent A good or a bad?. Common methods of transferring files in the internet: Client-Server Model Peer-to-Peer Network.

Chord: A Scalable Peer-to-peer Lookup Protocol for Internet Applications Xiaozhou Li COS 461: Computer Networks (precept 04/06/12) Princeton University.

1 Distributed Hash Tables (DHTs) Lars Jørgen Lillehovde Jo Grimstad Bang Distributed Hash Tables (DHTs)

Security Michael Foukarakis – 13/12/2004 A Survey of Peer-to-Peer Security Issues Dan S. Wallach Rice University,

Super-peer Network. Motivation: Search in P2P Centralised (Napster) Flooding (Gnutella)  Essentially a breadth-first search using TTLs Distributed Hash.

Kiew-Hong Chua a.k.a Francis Computer Network Presentation 12/5/00.

Views Lesson 7.

MySQL spatial indexing for GIS data in a web 2.0 internet application Brian Toone Samford University

Kaleidoscope – Adding Colors to Kademlia Gil Einziger, Roy Friedman, Eyal Kibbar Computer Science, Technion 1.

Paper Survey of DHT Distributed Hash Table. Usages Directory service  Very little amount of information, such as URI, metadata, … Storage  Data, such.

Flashback: A Peer-to-Peer Web Server for Flash Crowds Presented by Tom Batkiewicz CS 587x Fall ‘07.

CSE378 Intro to caches1 Memory Hierarchy Memory: hierarchy of components of various speeds and capacities Hierarchy driven by cost and performance In early.

1. Outline  Introduction  Different Mechanisms Broadcasting Multicasting Forward Pointers Home-based approach Distributed Hash Tables Hierarchical approaches.

Plethora: Infrastructure and System Design. Introduction Peer-to-Peer (P2P) networks: –Self-organizing distributed systems –Nodes receive and provide.

Peer to Peer Network Design Discovery and Routing algorithms

Dynamo: Amazon’s Highly Available Key-value Store DAAS – Database as a service.

CS Spring 2014 CS 414 – Multimedia Systems Design Lecture 37 – Introduction to P2P (Part 1) Klara Nahrstedt.

INTERNET TECHNOLOGIES Week 10 Peer to Peer Paradigm 1.

P2P Search COP P2P Search Techniques Centralized P2P systems  e.g. Napster, Decentralized & unstructured P2P systems  e.g. Gnutella.

NGMAST Mobile DHT Energy1 Optimizing Energy Consumption of Mobile Nodes in Heterogeneous Kademlia-based Distributed Hash Tables Imre Kelényi Budapest.

Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications * CS587x Lecture Department of Computer Science Iowa State University *I. Stoica,

CASCADE: AN ATTACK-RESISTANT DHT WITH MINIMAL HARD STATE

Swapping Segmented paging allows us to have non-contiguous allocations

Internet Networking recitation #12

Ordering Healthy Food Online

Introduction to Execution Plans

PROSE CS 218 Fall 2017.

Peer to Peer Information Retrieval

5.2 FLAT NAMING.

Introduction to Execution Plans

Introduction to Execution Plans

Presentation transcript:

Towards a Distributed WMS Cache... The Problem: NASA's World Wind offers a case in point that delivering multiple terabytes of free map data to the public on demand is... hard! Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... More specifically: No one really has that much bandwidth to give away at once! (except the Internet Archive, and they're “a great archive but a lousy ISP”) Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... But everyone has a little bandwidth to give away from time to time! Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... The Solution: Distributed, Peer-to-Peer, Caching WMS Proxies! Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... Distributed Hash Tables offer an example of robust peer-to-peer metadata search » Like ordinary hash tables, DHTs offer O(log n) search » DHTs can also be tuned to provide redundant storage » Kademlia is a good example (go look it up) » q.v. the DHT metadata search in eDonkey, BitTorrent, et al. Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... The Twist: Instead of a standard text-centric hash, we'll use the bounding box of our AOI to locate imagery tiles within the network. This is starting to sound more like a Distributed R-Tree than a traditional DHT... But no matter – it can be done! Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... It might work something like this... Every peer would “occupy” a random location in geographic space and would query the network on joining, to announce its presence, and find its spatial neighbors... Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... Before requesting a tile from a real WMS, a peer would traverse the DiRT looking for the k “closest” peers to the AOI, and ask them for any tiles they might have... Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... If they don't have all the tiles we need, then we fetch the missing ones from the actual WMS – e.g. JPL's OnEarth – and then store those tiles with the aforementioned k closest peers... Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... The next time someone wants that tile, it's already stored in our network multiple times – and the source WMS doesn't have to be involved! In fact, once the distributed cache is fully populated, the original WMS can fall right off the 'Net, and we don't even have to care! Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... or... you know... something like that... (please excuse the vigorous handwaving) Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... World Wind tiles tend to run to less than 60 Kb, so we can use UDP for transport! Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005 » UDP is fast » UDP is asynchronous » it can get around NAT firewalls » it will work

Towards a Distributed WMS Cache... Since each peer in the distributed tile cache would also be a WMS proxy, the cache could be transparently accessible from any WMS- aware application and the end user might never have to know (or care) Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache... So why should we need JPL or Terraserver or Google or even the Internet Archive to serve us Landsat imagery – or any other kind of imagery – when, together, we have enough disk space and enough bandwidth between us to do it ourselves? Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005

Towards a Distributed WMS Cache now we just need to do it! (thank you) Towards a Distributed WMS Cache: A Call to Action MapServer User Meeting 2005