ELFms meeting, 2/3/04 German Cancio, 2/3/04 Proxy servers in CERN-CC.

Slides:



Advertisements
Similar presentations
Welcome to Middleware Joseph Amrithraj
Advertisements

W3C Workshop on Web Services Mark Nottingham
2.1 Installing the DNS Server Role Overview of the Domain Name System Role Overview of the DNS Namespace DNS Improvements for Windows Server 2008 Considerations.
1 Content Delivery Networks iBAND2 May 24, 1999 Dave Farber CTO Sandpiper Networks, Inc.
ASIS et le projet EU DataGrid (EDG) Germán Cancio IT/FIO.
Module 8: Concepts of a Network Load Balancing Cluster
Logically Centralized, Physically Distributed Mark Stuart Day Cisco Systems.
Web Servers How do our requests for resources on the Internet get handled? Can they be located anywhere? Global?
Scalability and planning for growth 1WUCM1. Content management issues Structural – Naming (e.g. file, URL) policy – File and directory naming needs: invent/design/borrow.
Hands-On Microsoft Windows Server 2003 Networking Chapter 6 Domain Name System.
Current Status of Fabric Management at CERN, 26/7/2004 Current Status of Fabric Management at CERN CHEP 2004 Interlaken, 27/9/2004 CERN IT/FIO: G. Cancio,
Implementing ISA Server Caching. Caching Overview ISA Server supports caching as a way to improve the speed of retrieving information from the Internet.
1 Web Content Delivery Reading: Section and COS 461: Computer Networks Spring 2007 (MW 1:30-2:50 in Friend 004) Ioannis Avramopoulos Instructor:
DotSlash: Providing Dynamic Scalability to Web Applications Weibin Zhao and Henning Schulzrinne Department of Computer Science, Columbia University More.
Web Cache. Introduction what is web cache?  Introducing proxy servers at certain points in the network that serve in caching Web documents for faster.
Hands-On Microsoft Windows Server 2008 Chapter 8 Managing Windows Server 2008 Network Services.
Christopher M. Pascucci Basic Structural Concepts of.NET Browser – Server Interaction.
CERN DNS Load Balancing Vladimír Bahyl IT-FIO. 26 November 2007WLCG Service Reliability Workshop2 Outline  Problem description and possible solutions.
10/02/2004ELFms meeting1 Linux Virtual Server Miroslav Siket FIO-FS.
1 Content Distribution Networks. 2 Replication Issues Request distribution: how to transparently distribute requests for content among replication servers.
Distributed Data Stores – Facebook Presented by Ben Gooding University of Arkansas – April 21, 2015.
INTRODUCTION TO WEB DATABASE PROGRAMMING
Experiences Deploying Xrootd at RAL Chris Brew (RAL)
Web Servers Web server software is a product that works with the operating system The server computer can run more than one software product such as .
DEMIGUISE STORAGE An Anonymous File Storage System VIJAY KUMAR RAVI PRAGATHI SEGIREDDY COMP 512.
WP4-install task report WP4 workshop Barcelona project conference 5/03 German Cancio.
CP476 Internet Computing Lecture 5 : HTTP, WWW and URL 1 Lecture 5. WWW, HTTP and URL Objective: to review the concepts of WWW to understand how HTTP works.
70-291: MCSE Guide to Managing a Microsoft Windows Server 2003 Network Chapter 7: Domain Name System.
5 Chapter Five Web Servers. 5 Chapter Objectives Learn about the Microsoft Personal Web Server Software Learn how to improve Web site performance Learn.
Chapter 8 Implementing Disaster Recovery and High Availability Hands-On Virtual Computing.
Traffic Control in Apache Jed Reynolds Blog.Bitratchet.com Where in your web site do you need traffic control?
1 Apache. 2 Module - Apache ♦ Overview This module focuses on configuring and customizing Apache web server. Apache is a commonly used Hypertext Transfer.
Large Computer Centres Tony Cass Leader, Fabric Infrastructure & Operations Group Information Technology Department 14 th January and medium.
EDG WP4: installation task LSCCW/HEPiX hands-on, NIKHEF 5/03 German Cancio CERN IT/FIO
2: Application Layer1 Chapter 2 outline r 2.1 Principles of app layer protocols r 2.2 Web and HTTP r 2.3 FTP r 2.4 Electronic Mail r 2.5 DNS r 2.6 Socket.
On the use of Reliable Multicast for Content Distribution Vassilis Chatzigiannakis
Module 11: Implementing ISA Server 2004 Enterprise Edition.
Web Cache Redirection using a Layer-4 switch: Architecture, issues, tradeoffs, and trends Shirish Sathaye Vice-President of Engineering.
1 The new Fabric Management Tools in Production at CERN Thorsten Kleinwort for CERN IT/FIO HEPiX Autumn 2003 Triumf Vancouver Monday, October 20, 2003.
Module 9: Implementing Caching. Overview Caching Overview Configuring General Cache Properties Configuring Cache Rules Configuring Content Download Jobs.
Deployment work at CERN: installation and configuration tasks WP4 workshop Barcelona project conference 5/03 German Cancio CERN IT/FIO.
20-May-2003HEPiX Amsterdam EDG Fabric Management on Solaris G. Cancio Melia, L. Cons, Ph. Defert, I. Reguero, J. Pelegrin, P. Poznanski, C. Ungil Presented.
G. Cancio, L. Cons, Ph. Defert - n°1 October 2002 Software Packages Management System for the EU DataGrid G. Cancio Melia, L. Cons, Ph. Defert. CERN/IT.
SPMA & SWRep: Basic exercises HEPiX hands-on, NIKHEF 5/03 German Cancio
Software Management with Quattor German Cancio CERN/IT.
Implementing ISA Server Caching
CERN DNS Load Balancing VladimírBahylIT-FIO NicholasGarfieldIT-CS.
DotSlash – or how to deal with 15 minutes of fame Weibin Zhao Henning Schulzrinne Columbia University CATT/WICAT Annual Research Review November 14, 2003.
CERN IT Department CH-1211 Genève 23 Switzerland PES 1 Ermis service for DNS Load Balancer configuration HEPiX Fall 2014 Aris Angelogiannopoulos,
Module 4: Design IIS Maintenance and UDDI. Designing Internet Information Services Backup and Recovery Specifying Monitoring requirements Deploying UDDI.
EE 122: Lecture 21 (HyperText Transfer Protocol - HTTP) Ion Stoica Nov 20, 2001 (*)
Maite Barroso - 10/05/01 - n° 1 WP4 PM9 Deliverable Presentation: Interim Installation System Configuration Management Prototype
Cloud Computing Computer Science Innovations, LLC.
By Ruizhe Ma, Avinash Madineni Sidoine Lafleur Kamgang Nov,
Web Server Security: Protecting Your Pages NOAA OAR WebShop 2001 August 2 nd, 2001 Jeremy Warren.
Quattor tutorial Introduction German Cancio, Rafael Garcia, Cal Loomis.
Web Cache. What is Cache? Cache is the storing of data temporarily to improve performance. Cache exist in a variety of areas such as your CPU, Hard Disk.
Free, online, technical courses Take a free online course. Microsoft Virtual Academy.
Fabric Management: Progress and Plans PEB Tim Smith IT/FIO.
Quattor: An administration toolkit for optimizing resources Marco Emilio Poleggi - CERN/INFN-CNAF German Cancio - CERN
1 Chapter 1 INTRODUCTION TO WEB. 2 Objectives In this chapter, you will: Become familiar with the architecture of the World Wide Web Learn about communication.
Presented by Michael Rainey South Mississippi Linux Users Group
Web and Proxy Server.
Understanding Solutions
High Availability Linux (HA Linux)
Germán Cancio CERN IT/FIO LCG workshop, 24/3/04
German Cancio CERN IT .quattro architecture German Cancio CERN IT.
Processes The most important processes used in Web-based systems and their internal organization.
CompTIA Server+ Certification (Exam SK0-004)
EE 122: HyperText Transfer Protocol (HTTP)
Presentation transcript:

ELFms meeting, 2/3/04 German Cancio, 2/3/04 Proxy servers in CERN-CC

ELFms meeting - n° 2 The problem u Service availability on all (quattor managed) CC nodes n Base installation (Anaconda server->client) n Software packages (SWRep->SPMA) n CDB configuration profiles (CDB->CCM/NCM) n User/password information (Regis server->client) u How to offer a reliable, redundant and load balanced access

ELFms meeting - n° 3 Current deployment architecture LXSERV linuxsoft/AIMS DNS-load balanced HTTP DNS load balanced NFS/(HTTP) M M M’ HHH rsync mirror backend frontend

ELFms meeting - n° 4 Problems with current solution u Scalability is limited n Bottlenecks eg. network and switches u Efficiency n Server->server: All contents need to be replicated (disk size, speed) n Server->client: Multiplication of identical transfers u Reliability n Low ratio server/clients (few servers need to cope with 1000’s of clients) n Requires smart and quick load balancing in case a server becomes unavailable (not always there – eg. swrep.cern.ch round robin)

ELFms meeting - n° 5 Proxies u Proxy caching: concepts n Proxy: Intermediary between client/server interactions n Caching proxy: Acts as a transparent store for server objects u Caching proxy types n Transparent: client is completely unaware of proxy (same service endpoints). Requires IP routing modifications (eg. ipchains or switches reconfig) n Forward (or ‘cient-side’): client makes server requests via specified proxy server. Caching typically done on (or near to) the client, to reduce outgoing connections n Reverse (or ‘server-side’): client application talks to front-end server(s), which forwards requests to back-end server(s). u Reverse proxy easiest to set up, as only requires client reconfiguration

ELFms meeting - n° 6 Proxies (II) u Proxy hierarchies n Possible to have chains of proxies, which can be of different type u Protocol types n Protocols eg. HTTP(S), FTP used within stateless services u Proxy implementations: n Apache (via plug-in modules (mod_proxy/mod_rewrite/mod_expire) n Squid u Apache used for tests n + standard, and reliable n clear and flexible configuration n no functionality duplication (can be used as proxy and non-proxy)

ELFms meeting - n° 7 Data caching u Invariant objects n Same object ID (file name) -> same contents n No expiry (excepting object deletion). Lifetime = ∞ n Example: Software packages (RPM’s), Linux base install images (in principle…) u Dynamic objects n The same object may change over time n Expiry lifetime can be < ∞, or even 0 n Requires server revalidation check after expiry n Example: CDB XML profiles, passwd files n Cannot be cached: cgi scripts, ASP pages u Objets on a proxy can be forced to expire independently of their remaining lifetime n Useful for regular garbage cleanup (..and unexpected updates) n Can be done per object type or location

ELFms meeting - n° 8 Proxy architecture LXSERV linuxsoft/AIMS DNS-load balanced HTTP DNS load balanced HTTP M M M’ HHH backend frontend

ELFms meeting - n° 9 Proxy support u quattor client/servers are by design reverse proxy compatible n Stateless, no server-based processing or queries, in order to work with any proxy/replication system n Server location is configurable per client u Anaconda (Linux installer) as well (using HTTP installs) n Per-node KS file contains server location u SPMA enhanced support (since v1.9.1): n Multiple proxy servers – uses the first one available, if none it reverts to the original package SW repository locations n Forward proxies (via delegation to syspackager) n Configurable via CDB u Other applications eg. regis client, GPG keypairs compatible as well (check if location is configurable)

ELFms meeting - n° 10 apache configuration u Apache set up as reverse proxy on LXSERV front end and head nodes. u Most important settings: Load and enable mod_proxy ( libproxy.so ) n Enable cache (/var/cache/httpd), size ~ 6GB n Cache garbage collection runs every 4h n Cache max expiry: 24h n Set father server for each proxy directory (/xml,/swrep, /redhat, /regis,..) n Force complete file download in case of partial requests (eg. rpmt/SPMA asking for rpm header information) u LXSERV master (not a proxy server!) n Set expiry type (via mod_expire: ExpiresActive and ExpiresDefault rules) for dynamic objects (/regis, /encrypted/sensitive-files, /xml). Set to ‘now’, but could be fine- tuned u Enhanced verbosity: Add X-Cache header contents into HTTPD log file (cache operations, HIT and MISS) n Enable server-status pagesserver-status u No changes done to linuxsoft (to be discussed with ADC). No merge SWRep+Linuxsoft needed anylonger.

ELFms meeting - n° 11 Current experience u LXSERV front-end nodes: proxy in production since ~ 2w, no problems experienced n Lxserv01, 02, 03 pointing to lxservb01 n SWRep, XML profiles, regis n Rsync disabled: s speedup of 50% on complete CDB recompilation s SWRep nightly upload speedup (negligible) n Lxservb02 not in proxy mode since not everything is backed up on lxservb01 u Head nodes: tests on lxc1m603 n ccconf DNS alias (XML), lxservb01 and lxserv01 (SWRep), linuxsoft (base installation) n Complete reinstallation of test node only using lxc1m603 as proxy n Lxplus001,lxb0001 using lxc1m603 as proxy since 2w u Server interruptions in the proxy chain: n If cached and invariant -> OK n (dynamic AND expired) OR not cached -> proxy error. Not problematic in case of CCM requests, as local cache exists

ELFms meeting - n° 12 Remaining issues, next steps u Making remaining services head-node aware in terms of cfg n Regis n NCM component for CCM configuration n KSGenerator u Apache NCM configuration component u Possible extension: adding failover capabilities to other services than SPMA … Improve SPMA failover procedure (currently using ping ) n Implement SPMA retry u.. or provide DNS aliases to redirect head node requests u Entering client->head node info into CDB, including failovers n Requires head nodes to be known n Will require adapting CDB schema to accommodate other head node info into CDB u Operator alarms and procedures for head nodes n Single-big-point of failure -> many-small-point-of failures n Future extensions, eg. dynamic experiment software distribution