Dan Bradley Condor Project CS and Physics Departments University of Wisconsin-Madison CCB The Condor Connection Broker.

Slides:

Advertisements

Similar presentations

Todd Tannenbaum Condor Team GCB Tutorial OGF 2007.

Advertisements

Jaime Frey Computer Sciences Department University of Wisconsin-Madison OGF 19 Condor Software Forum Routing.

Current methods for negotiating firewalls for the Condor ® system Bruce Beckles (University of Cambridge Computing Service) Se-Chang Son (University of.

Building a secure Condor ® pool in an open academic environment Bruce Beckles University of Cambridge Computing Service.

The Future of HTCondor's Networking or: How I Learned to Stop Worrying and Love IPv6 Alan De Smet

Dan Bradley Computer Sciences Department University of Wisconsin-Madison Schedd On The Side.

A Computation Management Agent for Multi-Institutional Grids

MCTS Guide to Microsoft Windows Server 2008 Network Infrastructure Configuration Chapter 8 Introduction to Printers in a Windows Server 2008 Network.

Jaeyoung Yoon Computer Sciences Department University of Wisconsin-Madison Virtual Machine Universe in.

Jaeyoung Yoon Computer Sciences Department University of Wisconsin-Madison Virtual Machines in Condor.

Zach Miller Computer Sciences Department University of Wisconsin-Madison What’s New in Condor.

Intranet, Extranet, Firewall. Intranet and Extranet.

Sonny (Sechang) Son Computer Sciences Department University of Wisconsin-Madison Dealing with Internet Connectivity in Distributed.

Hao Wang Computer Sciences Department University of Wisconsin-Madison Security in Condor.

BaBar MC production BaBar MC production software VU (Amsterdam University) A lot of computers EDG testbed (NIKHEF) Jobs Results The simple question:

Module 12: Routing Fundamentals. Routing Overview Configuring Routing and Remote Access as a Router Quality of Service.

1 Chapter 12: VPN Connectivity in Remote Access Designs Designs That Include VPN Remote Access Essential VPN Remote Access Design Concepts Data Protection.

Campus Grids Report OSG Area Coordinator’s Meeting Dec 15, 2010 Dan Fraser (Derek Weitzel, Brian Bockelman)

VIRTUAL PRIVATE NETWORK By: Tammy Be Khoa Kieu Stephen Tran Michael Tse.

Condor Project Computer Sciences Department University of Wisconsin-Madison What’s new in Condor? What’s c Condor Week 2010.

Experiences with a HTCondor pool: Prepare to be underwhelmed C. J. Lingwood, Lancaster University CCB (The Condor Connection Broker) – Dan Bradley

Module 11: Implementing ISA Server 2004 Enterprise Edition.

1 The Roadmap to New Releases Todd Tannenbaum Department of Computer Sciences University of Wisconsin-Madison

ETICS All Hands meeting Bologna, October 23-25, 2006 NMI and Condor: Status + Future Plans Andy PAVLO Peter COUVARES Becky GIETZEL.

Condor: High-throughput Computing From Clusters to Grid Computing P. Kacsuk – M. Livny MTA SYTAKI – Univ. of Wisconsin-Madison

Todd Tannenbaum Computer Sciences Department University of Wisconsin-Madison Condor RoadMap.

The Roadmap to New Releases Derek Wright Computer Sciences Department University of Wisconsin-Madison

Derek Wright Computer Sciences Department University of Wisconsin-Madison MPI Scheduling in Condor: An.

1 Firewalls Types of Firewalls Inspection Methods  Static Packet Inspection  Stateful Packet Inspection  NAT  Application Firewalls Firewall Architecture.

Todd Tannenbaum Computer Sciences Department University of Wisconsin-Madison Quill / Quill++ Tutorial.

Dan Bradley University of Wisconsin-Madison Condor and DISUN Teams Condor Administrator’s How-to.

Condor Project Computer Sciences Department University of Wisconsin-Madison Grids and Condor Barcelona,

Condor Week 2004 The use of Condor at the CDF Analysis Farm Presented by Sfiligoi Igor on behalf of the CAF group.

Pilot Factory using Schedd Glidein Barnett Chiu BNL

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE Site Architecture Resource Center Deployment Considerations MIMOS EGEE Tutorial.

Module 10: Windows Firewall and Caching Fundamentals.

Condor week – March 2005©Gabriel Kliot, Technion1 Adding High Availability to Condor Central Manager Gabi Kliot Technion – Israel Institute of Technology.

Ian D. Alderman Computer Sciences Department University of Wisconsin-Madison Condor Week 2008 End-to-end.

1 Chapter 13: RADIUS in Remote Access Designs Designs That Include RADIUS Essential RADIUS Design Concepts Data Protection in RADIUS Designs RADIUS Design.

Computer Communication: An example What happens when I click on

22 nd Oct 2008Euro Condor Week 2008 Barcelona 1 Condor Gotchas III John Kewley STFC Daresbury Laboratory

Jaime Frey Computer Sciences Department University of Wisconsin-Madison What’s New in Condor-G.

INFSO-RI ETICS Local Setup Experiences A Case Study for Installation at Customers Location 4th. All Hands MeetingUwe Müller-Wilm VEGA Bologna, Nov.

HTCondor Security Basics HTCondor Week, Madison 2016 Zach Miller Center for High Throughput Computing Department of Computer Sciences.

IP packet filtering Breno de Medeiros. Florida State University Fall 2005 Packet filtering Packet filtering is a network security mechanism that works.

Campus Grid Technology Derek Weitzel University of Nebraska – Lincoln Holland Computing Center (HCC) Home of the 2012 OSG AHM!

Gabi Kliot Computer Sciences Department Technion – Israel Institute of Technology Adding High Availability to Condor Central Manager Adding High Availability.

IST 201 Chapter 11 Lecture 2. Ports Used by TCP & UDP Keep track of different types of transmissions crossing the network simultaneously. Combination.

Sonny (Sechang) Son Computer Sciences Department University of Wisconsin-Madison Dealing with Internet Connectivity in Distributed.

Condor Week 2006, University of Wisconsin 1 Matthew Norman Using Condor Glide-ins and GCB to run in a grid environment Elliot Lipeles, Matthew Norman,

Condor Week 09Condor WAN scalability improvements1 Condor Week 2009 Condor WAN scalability improvements A needed evolution to support the CMS compute model.

An Analysis on NAT Security

Jean-Philippe Baud, IT-GD, CERN November 2007

NAT、DHCP、Firewall、FTP、Proxy

HTCondor Networking Concepts

HTCondor Networking Concepts

HTCondor Security Basics

Dynamic Deployment of VO Specific Condor Scheduler using GT4

Operating a glideinWMS frontend by Igor Sfiligoi (UCSD)

High Availability in HTCondor

Adding High Availability to Condor Central Manager Tutorial

Implementing TMG Server Publishing

Introducing To Networking

Building Grids with Condor

HTCondor Security Basics HTCondor Week, Madison 2016

Condor Glidein: Condor Daemons On-The-Fly

Condor: Firewall Mirroring

Condor-G Making Condor Grid Enabled

Computer Networks Protocols

Presentation transcript:

Dan Bradley Condor Project CS and Physics Departments University of Wisconsin-Madison CCB The Condor Connection Broker

Condor Connections Central Manager Job Submit Point Execute Node run this job transfer files advertise negotiate you’ve been matched

Execute Node Unreachable Central Manager Job Submit Point Execute Node run this job transfer files advertise negotiate you’ve been matched Execute node is behind a firewall or is NATed. no go!

Submit Node Unreachable Submit node is behind a firewall or is NATed. Central Manager Job Submit Point Execute Node run this job transfer files advertise negotiate you’ve been matched no go!

Common Scenarios › Why cross private network boundaries?  Flocking  Multi-site Condor pool  Glidein

CCB: Condor Connection Broker › Condor wants two-way connectivity › With CCB, one-way is good enough run this job transfer files I want to connect to the submit node Job Submit Point Execute Node CCB_ADDRESS=ccb.host.name reversed connection

CCB: Condor Connection Broker › Works in the mirror case too run this job transfer files I want to connect to the execute node Job Submit Point Execute Node CCB_ADDRESS=ccb.host.name reversed connection

Execute Node CCB_ADDRESS=ccb1.host CCB_ADDRESS=ccb2.host Job Submit Point Limitations of CCB 1. Doesn’t help with standard universe 2. Requires one-way connectivity no go! GCB or VPN can help

Connecting to CCB CCB Server Job Submit Point Execute Node CCB listen CCB connect CCB server must be reachable by both sides. CCB_ADDRESS=ccb.host READ authorization level DAEMON authorization level

CCB Server Behind Firewall CCB Server Job Submit Point Execute Node CCB listen CCB connect Must have an open port to connect to CCB CCB_ADDRESS=ccb.host open port here (default 9618)

Security on Reversed Connection CCB Server Job Submit Point Execute Node CCB listen CCB connect Client and server security policies are enforced in logical direction CCB_ADDRESS=ccb.host client-side daemon-side reversed connection run this job

GCB: Generic Connection Broker › GCB: Condor  Clever: mostly invisible to Condor code  However, this makes some things difficult! › CCB: Condor  Inspired by GCB  More tightly integrated into Condor  Not a complete replacement

Why CCB? › Secure  supports full Condor security set › Robust  supports reconnect, failover › Portable  supports all Condor platforms, not just Linux

Why CCB? › Dynamic  CCB clients and servers configurable without restart › Informative log messages  Connection errors are propagated  Names and local IP addresses reported (GCB replaces local IP with broker IP) › Easy to configure  automatically switches UDP to TCP in Condor protocols  CCB server only needs one open port

Configuring CCB › The Server:  The collector is a CCB server  UNIX: MAX_FILE_DESCRIPTORS=10000 › The Client: 1. CCB_ADDRESS = $(COLLECTOR_HOST) 2. PRIVATE_NETWORK_NAME = your.domain (optimization: hosts with same network name don’t use CCB to connect to each other)

Tests of CCB › Igor Sfiligoi’s Cross-Atlantic Mega Condor Glidein Test Pool for CMS  one machine with 70 CCB collectors  execute nodes in private networks  GSI authentication  100,000 registered Condor daemons  200,000 jobs/day with one schedd

Summary › CCB makes Condor work if  You have one-way connectivity  And using Condor 7.3+  And the private side sets CCB_ADDRESS  And the private side is authorized at the DAEMON authorization level by CCB  And the public side can connect to CCB  And the public side is authorized at the READ authorization level by CCB  And not using “standard universe” Fine Print: