Condor Team 2008

Welcome to Condor Week #10 (year #25 for the project)

Goodbye Grids Welcome Clouds! we survived grids, we are ready for clouds and prepare for what will follow

Subject: PS From: Ian Foster Date: Thu, 4 Dec :40: To: Miron Livny Miron: I visited CSIRO today and they told me that they are deploying a "desktop cloud." I asked them what "cloud software" they are using, and I was pleased to hear that they were using Condor. So Condor is officially cloud software, at least in Australia. I thought you'd like to know. Ian.

Stay away from the hype and stick to the principals!

High Throughput Computing became a well established and widely used paradigm

Condor-G Glide-ins Matchmaking carry the load of EGEE and OSG

Virtual Machines fit our principals (they look like a job, they ‘come and go’, they checkpoint, they have many attributes, the can be matched, … )

Innovation through widely adopted technology partnerships with users in academia and industry, and IT providers

Miron Livny Center for High Throughput Computing Computer Sciences Department University of Wisconsin-Madison The Condor Project – an Experiment in Experimental Computer Science

Why am I here today? Because I believe that:  the Condor project is ‘different’,  we do not have many (any?) other such Computer Science projects,  it is important for the future of our science to have more (many?) such projects,  we can and should learn from what we have done so far what it takes to build and sustain such projects and  CISE can and should play a leadership role in changing the dynamics of our field so that more Computer Science departments will have such projects

“Why are you leaving academia and taking a job in industry?” “I want to have impact! ”

In the words of Mike Carey “ I left academia for industry because I was drawn to the idea of getting more direct access to real problems - from customers and challenges encountered while building commercial-grade software - because I felt like I was in somewhat of a mode of inventing and solving problems, at least w.r.t. some of the things I'd been working on. Sure, that was leading to many written/submitted/accepted papers, but it was somehow less than satisfying after awhile.”

It is all about Software I argue that ‘we’ do not know how  to develop, maintain, support and evolve (dependable) software,  to integrate software components into end-to- end (dependable) capabilities,  to evaluate the quality of software,  to estimate the cost (effort) related to software and  to treat software as infrastructure

(my) terminology › Experiment - an act or operation for the purpose of discovering something unknown or of testing a principle, supposition, etc.: › Technology Adoption – to select a technology as a means to meet an ends of significant importance/value › Real users – individuals or groups who adopt (and use) a computing technology › Experimental Computer Science – advance the state of the art of computing (new frameworks, new technologies, new abstractions) through experiments that involve real users

Two new Institutes on the UW Campus - MIR & WID 16

The MIR Three-Plane Model 17

Computer Systems and Operations Clemson Computing and Information Technology Using Condor with Blackboard Sam Hoover IT Systems Architect Computer Systems and Operations CCIT, Clemson University

Computer Systems and Operations Clemson Computing and Information Technology Problem Statement Courses in Blackboard need to be archived Blackboard provides a batch archive script Over 6,200 courses are currently active The course list was split into equal numbers across 5 servers Archives took 60 hours to complete

Computer Systems and Operations Clemson Computing and Information Technology Implementing Condor with Blackboard Blackboard is a “heavy” Java Enterprise application We wanted to take advantage of the multiple cores per server Security is a requirement Performance of the application during processing MUST be maintained

Sam Hoover:

Computer Systems and Operations Clemson Computing and Information Technology Benefits of using Condor with Blackboard Job scheduling ensures load balancing and enables higher throughput through use of all available hardware resources Post processing with DAGMan streamlines operation Jobs are suspended and resumed automatically to protect user performance Archive time has been reduced by 65% End of Semester “full” archive is now feasible

Computer Systems and Operations Clemson Computing and Information Technology Credits Clemson could not have achieved this innovation without the contributions of the following people and groups: Sam Hoover, CSO - Principal Innovator Randy Martin and Matt Garrett, CSO NSF CI-TEAM award (RENCI principal) Sebastien Goasguen, School of Computing Blackboard Operations Team (CCIT-wide representation)

Thank you for building such a wonderful community