A Guided Tour of BOINC David P. Anderson Space Sciences Lab University of California, Berkeley TACC November 8, 2013.

Slides:



Advertisements
Similar presentations
BOINC The Year in Review David P. Anderson Space Sciences Laboratory U.C. Berkeley 22 Oct 2009.
Advertisements

BOINC The Year in Review David P. Anderson Space Sciences Lab U.C. Berkeley 12 Sept 2008.
Apache Jakarta Tomcat Suh, Junho. Road Map Tomcat Overview Tomcat Overview History History What is Tomcat? What is Tomcat? Servlet Container.
Platform as a Service (PaaS)
Volunteer Computing and Hubs David P. Anderson Space Sciences Lab University of California, Berkeley HUBbub September 26, 2013.
Operating Systems Operating System
Cloud Computing for the Enterprise November 18th, This work is licensed under a Creative Commons.
Hands-On Microsoft Windows Server 2008 Chapter 1 Introduction to Windows Server 2008.
1 port BOSS on Wenjing Wu (IHEP-CC)
Achievements and Opportunities in Volunteer Computing David P. Anderson Space Sciences Lab U.C. Berkeley 18 April 2008.
Connecting OurGrid & GridSAM A Short Overview. Content Goals OurGrid: architecture overview OurGrid: short overview GridSAM: short overview GridSAM: example.
DB Libraries: An Alternative to DBMS By Matt Stegman November 22, 2005.
HTCondor and BOINC. › Berkeley Open Infrastructure for Network Computing › Grew out of began in 2002 › Middleware system for volunteer computing.
A Distributed Computing System Based on BOINC September - CHEP 2004 Pedro Andrade António Amorim Jaime Villate.
Nimrod/G GRID Resource Broker and Computational Economy David Abramson, Rajkumar Buyya, Jon Giddy School of Computer Science and Software Engineering Monash.
Volunteer Computing David P. Anderson Space Sciences Lab U.C. Berkeley May 7, 2008.
Volunteer Computing with BOINC David P. Anderson Space Sciences Laboratory University of California, Berkeley.
Scientific Computing in the Consumer Digital Infrastructure David P. Anderson Space Sciences Lab University of California, Berkeley The Austin Forum November.
David P. Anderson Space Sciences Laboratory University of California – Berkeley Designing Middleware for Volunteer Computing.
Exa-Scale Volunteer Computing David P. Anderson Space Sciences Laboratory U.C. Berkeley.
Introduction to the BOINC software David P. Anderson Space Sciences Laboratory University of California, Berkeley.
Wenjing Wu Computer Center, Institute of High Energy Physics Chinese Academy of Sciences, Beijing BOINC workshop 2013.
07:44:46Service Oriented Cyberinfrastructure Lab, Introduction to BOINC By: Andrew J Younge
Volunteer Computing with BOINC Dr. David P. Anderson University of California, Berkeley SC10 Nov. 14, 2010.
Volunteer Computing with GPUs David P. Anderson Space Sciences Laboratory U.C. Berkeley.
BOINC: Progress and Plans David P. Anderson Space Sciences Lab University of California, Berkeley BOINC:FAST August 2013.
David P. Anderson Space Sciences Laboratory University of California – Berkeley Designing Middleware for Volunteer Computing.
How* to Win the #BestMicrosoftHack Shahed Chowdhuri Sr. Technical WakeUpAndCode.com *Hint: Use the Cloud.
David P. Anderson Space Sciences Laboratory University of California – Berkeley Public and Grid Computing.
TEMPLATE DESIGN © BOINC: Middleware for Volunteer Computing David P. Anderson Space Sciences Laboratory University of.
David P. Anderson Space Sciences Laboratory University of California – Berkeley Public Distributed Computing with BOINC.
Building and Diagnosing Applications using Visual Studio and Azure SDK Paul Yuknewicz Principal PM Manager.
BOINC: An Open Platform for Public-Resource Computing David P. Anderson Space Sciences Laboratory U.C. Berkeley.
Celebrating Diversity in Volunteer Computing David P. Anderson Space Sciences Lab U.C. Berkeley Sept. 1, 2008.
David P. Anderson Space Sciences Laboratory University of California – Berkeley Public Distributed Computing with BOINC.
Cloud Computing from a Developer’s Perspective Shlomo Swidler CTO & Founder mydrifts.com 25 January 2009.
Amazon Web Services. Amazon Web Services (AWS) - robust, scalable and affordable infrastructure for cloud computing. This session is about:
Volunteer Computing with BOINC Dr. David P. Anderson University of California, Berkeley SC10 Nov. 14, 2010.
Volunteer Computing and BOINC Dr. David P. Anderson University of California, Berkeley Dec 3, 2010.
Frontiers of Volunteer Computing David Anderson Space Sciences Lab UC Berkeley 30 Dec
Emulating Volunteer Computing Scheduling Policies Dr. David P. Anderson University of California, Berkeley May 20, 2011.
Volunteer Computing: Involving the World in Science David P. Anderson U.C. Berkeley Space Sciences Lab February 16, 2007.
The Limits of Volunteer Computing Dr. David P. Anderson University of California, Berkeley March 20, 2011.
Volunteer Computing and Large-Scale Simulation David P. Anderson U.C. Berkeley Space Sciences Lab February 3, 2007.
Volunteer Computing with BOINC: a Tutorial David P. Anderson Space Sciences Laboratory University of California – Berkeley May 16, 2006.
Frontiers of Volunteer Computing David Anderson Space Sciences Lab UC Berkeley 28 Nov
Session 2: Creating a BOINC project. The BOINC database (MySQL) ● Application ● Application version ● Platform ● Workunit ● Result ● Host ● User ● Trickle.
Volunteer Computing David P. Anderson U.C. Berkeley Space Sciences Lab January 30, 2007.
Session 3 Developing a BOINC application. Development environments ● Windows – Visual Studio 2003 or 2005 – Tortoise CVS ● Mac OS X – Xcode ● Linux –
A Brief History of BOINC
Compute and Storage For the Farm at Jlab
Platform as a Service (PaaS)
Volunteer Computing and BOINC
A new model for volunteer computing
Android Mobile Application Development
The 9th Annual BOINC Workshop
Platform as a Service (PaaS)
University of California, Berkeley
Volunteer Computing: SETI and Beyond David P
Platform as a Service (PaaS)
Volunteer Computing for Science Gateways
Designing a Runtime System for Volunteer Computing David P
Nebula A cloud-based back end for
Platform as a Service.
Chapter 3: Windows7 Part 4.
The software infrastructure of II
Introduction to Apache
Google App Engine Ying Zou 01/24/2016.
University of California, Berkeley
Presentation transcript:

A Guided Tour of BOINC David P. Anderson Space Sciences Lab University of California, Berkeley TACC November 8, 2013

BOINC in a nutshell ● BOINC is a batch system for resources that are – Extremely heterogeneous – Numerous – Sporadically available and connected – High-churn – Error-prone – Untrusted and anonymous

BOINC components Job handling (C++) Web interfaces (PHP) BOINC client (C++) BOINC API (C++) application client server

BOINC abstractions ● Application ● Platform – windows_x86_64 – x86_64-linux-gnu – x86_64_apple-darwin – arm-android-linux-gnu

BOINC abstractions ● App version – App, platform, version# – A set of files, including executable – “plan class”; determines processor usage ● # CPUs ● # GPUs

BOINC abstractions ● Job – List of input files – Latency bound – Resource usage ● Disk, memory, FLOPS ● Job instance

BOINC abstractions App App version Platform Job Job instance

BOINC client: directory structure Boinc/ projects/ proj1_url/ data1.txt slots/ 0/ in.txt link

The BOINC data model ● Files are immutable ● Client garbage-collects files ● Data files can be marked as “sticky” ● App version files are automatically sticky

BOINC client: runtime system Shared memory Message-passing BOINC client application Main thread API thread ● Process control – Suspend/resume/quit ● Fraction done reporting

The BOINC API ● boinc_init() ● boinc_resolve_filename() ● boinc_time_to_checkpoint() ● boinc_checkpoint_done() ● boinc_finish()

Building apps for BOINC ● Native (C/C++, FORTRAN, Java, Python) – Must call boinc_init() – Win: Visual Studio or MinGW – Mac: Xcode – Unix: gcc ● BOINC wrapper ● Vbox wrapper

Anonymous platform mechanism ● Volunteer (not project) supplies app versions ● Purposes: – Unusual platforms or coprocessors – Optimized apps – Security paranoia

BOINC server structure ● MySQL database ● Directory structure ● Process structure – Daemons, CGI programs ● User/group structure – apache, boincadm

Creating a BOINC project ● Deployment options – Server VM image (Virtualbox, Debian) – Amazon EC2 image – configure/make ● make_project script

Deploying application versions ● Directory structure apps/ appname1/ 1.0/ windows_intelx86/ (files) windows_intelx86__cuda/ (files) i686-apple-darwin/ (files) ● Code signing ● update_versions script

Plan class mechanism ● App versions can be tagged with a “plan class” ● app_plan(plan_class, host) function: – Can host run app version of that plan class? – If so, compute resource usage (CPUs, GPUs) – If so, estimate FLOPS ● Examples – vbox32 – cuda23 – opencl_nvidia_101

Job submission ● Input file staging – upload/download directory hierarchies ● Input, output templates – describe job’s input/output files ● Local job submission – C++, command-line interfaces ● Remote job submission – Web RPCs; C++, PHP interfaces

Job processing ● Validation – Replication – Homogeneous redundancy – Adaptive replication ● Assimilation

Other scheduling features ● Locality scheduling – sticky data files – preferentially send jobs to clients that already have the needed files ● Multi-size applications – send large jobs to fast devices

Multi-user projects ● Job submitters have accounts ● access control ● quotas ● batch scheduling – run small batches first – don’t starve large batches – enforce quotas

Server processes Shared memory Job cache scheduler transitioner feeder MySQL DB validator assimilator db_purge file_deleter per app

Scaling server performance ● Distribute daemons across machines ● Parallelize daemons – on same or different machines ● MySQL server performance ● example: server – ~50 machines – 1 million jobs/day

Web interface (public) ● preferences ● forums ● teams ● profiles ● leader boards ● host, job info ● social network features

Contacts ● ● ● lists: – boinc_projects – boinc_dev