The EDGeS project receives Community research funding 1 Porting Applications to the EDGeS Infrastructure A comparison of the available methods, APIs, and technologies Attila Csaba Marosi MTA SZTAKI

Porting Applications to a BOINC Infrastructure 2 Porting Applications to BOINC Porting applications to BOINC is not a straightforward processPorting applications to BOINC is not a straightforward process The infrastructure requires the implementation of various components (work unit generator, assimilator, worker application, etc.)The infrastructure requires the implementation of various components (work unit generator, assimilator, worker application, etc.) Developers can find a wide variety of APIs and methods to port their applicationsDevelopers can find a wide variety of APIs and methods to port their applications It is not easy to select the best one for our applicationIt is not easy to select the best one for our application This presentation gives an overview of the available methods, APIs, and technologiesThis presentation gives an overview of the available methods, APIs, and technologies

Porting Applications to a BOINC Infrastructure 3 Application Types There are two types of applications:There are two types of applications: –Applications whose source code is available and can be modified easily –Legacy applications that cannot be modified (e.g. the source code is not available) or it is not advisable to modify them

Porting Applications to a BOINC Infrastructure 4 Available Methods, APIs, and Technologies Server sideServer side –BOINC API –DC-API –gUSE Client sideClient side –BOINC API –DC-API –Boinc Wrapper technology –GenWrapper technology

Porting Applications to a BOINC Infrastructure 5 BOINC API The original API, can only be used with BOINCThe original API, can only be used with BOINC A set of C and C++ functionsA set of C and C++ functions Rich feature set at the expense of complexityRich feature set at the expense of complexity 4 separate applications (WU generator, validator, assimilator, worker)4 separate applications (WU generator, validator, assimilator, worker) Complex porting processComplex porting process

Porting Applications to a BOINC Infrastructure 6 BOINC API Functionality WU Generator: - work unit generation, submission - work unit generation, submission - input and output file handling - input and output file handlingValidator: - validator frameworks (simple, low-level) - validator frameworks (simple, low-level)Assimilator: - handler function linked with a special program - handler function linked with a special program Initialisation Input and output file identification I/O wrappers Checkpointing Progress, credit reporting Critical sections Atomic file update Time handlers Graphics API (screen savers, icons) Other APIs (status, diagnostics, etc.) Grid System Client Application BOINC API Client Application BOINC API WU GeneratorValidatorAssimilator BOINC API

Porting Applications to a BOINC Infrastructure 7 DC-API Distributed Computing Application Programming InterfaceDistributed Computing Application Programming Interface Allows easy implementation and deployment of distributed applications on multiple grid environmentsAllows easy implementation and deployment of distributed applications on multiple grid environments An API that hides the details of BOINCAn API that hides the details of BOINC The DC-API is designed to be simple and easy to useThe DC-API is designed to be simple and easy to use Supports a master-worker programming modelSupports a master-worker programming model Work units are sequential applicationsWork units are sequential applications

Porting Applications to a BOINC Infrastructure 8 DC-API Functionality Grid System Client Application DC-API Client Application DC-API Initialisation Work unit generation Input and output file handling Work unit submission Event processing (e.g. results) Result assimilation Work unit deletion Initialisation Input and output file identification Event processing (e.g. checkpointing) Progress reporting Configuration, logging, messaging Common features: ValidatorMaster Application BOINC APIDC-API

Porting Applications to a BOINC Infrastructure 9 BOINC Wrapper Technology Can be used to port legacy applications easilyCan be used to port legacy applications easily No need the change the original applicationNo need the change the original application The wrapper runs legacy applications as sub- processesThe wrapper runs legacy applications as sub- processes The wrapper plays the role of the BOINC client applicationThe wrapper plays the role of the BOINC client application Handles all communication with the core client (fraction, CPU time reporting, suspend, resume, abort, etc.)Handles all communication with the core client (fraction, CPU time reporting, suspend, resume, abort, etc.) Tasks:Tasks: –Compile the Wrapper –Create a Job description file (XML)

Porting Applications to a BOINC Infrastructure 10 BOINC Wrapper Structure BOINC core client BOINC application compiled with the BOINC libraries Wrapper Legacy Application Without the Wrapper With the Wrapper Executes the legacy application(s) in a specific order

Porting Applications to a BOINC Infrastructure 11 GenWrapper technology Generic wrapper interpreter based on GitBoxGeneric wrapper interpreter based on GitBox Runs legacy applications without BOINCificationRuns legacy applications without BOINCification Makes the BOINC API / DC-API available in POSIX shell scriptingMakes the BOINC API / DC-API available in POSIX shell scripting A shell interpreter is started instead of the real application that executes an application scriptA shell interpreter is started instead of the real application that executes an application script The scriptThe script –realizes BOINCification through script commands –may run legacy applications in any way –may perform any preparation on input-, output files, environment, etc. –may do whatever you can do in a script

Porting Applications to a BOINC Infrastructure 12 GenWrapper Structure BOINC core client Application Work unit A ZIP file (optional) Launcher GenWrapper binaries Legacy application Profile script Application script Input files 3. Extracts 5. Executes 1. Downloads 2. Downloads Acts as the BOINC client application 6. Gets executed by GitBox 4. Launches GitBox Can do almost anything that a shell script can (e.g. can start the legacy application) Output files 7. Produces

Porting Applications to a BOINC Infrastructure 13 gUSE & WS-PGRADE gUSE (grid User Support Environment)gUSE (grid User Support Environment) –is a grid virtualization environment –exposes the grid as a workflow –enables the execution of workflows simultaneously in many grids no matter what their middleware is WS-PGRADE is the user interface to supportWS-PGRADE is the user interface to support –editing, configuring, publishing workflows (as grid applications) Workflows can be built from four types of jobsWorkflows can be built from four types of jobs –Normal, sequential job –Generator –PS (Parameter Sweep) –Collector Jobs can be submitted to a BOINC DGJobs can be submitted to a BOINC DG

Porting Applications to a BOINC Infrastructure 14 gUSE Parameter Study Workflow GEN Grid job generates input parameter space COLL Collector grid job evaluates the results of the simulation SEQ Parameter sweep grid jobs (can run on a DG for example) This could be any workflow

Porting Applications to a BOINC Infrastructure 15 Comparison of the methods, APIs, and technologies (server side) BOINC API DC-APIgUSE Programming languageC/C++/Fortran/PythonC/C++(/Python) Jobs can execute any kind of binaries Number of applications to be written 3 (WU generator, Validator, Assimilator) 2 (Master application, Validator) 1 Workflow with at least 2 jobs (Generator, Collector) Easy to use GUI Can be used with other Grid middlewares Implementation effort (1. Maximum, 3. Minimum)122.5 Programming conceptMaster-workerMaster-worker Workflow based Parameter study support Method of execution From the command line From a portal

Porting Applications to a BOINC Infrastructure 16 Comparison of the methods, APIs, and technologies (client side) BOINC API DC-API BOINC Wrapper GenWrapper Programming language C/C++ /Fortran /Python C/C++ (/Java, experimental /Python) No programming, only a job description file POSIX shell scripting Suitable for legacy applications (difficult implementation) (difficult implementation) Suitable for non- legacy applications (if it is based on shell scripts) (if it is based on shell scripts) Application specific configuration support Logging support Checkpointing support (application level)

Porting Applications to a BOINC Infrastructure 17 Resources BOINC API (BOINC Wiki):BOINC API (BOINC Wiki): – DC-API:DC-API: – gUSE:gUSE: – BOINC Wrapper:BOINC Wrapper: – GenWrapper:GenWrapper: –