Grid Computing Framework A Java framework for managed modular distributed parallel computing

Presentation Plan 1. Discussion on distributed computing (Concept, examples, insight) 2. What are we doing (new?) (Our concept, innovation, vision) 3. Who is doing what (Technical details, responsibilities, SE Practices) 4. What remains to be done (Or as they say – ‘future scope’) 11/11/2010Abhishek Ganguly | Apoorv Pandey2 Grid Computing Framework

Parallel Processing (problem classes) 1.SIMD (Single Instruction Multiple Data) Used mostly in DSP i.e. Digital Signal Processing applications 11/11/2010Abhishek Ganguly | Apoorv Pandey3 Grid Computing Framework – Discussion on distributed computing Bass Boost Left | Center | Right

Parallel Processing (problem classes) 1.SIMD (Single Instruction Multiple Data) 2.MISD (Multiple Instruction Single Data) Very few actually use or implement this aspect of parallelism Task replication systems may use this technique for fault tolerance or error masking etc 11/11/2010Abhishek Ganguly | Apoorv Pandey4 Grid Computing Framework – Discussion on distributed computing

Parallel Processing (problem classes) 1.SIMD (Single Instruction Multiple Data) 2.MISD (Multiple Instruction Single Data) 3.MIMD (Multiple Instruction Multiple Data) Most commonly adopted distributed processing strategy 11/11/2010Abhishek Ganguly | Apoorv Pandey5 Grid Computing Framework – Discussion on distributed computing

Parallel Processing (Solution Approaches) 11/11/2010Abhishek Ganguly | Apoorv Pandey6 Grid Computing Framework – Discussion on distributed computing Parallel ComputersCentralizedMulti CoreSMPDistributedSuper ComputerClusterGrid Computer

Parallel Processing (Solution Approaches) 11/11/2010Abhishek Ganguly | Apoorv Pandey7 Grid Computing Framework – Discussion on distributed computing Centralized parallelism Common Memory (RAM) Common Die (Multi Core) Common Board (Multi Processor)

Parallel Processing (Solution Approaches) 11/11/2010Abhishek Ganguly | Apoorv Pandey8 Grid Computing Framework – Discussion on distributed computing Centralized parallelism Multi Core Common Memory (RAM) Common Die (Multi Core) Common Board (Multi Processor) Common Die or Chip. Everything on a single Chip. Common System bus. Common L2 cache Individual L1 cache Individual Register file Individual ALU May or may not share a common Clock

Parallel Processing (Solution Approaches) 11/11/2010Abhishek Ganguly | Apoorv Pandey9 Grid Computing Framework – Discussion on distributed computing Multi Processor Common Memory (RAM) Common Die (Multi Core) Common Board (Multi Processor) Common Motherboard Common OS instance Common RAM Individual Clock Individual System Bus Individual RAM access Higher Throughput than multi core Higher power consumption than multi core

11/11/2010Abhishek Ganguly | Apoorv Pandey10 Grid Computing Framework – Discussion on distributed computing Distributed parallelism Parallel Processing (Solution Approaches) Reduced Coupling

11/11/2010Abhishek Ganguly | Apoorv Pandey11 Grid Computing Framework – Discussion on distributed computing Distributed parallelism – SUPER COMPUTER Parallel Processing (Solution Approaches) Highly coupled systems with specialized hardware built for the purpose May be composed of hundreds of motherboards, each housing multiple CPUs RAMs spread all over the housing Interconnected by special high speed bus (Network) Each board may have its own copy of the OS. Requires Special OS Special software to run atop it Special programming methodology

11/11/2010Abhishek Ganguly | Apoorv Pandey12 Grid Computing Framework – Discussion on distributed computing Distributed parallelism – CLUSTERS Parallel Processing (Solution Approaches) These fall between super computers and grid computers These are networked computers Computers are connected through high speed LAN Each computer has its own instance of OS running for it These are the highest deployed distributed systems. Uses Load balancing (as in web servers) Availability, Robustness, fail-safe (DNS serves) Throughput increase (database clusters)

11/11/2010Abhishek Ganguly | Apoorv Pandey13 Grid Computing Framework – Discussion on distributed computing Distributed parallelism – GRID COMPUTERS Parallel Processing (Solution Approaches) Most diffused, heterogeneous and independent form of multi node alliance. Components of a grid computer can be anything from a commodity PC to clusters of supercomputers. Components can be located al over the globe. Internet is the basic networking medium between the components. Requires Special programming technique Specially written software Extensive GRID framework

11/11/2010Abhishek Ganguly | Apoorv Pandey14 Grid Computing Framework – Discussion on distributed computing Grid Computer Architecture Grid User (Programmer) Grid Coordinator Server Workers

11/11/2010Abhishek Ganguly | Apoorv Pandey15 Grid Computing Framework – Discussion on distributed computing Grid Computer Architecture

11/11/2010Abhishek Ganguly | Apoorv Pandey16 Grid Computing Framework – Discussion on distributed computing Grid Computer Architecture

11/11/2010Abhishek Ganguly | Apoorv Pandey17 Grid Computing Framework – What are we doing (new?) Project Proposal To implement such a distributed framework in java To adapt java multi-threading concepts for scalable distributed computing

11/11/2010Abhishek Ganguly | Apoorv Pandey18 Grid Computing Framework – What are we doing (new?) Implementing Distributed Framework Network Machine JVM Grid Frame Work User Program User Sub-Program User Sub-Program Grid Code Grid API

11/11/2010Abhishek Ganguly | Apoorv Pandey19 Grid Computing Framework – What are we doing (new?) Implementing Distributed Framework Network Machine JVM Grid Frame Work User Program User Sub-Program User Sub-Program Grid Code Grid API

11/11/2010Abhishek Ganguly | Apoorv Pandey20 Grid Computing Framework – What are we doing (new?) Adapting Multi Threading Concepts Common Java Code

11/11/2010Abhishek Ganguly | Apoorv Pandey21 Grid Computing Framework – What are we doing (new?) Adapting Multi Threading Concepts Grid Aware java Code

11/11/2010Abhishek Ganguly | Apoorv Pandey22 Grid Computing Framework – Who is doing what, Technical details etc Functional Objectives Research Design Coding Testing by using

11/11/2010Abhishek Ganguly | Apoorv Pandey23 Grid Computing Framework – Who is doing what, Technical details etc Research on existing work Basic understanding of parallel computing Abhishek & Apoorv MPI, Java RMI, Directory and Naming services Apoorv Object Serialization, CORBA, Dot net Remoting Apoorv Existing grid computing frameworks. BOINC, OPEN-Grid, Abhishek Distributed file systems, Distributed Synchronization Abhishek

Software Design System Architecture 11/11/2010Abhishek Ganguly | Apoorv Pandey24 Grid Computing Framework – Who is doing what, Technical details etc Client Job Server Worker

Software Design System Architecture 11/11/2010Abhishek Ganguly | Apoorv Pandey25 Grid Computing Framework – Who is doing what, Technical details etc Client Job Server Worker Client uses Grid API org.accurate.grid org.accurate.grid. Job Job RunnableSerializableThread void start() void Run() void join()

Software Design System Architecture 11/11/2010Abhishek Ganguly | Apoorv Pandey26 Grid Computing Framework – Who is doing what, Technical details etc Client WorkerJob Server Server Manages job Queue & Manages Completed jobs’ bin

Software Design System Architecture 11/11/2010Abhishek Ganguly | Apoorv Pandey27 Grid Computing Framework – Who is doing what, Technical details etc Client WorkerJob Server Server Manages job Queue & Manages Completed jobs’ bin JobServer.submitJob( Job j ); >>

Software Design System Architecture 11/11/2010Abhishek Ganguly | Apoorv Pandey28 Grid Computing Framework – Who is doing what, Technical details etc Client WorkerJob Server Server Manages job Queue & Manages Completed jobs’ bin Job j=JobServer.getJob() >>

Software Design System Architecture 11/11/2010Abhishek Ganguly | Apoorv Pandey29 Grid Computing Framework – Who is doing what, Technical details etc Client WorkerJob Server Server Manages job Queue & Manages Completed jobs’ bin JobServer.doneJob( Job j ) >>

Software Design System Architecture 11/11/2010Abhishek Ganguly | Apoorv Pandey30 Grid Computing Framework – Who is doing what, Technical details etc Client WorkerJob Server Server Manages job Queue & Manages Completed jobs’ bin Job j =JobServer.getCompleted(this) >>

Software Design Data Flow Diagram 0 11/11/2010Abhishek Ganguly | Apoorv Pandey31 Grid Computing Framework – Who is doing what, Technical details etc Grid Framework Jobs Completed Results

Software Design Data Flow Diagram 1 11/11/2010Abhishek Ganguly | Apoorv Pandey32 Grid Computing Framework – Who is doing what, Technical details etc 1.2 Job Queuing and dispatching 1.2 Job Queuing and dispatching 1.1 Job Creation 1.1 Job Creation 1.3 Job Execution 1.3 Job Execution User Program Job QueueCompleted Job Store User Program Child job modules

All that remains to be done 11/11/2010Abhishek Ganguly | Apoorv Pandey33 Grid Computing Framework – Future Scope Virtual distributed file system. A distributed file system for high volume data interchange Dedicated application servers. So that long running applications do not require their originating PCs to stay on. Integration with existing grid computing systems. So that jobs can be interchanged between them. Automatic parallelization. So that code can be parallelized without explicit programmer effort.

All that remains to be done… 11/11/2010Abhishek Ganguly | Apoorv Pandey34 Grid Computing Framework – Future Scope Wrapper to precompiled code So that precompiled modules can be reused without reengineering Worker credit tracking So that workers can be gifted benefit a/c to work they do Standardization of API So that programmers of different platforms can feel comfortable.

11/11/2010Abhishek Ganguly | Apoorv Pandey35 Grid Computing Framework – References 90% of the research material has been obtained from the internet And rest 10% from text books Text Books: Herbert Schildt, Java 2- The complete reference, Osborne. Client Server, Most of practical java Bruce Eckel, thinking in java, Pearson Most of theoretical java Galvin, Operating system concepts Scheduling, multitasking, synchronization

11/11/2010Abhishek Ganguly | Apoorv Pandey36 Grid Computing Framework – References 90% of the research material has been obtained from the internet And rest 10% from text books Websites : Wikipedia About everything has been consulted on Wikipedia Oracle sun tutorial-trail java code samples, in-depth class structure boinc.berkeley.edu Basic understanding of one of the most successful grid.

11/11/2010Abhishek Ganguly | Apoorv Pandey37 Grid Computing Framework – Rebuttle Questions

11/11/2010Abhishek Ganguly | Apoorv Pandey38 Grid Computing Framework – Thank you Thank You