Design and Implementation of PARK (PARallel Kernel for data fitting) Paul KIENZLE, Wenwu CHEN and Ziwen FU Reflectometry Group

Distributed Computing Environment Service Server Master Node User Cluster Working Nodes User/Client ServiceServer Management WorkingServer User

Service Server WorkingServer WorkingClient Service User JobQueueSys MessageQueueSy s ServiceServer

Prototype Demo Download Source code: –svn co Edit cluster config file: –park/config/hosts Start service server –park/servers/mapServer.py Start client –park/client/AppJob.py Provide services –park/services

Further Improvement (working server) 3 running modes –Command: simple commands, short time ls, top, kill, … –Thread: python functions, short time Multithreading, not recommend –Process: normal services, long time any executable programs Task and task queue management: –Restrict the number of running commands, threads, and processes, and their running times –Kill the task by its task ID –Query the task queue status

Further Improvement (Service server) –More stable and efficient for the service server –More functions for service server Submit job Register as a listener Query job status and services Kill job by its ID Run job on specified node (s) –Better job queue and message queue management –Pluggable reduce functions –Restart the working node when it crashes

User Interface Service Server Scripting User Interface GUI Files Web External Apps data

Software Infrastructure of PARK for data fitting Service Server Service Working Nodes User Interface Scientist View DeveloperReduce Service Developer Data reduction Model Developer Data simulation Data presentation Data View

Procedure for data fitting dataset Reduction / filter metadata Theory parameter Experimental data Simulation data uniplexor multiplexor optimizer Variables Dependency Constraints

XML tree for data fitting DataFitting Optimizers* Multiplexor DataSet Variables* DataFilterFunctions* Dependency Constraints Uniplexors* Theory* Parameters* Uniplexors DataSource Variable

Base Classes for XML/ Object tree For the developer, XML tree is actual the object tree. It is the common task to transfer between the XML tree and object tree

Classes from XML tree DataFitting Optimizer Multiplexor DataSetVariables DataFilterFunction Dependency Constraints Uniplexor Theory Parameter DataSource Variable XmlObject XmlAttributeObject Model DeveloperReduction Developer

Basic functions for classes Datasource: –read(), write() :read/write to local file system –fetch(), send() : read/write to remote file system –getdata(hints:object), getmetadata(hints:object): Get the data and meta data DatafilterFunction: –filter (input:object, hints: object)

Basic functions for classes Theory: –calculate(): do the calculation –getSimulationData(): get the simulation data Parameter: –hasDerivative(order: int, hints:object) Query whether the its derivatives are available

Example: sans optimization

XML tree for request Request JobQuery JobSubmission* JobGroup* DataFitting JobRegister JobKill ReduceFunction Jobs* Task TaskQuery Command User Inputs* TaskKill Reduction function Developer External application Developer, UI Developer Service Server Working Server

XML tree for UI UI Request Viewer Viewer Developer, UI developer WebViewerFileViewer GUI Viewer Viewer

GUI (MVC pattern) OptimizerViewer ParameterViewer ModelViewer DataSetViewer Variables Dependency Constraints ReplyViewerRequestViewer GUIViewer StatusViewer

GUI Demo Prototype GUI MVC GUI