1 Towards a Generic Deployment Framework for J2EE, OSGi, Web Services (or Everything) Tuesday, February 28th, 2006 Philippe Merle Jacquard Project - INRIA Futurs, Laboratoire d‘Informatique Fondamentale de Lille (LIFL), Université des Sciences et Technologies de Lille (USTL), Villeneuve d’Ascq, France ITEA S4ALL Task 3.5 Meeting, INT, Evry, France

2 Un brin d’histoire OpenCCM Plate-forme de composants répartis CORBA Automatisation du déploiement Tout est spécifique au CCM Identifier des abstractions logicielles pour construire des plates- formes de déploiement Qu’est ce qui peut être factoriser en concepts/codes entre le déploiement OpenCCM et le déploiement Fractal ? Voir articles [DECOR 2004], [LMO 2005] Le travail présenté aujourd’hui est dans la poursuite de cette recherche du Graal

3 Agenda Context Deployment of distributed services and systems Motivation No generic deployment approach/tool/framework exists! Objectives Build a generic deployment framework Principles A fine grain component-based framework with Fractal Illustrations Some of the Fractal-based components Work plan Conclusion and perspectives

4 Context - Multiple Services and Providers The user-centric approach considers the users’ services as gathering of the many services provided by the all- around world. Within each involved platform, deploying these services is made easy. Why not everything deployed in once? OSGi GatewayJ2EE Server Web Server Directories Banking Facilities Extra Stuff Monitoring

5 We Would Like To Have Something Which… Starts various processes on remote hosts Enables complex configurations for the launched processes Deploys applications’ services inside the hosting platforms Adds some test / control over this deployment Relays on nothing more than existing facilities Operating systems, remote shell accesses, etc. Via an easy to use, automation, extensible, and efficient approach!

6 Dependencies for remote deployment/launching Service Towards Users Towards Systems But Heterogeneity At Every Layer Operating System LinuxWindowsMac OS SSHTelnetOSGi Remote Access ProActive JOnASOscarJOnAS Application Server JOnAS Requires a unified approach!

7 Motivation Deployment of distributed applications is a complex task, e.g.: Connect to remote hosts Download application (and middleware) binaries Launch middleware services and application servers –Set up environment variables –Start processes Install application binaries into application servers/gateways Configure and publish application services Many other specific tasks!

8 Motivation - Related Work Scripting is just a tool instead of a paradigm, e.g.: How to reuse scripts? Many work done to automatically deploy applications Into application servers/gateways, e.g.: –J2EE, OSGi, Fractal, etc. On distributed systems, e.g.: –OLAN, OpenCCM, ProActive, JADE, etc. Via generic frameworks, e.g. –[DECOR 2004], etc. Via model driven approaches, e.g. –[LMO 2005], ORYA, OMG DnC, etc.

9 Motivation - Related Work No work cover the whole deployment process in a generic and technological independent way, e.g.: J2EE/OSGi do not address the launching of J2EE/OSGi gateways OSGi does not address deployment of distributed applications ProActive only deploys ProActive-based servers and applications OpenCCM only deploys CCM-based applications but not OpenCCM servers JADE deploys J2EE systems but not JADE servers Etc.

10 Motivation - Requirements Providing a unified approach Heterogeneity of technologies Independent against granularity Automation instead of manual operations Distributed synchronizations, e.g.: Connect to remote hosts before do anything Set up environment variables before launch processes Launch a transaction service before launch application servers Launch a registry service before publish application services Etc. Describing instead of programming

11 Objectives Build a generic deployment framework Unified approach Covering any task required to deploy Independent against granularity Distributed systems, applications, services, components, objects, etc. Independent against technology Usable with J2EE, OSGi, Web Services, Fractal, OpenCCM, etc. Easy to use by end-users Describing what must be deployed instead of how to do Highly customizable and extensible by developers Adapting the framework to specific purposes Adding new concepts/mechanisms easily

12 Principles for a Unified Approach A framework instead of a single tool Build tools from the framework Everything is reified as components From end-user configurations to technical entities Fine grain component-based approach Micro components and interfaces Intensive use of composition and sharing

13 Everything is Reified as Components A hosting system is a component containing a remote access This remote shell provides the interfaces to set variables and execute processes on the hosting system An execution platform is a component managing applications and services, with many interfaces such as deployment, monitoring, configuration, and so on. hosting system remote shell execution platform service

14 Everything is Reified as Components Network hosting system remote shell execution platform service Reify and Control

15 Everything Reified as Components Remote access protocols such as SSH, TELNET, etc. File transfer protocols such as SCP, FTP, HTTP, etc. Shells such as SH, CSH, WINBAT, etc. Bound to a protocol Shell variables and commands Bound to a shared shell Servers such as registry services and transaction managers (RMI, CORBA, WS), etc. Composed of shell variables and commands Application servers as JOnAS, OSGi, etc. Composed of shell variables and commands Application beans as J2EE jars, OSGi bundles, etc. Bound to a shared application server Etc.

16 Everything Reified as Components protocol: SSH SSHd SH shell: SHhost Reified SystemReal System JOnAS EJB server: JOnASbean: EJB

17 Technical Approach This generic deployment framework is Designed with the Fractal component model Each deployment concept/mechanism is a Fractal component Implemented with the Fraclet annotation framework Leverage Component-Based Software Engineering Configured with Fractal ADL End-users’ configurations Developers’ components Managed with Fractal Explorer Distributed with Fractal RMI Could benefit from any other Fractal-based work AOKell, FAC, etc.

18 Technical Architecture Network End-User Configuration Fractal ADL Fractal ADL Components Deployment Components hosting system remote shell execution platform service Fractal Explorer Deploy Translation Manage Instantiate Fraclet

19 A Simple End-User Configuration The End-User Description MySystem = { MyBean = BEAN { location = URL( ; server = JONAS { home = JONAS_HOME(/usr/jonas) ; host = HOST { hostname = HOSTNAME(host.lifl.fr) ; protocol = SSH ; shell = SH ; }; }; An easy to use end-users’ configuration language Describing what the configuration is instead of how to deploy it

20 A Simple End-User Configuration The Fractal Components MySystem MyBean: BEAN location: URL server: JONAS home: JONAS_HOME host: HOST hostname: HOSTNAME protocol: SSHshell: SH

21 A Simple End-User Configuration The Fractal ADL Definition

22 A Simple End-User Configuration The Expected Runtime Behavior Deploying MyBean implies to automatically 1.Connect to host.lifl.fr via the SSH protocol 2.Setup the JONAS_HOME variable to /usr/jonas 3.Add $JONAS_HOME/bin/unix to the PATH variable 4.Launch the command jonas start 5.Download 6.Launch the command jonas admin –a bean.jar

23 A More Complex End-User Configuration MySystem = ITEA_S4ALL { hosts = {... }; servers = {... }; beans = AUTO_INSTALL {... };

24 A More Complex End-User Configuration MySystem = ITEA_S4ALL { hosts = { M = HOST { hostname = HOSTNAME(host.lifl.fr) ; protocol = SSH ; shell = SH ; }; … other host definitions... };

25 A More Complex End-User Configuration MySystem = ITEA_S4ALL { servers = { jonas_on_M = JONAS { home = JONAS_HOME(/usr/jonas) ; host = hosts/M ; }; oscar_on_M = OSCAR { home = OSCAR_HOME(/usr/oscar) ; host = hosts/M ; };... Other server definitions … }; Sharing: - The M host - The SSH channel - The SH shell

26 A More Complex End-User Configuration MySystem = ITEA_S4ALL { beans = AUTO_INSTALL { bean1 = BEAN { location = LOCAL_FILE(bean1.jar) ; server = servers/jonas_on_M ; }; bean2 = BEAN { location = LOCAL_FILE(bean2.jar) ; server = servers/jonas_on_M ; }; bean3 = BEAN { location = LOCAL_FILE(bean3.jar) ; server = servers/oscar_on_M ; }; bean4 = BEAN { location = LOCAL_FILE(bean4.jar) ; server = servers/oscar_on_M ; };... Other bean definitions … }; JOnAS on host M shared by beans OSCAR on host M shared by beans Ask for auto installation

27 A More Complex End-User Configuration The Expected Runtime Behavior 1.Connect to host.lifl.fr via the SSH protocol 2.Setup the JONAS_HOME variable to /usr/jonas 3.Add $JONAS_HOME/bin/unix to the PATH variable 4.Launch the command jonas start 5.Download 6.Launch the command jonas admin –a bean1.jar 7.Download 8.Launch the command jonas admin –a bean2.jar 9.Setup system variables related to OSCAR 10.Launch OSCAR 11.Execute OSCAR commands to install bean3.jar and bean4.jar Other execution sequences and/or parallelization are possible

28 Everything is Reified as Components Network hosting system remote shell execution platform service Control

29 Everything is a Service (Not a Web Service) Each component has a Service interface to control the life cycle of the real entity reified interface Service { void start(); void stop(); } Semantics depends on components/implementations Protocol.start ~ initializes the network connection Protocol.stop ~ shutdowns the network connection Server.start ~ launches the server process Server.stop ~ stops the server process Bean.start ~ launches the bean into the application server Bean.stop ~ uninstalls the bean from its server

30 Composition Composition of Services Host.inria.fr Shell SH Protocol SSH JOnAS OSCAR Bean

31 Orchestration of Services Goal Address distributed synchronizations of composition of services Orchestration Implements the Service interface Schedules execution of services automatically Implementation Introspects the composition and bindings between components to plan the execution of services Various execution strategies –Sequential –Parallel

32 Orchestration Orchestration of Services Host.inria.fr Shell SH Protocol SSH JOnAS OSCAR Bean Service

33 Orchestration Orchestration of Services Host.inria.fr Shell SH Protocol SSH JOnAS OSCAR Bean (1) start (2) start (3) start (4) start (5b) start (6) start (5a) start (6) start

34 Orchestration Orchestration of Services Host.inria.fr Shell SH Protocol SSH JOnAS OSCAR Bean (1) stop (6) stop (5) stop (4) stop (3) stop (2) stop (3) stop (2) stop

35 Protocol Components Reify and encapsulate protocols to access the real system interface Protocol { void send(String command); } Possible implementations LOCAL by wrapping an external local shell SSH by wrapping –an external local ssh process –the BSD open source JSCH library – TELNET by wrapping – an external local telnet process –the open source JTelnet library – Source Forge –the Apache open source Jakarta Commons Net library – Protocol Service

36 Internet Protocol Components Internet Protocol Basic Protocol as SSH, TELNET IP Host Name IP Port

37 Implementation of the IP Port Component with Fraclet name=“port” */ interface IpPort { int getIpPort(); } class PORT implements IpPort { */ protected int port; public int getIpPort() { return port; } } Fractal AttributeController interface & implementation, and Fractal ADL definitions generated automatically by Fraclet :-)

38 name=“hostname” */ interface IpHostName { String getIpHostName(); } class HOST_NAME implements IpHostName { */ protected String hostname; public String getIpHostName() { return hostname; } } Fractal AttributeController interface & implementation, and Fractal ADL definitions generated automatically by Fraclet :-) Implementation of the IP Host Name Component with Fraclet

39 Implementation of the Basic Internet Protocol Components with Fraclet abstract class InternetProtocol implements Service, Protocol { */ protected IpHostName hostname; */ protected IpPort port; } class Telnet extends InternetProtocol { public void start() {... hostname.getIpHostName() port.getIpPort()... } public void stop() {... } public void send(String command) {... } } Ditto for all possible implementations, e.g.: TELNET, SSH, etc. Via an external process or an embedded Java library

40 Shell Components Reify shells executed in the real system interface Shell { void set_variable(String name, String value); void unset_variable(String name); void execute(String command); } Each shell is bound to a single Protocol component Behavior Construct commands according to the shell syntax Format path separator, i.e. / \ : ; Possible implementations SH for Borne Unix shells CSH for C Unix shells WINBAT for Windows command shells ProtocolShell

41 The HOST Component Provide an abstract composition of (Shell, InternetProtocol, Hostname) Simplify the configuration for end-users shell protocol hostname

42 The VARIABLE Component Reify shell variables Each variable is bound to a shared Shell component Possible implementation class VARIABLE implements Service { */ protected Shell shell; */ protected String name; */ protected String value; public void start() { shell.set_variable(name, value); } public void stop() { shell.unset_variable(name); } } Possible instantiations JONAS_HOME(path) = VARIABLE(JONAS_HOME, path) OSCAR_HOME(path) = VARIABLE(OSCAR_HOME, path) VARIABLE ShellService

43 The Command Component Reify shell commands on the real system interface Command { void execute(Map arguments); } Each command is bound to a shared Shell component Possible implementation class COMMAND implements Command { */ protected Shell shell; */ protected String command; private String computeCommand(Map arguments) {... } public void execute(Map arguments) { shell.execute(computeCommand(arguments)); }} Possible instantiations COMMAND(echo hello world!) COMMAND(jonas start) COMMAND(jonas admin –a %url%) COMMAND Shell Command

44 Server Components Reify a server/daemon executed in the real system Examples Middleware servers –Registry –Transaction Manager Apache servers Various implementations Wrapping a protocol to interact with the remote server Bound to a shell Etc. SERVER Service

45 Implementation of the SERVER Component by Composition VARIABLE shell COMMAND start Server Impl SERVER Service * COMMAND stop SERVER * depend

46 Implementation of the SERVER Component by Composition (Java code) class ServerImpl implements Service { signature=“Service” */ protected Map required-servers; signature=“Service” */ protected Map variables; */ protected Command start-command; */ protected Command stop-command; public void start() { start-command.execute(null); } public void stop() { stop-command.execute(null); } }

47 Implementation of the SERVER Component by Composition (Fractal ADL)

48 Other Concerns to Address Display output from remote hosts Various GUI Also Fractal components Configurable by end-users Take care of system resource management Threads and Sockets are limited Study work done on DREAM Use AOKell to build new controllers for system resources Quid for deployment of large scale systems Distribute Fractal components into several JVM/host To address system resource limitation Using Fractal RMI Transactional deployment GoTM framework Dynamic reconfiguration Fractal Explorer to manage components, as done by JADE Rules for autonomous computing

49 Work Plan Proof of concept already done Areski Flissi and Philippe Merle involved Engineer recruitment from March 1st 2006 Define Fractal interfaces and components Develop micro-components with Fraclet Compose components with Fractal ADL Implement Fractal ADL translator Implement management with Fractal Explorer Evaluate on various ITEA S4ALL scenarios

50 Conclusion A deployment framework Generic Independent against technology and granularity Easy to use Highly customizable and extensible Fine grain component-based approach Each deployment mechanism is reified as a component Intensive use of the Fractal component model and tools Composition and sharing Julia, AOKell, Fraclet, Fractal ADL, Fractal Explorer, etc. Will be developed soon Prototyping already done Implementation details could change but not the overall approach

51 Perspectives Deployment on Grid’5000  Specific components for OAR and KaDeploy services Deployment of OpenCCM  Specific components for OpenCCM daemons  Automation on PDA prototyped Deployment of Fractal  Specific components for Fractal RMI Registry and Server Autonomous computing Rules for dynamic reconfiguration – Ph.D. Jérémy Dubus

52 Thank you for your attention… If you have any questions?

53 TODO The JRE HOST Component Counter Start/stop + synchronization sur start ApplicationServer JONAS OSGi BEAN File transfer

54 class COMPUTED_HOSTNAME implements IpHostName { */ protected IpHostName computer; protected String hostname = null; public String getIpHostName() { if(hostname==null) hostname = computer.getIpHostName(); return hostname; } } Fractal BindingController code generated automatically by Fraclet :-) Hostname computation could be done by a reservation system like OAR on Grid’5000 :-) Another Implementation of the IP Host Name Component with Fraclet COMPUTED_H OSTNAME HOSTNAME_C OMPUTER

55 Automation Protocol Components PROTOCOL Do start() at the 1st send() Any Protocol

56 Implementation of Automation Protocol Component with Fraclet class AutomationProtocol implements Protocol { */ protected Protocol next; private boolean isInit = false; public synchronized void init() { if(isInit == false) { next.init(); isInit = true; } } public void send(String command) { this.init(); next.send(command); } public void shutdown() { next.shutdown(); } }

57 The JRE Component Provides an abstract composition for JRE VARIABLE JAVA_HOME shell VARIABLE PATH JRE Impl JRE

58 Java Implementation of the JRE Component class JREImpl implements Environment { */ protected Environment home; */ protected Environment path; public void set(){ home.set(); path.set(); } public void unset() { home.unset(); path.unset(); } }

59 Fractal ADL for the JRE Component Fractal ADL definition Fractal ADL usage