© 2003 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice SmartFrog Overview HP Labs Version: 0.6 (Draft) Localized for US English

page 2July 2003 What is SmartFrog? SmartFrog: “Smart Framework for Object Groups” Describes services as collections of components Activates services by realizing service descriptions Framework elements: – language – engine – components A framework to build from, not a packaged solution Mature; already embedded in products

page 3July 2003 SmartFrog framework language parser process model transport protocol transactional lifecycle discovery naming scripting transforms templates component model reliability services deployment binding live-ness security model config servers workflows config. models Language Components Engine

page 4July 2003 What problem does it solve? How to configure & automatically activate complex, distributed apps – flexible – repeatable – sequencing, failure-recovery, evolution, adaptation, … Ideal for the utility computing model – automatic app activation on utility resources – enables rapid resource repurposing for different apps

page 5July 2003 How does SmartFrog address the problem? 1. The common problems No standard method to capture service config., multiple: – definitions of each value – mechanisms for access – notations (XML, ini files, SQL,...) No method of validation for the configuration as a whole – variable lifecycles – no up-front consideration of complete system startup/shutdown – lack of facility to compose systems from sub-systems No separation of concerns – functionality and configuration mixed together – rigid location and binding integrated into code Lack of auto-discovery, self-monitoring, and automation

page 6July 2003 How does SmartFrog address the problem? 2. Philosophy Services are collections of components – software, data: disk files, hardware Components collaborate to achieve some goal for which they must be appropriately organised: – required components must be defined – correctly initialised, with appropriate attributes – be able to locate each other as required – exchange information regarding their state

page 7July 2003 How does SmartFrog address the problem? 3. Approach Language allows rich system descriptions Engine automates application deployment, activation, management and shutdown Component model allows the engine to be extended

page 8July 2003 Examples Network monitoring application – many monitoring components spread throughout the network – each component must be correctly configured – service as a whole must be correctly configured / customized – service components must come up in the correct sequence Three-tier web application – database, application logic engine and web-server must each be installed, configured and started correctly – must be started on the correct host(s), in the correct sequence

page 9July 2003 SmartFrog elements Language – templates / descriptions Engine – interprets descriptions to activate running services Components – make up the running service – deployed, configured & activated by the engine Language Engine activates running components managed, monitored through lifecycle which service components? running where? how is each component initialised? how are components related? how are the component lifecycles sequenced?

page 10July 2003 Language properties Attribute:value pairs in a structured containment hierarchy Properties – declarative (a data description language, not a programming language) – instance-inheritance (prototype-based); allow templates to be progressively redefined – flexible linking between attributes and values – value resolution can be delayed to deployment time – composition: build larger descriptions out of smaller parts Note: not XML-based, but can use XML as an input or output format if needed

page 11July 2003 Language example webServerTemplate extends { sfProcessHost “localhost”; port 80; useDB; } dbTemplate extends { userTable extends { columns 4; rows 3; } dataTable extends { columns 2; rows 5; } #include “wstemplate.sf” #include “dbtemplate.sf” sfConfig extends { commonPort 8080; ws1 extends webServerTemplate { sfProcessHost “webserver1.hpl.hp.com”; port ATTRIB commonPort; } ws2 extends webServerTemplate { sfProcessHost “webserver2.hpl.hp.com”; port ATTRIB commonPort; useDB LAZY ATTRIB db; } db extends dbTemplate { userTable:rows 6; } wstemplate.s f dbtemplate.s f webservice.s f

page 12July 2003 Language: composing descriptions E.g. Apache Web-server e.g. reliability & recovery thresholding e.g. servers, storage networking e.g. n-tiers including firewall rules e.g. 3-tier web service on Linux cluster front-end, Superdome back-end. fully-specified solution description e.g. SLAs software component descriptions application patterns composed: solution templates ‘farm’ patterns resource descriptions customer preferences parameterized

page 13July 2003 Engine Fully distributed engine comprised of SmartFrog daemons that run on each node – deploy service descriptions on any node and the engine will create components in the right place Daemons interpret SmartFrog descriptions – load components in the right sequence, in the right place, with correct configuration parameters – orchestrate the whole system by stepping components through their complete lifecycles Provides liveness and component-tree navigation mechanisms Many services can be deployed simultaneously using the same engine Service un-deployment allows clean service removal

page 14July 2003 Security Engine provides secure deployment based on PKI All nodes have certificates installed All component code and descriptions must be signed If node has valid certificate and code/descriptions are correctly signed, then deployment is allowed Single-level security model (all deployed services operate within the same trust domain for a given certificate set)

page 15July 2003 Component model SmartFrog components all implement a simple lifecycle that allows the engine to control them Allows systems of components to be started consistently import com.hp.SmartFrog.Prim.*; import java.rmi.*; public class MyPrim extends PrimImpl implements Prim, … { /* any component specific declarations */ public MyPrim() throws RemoteException {} public void sfDeploy() throws Exception { super.sfDeploy(); /* any component specific initialization code */ } public void sfStart() throws Exception { super.sfStart(); /* any component specific start-up code */ } public void sfTerminateWith(TerminationRecored tr) { /* any component specific termination code */ super.sfTerminateWith(tr); } /* any component specific methods */ }

page 16July 2003 Components SmartFrog has an extensible component set Some basic components form the core of the system, e.g.: – “Prim”: the primitive component – everything inherits from it – “Compound”: implements a “shared-fate” lifecycle for groups of components – Sequence: implements a sequential lifecycle for components Other components provide system services: discovery mechanisms, reliability building blocks, host scripting, JMX support, … Components are written as needed to support new services and wrap application components

page 17July 2003 Encapsulating non-SmartFrog components We don’t need to write our whole application/service in SmartFrog Java components -- it’s easy to encapsulate non-SmartFrog things, e.g. Apache web-server: wrapper components to – install and configure software (and remove) – stop and start the service We describe Apache and its required configuration in a SmartFrog description (using an Apache SF template) When deployed, the SF Apache component can install, configure, start, stop and remove Apache Apache can be used as a component of larger services

page 18July 2003 Implementation details Implemented completely in Java (using JDK1.4) – works across all standard Java platforms Uses Java RMI as transport Core system size – ~10K non-commented lines of Java – ~750 lines of description files – binaries: KBytes depending on services included

page 19July 2003 Research questions Rich research space spanning language, engine, and components – reliability (basic mechanisms, generic recovery patterns) – security – combining reliability and security – scalability – language specification – service debugging tools – standardized interfaces to resource management subsystems – standardized interfaces to node image deployment subsystems – generic system/service visualization tools

page 20July 2003 Business opportunity App developers – better enable apps for Grid and utility computer models – optimize application for specific hardware configurations – improve development (rapid deployment) Solution providers – improve app configuration, deployment, and management – more secure and reliable solutions – quicker time to deployment and adaptation Happier users – more stable systems – customization to specific needs – more control over solutions

page 21July 2003 Framework status Developing an open source (LGPL-licensed) release of the core framework (availability calendar ‘03) Earlier preview / alpha releases planned Deployed within HP/Agilent products Experience with: – Apache – BEA Weblogic – Maya CGI rendering engine – Oracle

page 22July 2003 Some related work Grid standards for Grid/utility computing Imaging / provisioning systems: Altiris, Rembo, Novadigm (+ many others) MS Dynamic Systems Initiative, MS Provisioning System Adaptive systems: ThinkDynamics, Corosoft IBM eLiza, autonomic computing Proprietary (product-specific) solutions in place today for installation / configuration / activation

page 23July 2003 Standardization goals Improve Grid applications – specification, deployment, configuration & life-cycle mgmt Standardize SF language, app templates, and SF engine Create multiple reference implementations of – engine – app templates (Application Server, JBOSS, WebLogic) Define SF integration with Grid Form a GGF WG

page 24July 2003 SmartFrog benefits Increased operational reliability Improved quality Assured correctness and consistency Increased security Reduced cost Improved customer experience

page 25July 2003 Summary SmartFrog framework (language, templates, engine) supports: – specification, deployment, configuration, and life-cycle mgmt Well suited for a utility computing model Addresses needs of complex distributed applications Business benefits for application developers, ISPs, & users We are seeking partners & lighthouse users – jointly standardize key elements – develop with reference implementations – create templates for specific applications – establish a wide technical community

HP logo