1. HP © 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice Introduction to SmartFrog v 1.01 HP Laboratories, Bristol, UK http://www.smartfrog.org

2. SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Introduction SmartFrog Technology Example Use Cases: Distributed Testing Scalable Webserver Utility Rendering

3. 321 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Our research focus “Programming the computing utility” How to describe the infrastructure and services that we want to run on the utility How to realise these descriptions, creating the infrastructure and deploying the services securely and reliably How to manage all through its lifecycle

4. 421 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Introduction

5. 521 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Deployment: why does it always go wrong? Because – it gets ignored – configuration is half the problem – nobody ever automates it – the tools are inadequate – it always goes wrong just before you go live Deployment is unreliable, unrepeatable and doesn't scale

6. 621 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Configuration causes the problems It’s the difference between configurations that hurt All those things that need to be consistent −configuration files −registry settings −router bindings −firewall −database −run-time values Trying to track down mismatches is hard

7. 721 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Choreography is “tricky”

8. 821 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Deployment is an ongoing activity development staging live system redistribution test system(s) Remote installation with custom options “near-live” configuration. High Availability, Scale on demand

9. today's processes neglect deployment Waterfall Iterative 921 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P.

10. bring deployment into the loop! Integrated deployment enables functional testing, and ensures that deployment gets automated 1021 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P.

11. HomeLinuxCluster extends System { sfProcessHost "pelvoux"; database:sfProcessHost "eiger"; happy:sfProcessHost "eiger"; hostname "deployment"; home LAZY PROPERTY user.home jboss.home home ++ "/jboss"; jboss.home home ++ "/jdk"; jboss.home home ++ "/examples"; } 1121 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Configuration is deployment Imagine a file that could declare the desired configuration state of a distributed system −Define templates and extend them to describe different configurations −Cross-referencing to eliminate duplication errors −Composition for bigger systems

12. 1221 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Create reality to match HomeLinuxCluster extends System { sfProcessHost "pelvoux"; database:sfProcessHost "eiger"; happy:sfProcessHost "eiger"; hostname "deployment"; home LAZY PROPERTY user.home jboss.home home ++ "/jboss"; jboss.home home ++ "/jdk"; jboss.home home ++ "/examples"; } configure the declared items manage their lifecycle adapt to failure or changing load

13. 1321 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. access tier web tier application tier database tier edge routers routing switches authentication, DNS, intrusion detect, VPN 1st level firewall 2nd level firewall load balancing switches web servers web page storage (NAS) database SQL servers storage area network (SAN) application servers files (NAS) switches min/max no. of web servers static data min no. of app servers deployed applications size of data, no. of tables initial data Template parameters transaction rate response times A complex template can cover everything

14. 1421 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. SmartFrog: core technology

15. 1521 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. SmartFrog (Smart Framework for Object Groups) A framework for describing, deploying and managing distributed management components. A description language for specifying configuration A runtime for realising the descriptions A component model for managing service lifecycle Components to deploy specific things sfConfig extends WebService { WebServer extends Apache { port 8080; } AppServer extends Jboss; }

16. 1621 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. SmartFrog language a declarative data description language configuration data is organised as hierarchical attribute-value pairs Templates: inheritance and aggregation Parameterisation: aids template reuse flexible variable linking component binding ‘transformation framework’ to extend the core language HomeLinuxCluster extends System { hostname "deployment"; home LAZY PROPERTY user.home jboss.home home ++ "/jboss"; jboss.home home ++ "/jdk"; jboss.home home ++ "/examples"; }

17. 1721 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Linking, Relationships and Binding Links: references to data in other parts of a description, deployed application or to external services −Eliminates duplication −Enables template parameterization of templates Functions: over literals or values specified by links Assertions: conformance predicates over literals, types and link values Early binding: links resolved before deployment starts Late (LAZY) binding: links resolved on live systems −enables access to dynamic data −provides RMI references to deployed components

18. 1821 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. SmartFrog components Configuration interpreters −Deployed as required by configuration −Interpret configuration (provide configuration semantics) −Carry out actions/tasks on resources −Maintain state −Incorporate Liveness tests Service descriptions are interpreted by components hosted by the runtime −Semantics are not implemented in the language −Can accommodate wide range of services and models

19. 1921 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Component model The SmartFrog language describes components using attributes −component code −component location −component-specific configuration data A SmartFrog component −has specified management interfaces −implements a pre-defined lifecycle −is written in Java

20. 2021 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Build distributed systems through composition systems are collections of components −software: processes, objects, … −data: disk files, … −hardware resources: switches, firewalls, … collections of components “collaborate” to achieve some goal, for which they must be appropriately organised −the required components must be defined −they must be correctly initialised, with appropriate attributes −they must be able to locate each other as required −they must exchange information regarding their state we call these organisations of components system configurations

21. 21 SmartFrog deployment engine SmartFrog Node SmartFrog Components Description / Code Repositories RMI Deploy Descriptions SmartFrog Daemon SmartFrog Node SmartFrog Components SmartFrog Daemon SmartFrog Node SmartFrog Components SmartFrog Daemon RMI Distributed, decentralized, secure deployment engine Loads and instantiates the components making up each service Shares configuration data between the components

22. SmartFrog Daemon SmartFrog Tool Chain Eclipse for programming components and building base component descriptions, and deploying them Ant tasks and command line tools SmartFrog runtime management tool SmartFrog Daemon

23. 2321 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. SmartFrog: Description Language >> Deployment Engine >>

24. 2421 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Usage: Distributed Testing

25. 25 How Systems are Tested Today IDE/Ant/Maven hosted test runners Continuous Integration servers JUnit is primary test platform Bridges to other processes (Cactus, Selenium Server) Local system Local network System testing is tacked on to unit testing

26. 2621 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Tomorrow Applications & tests run on multiple different hosts Hosts (real and virtual) are dynamically allocated Host and networks configured as part of the test Reports include output from all hosts and network Result presentation scales up. Data mining of multiple tests runs Runs as a background task under the CI server.

27. 27 SmartFrog for testing –set up application with different options –preconfigure the database –bring up Xen/VMware images for hosting –deploy the tests and collect the results –clean up Tests are just something else to deploy

28. 28 Components for running tests HttpUnitTests extends JUnitTestSuite { package "d1.webapp.test"; name "HttpUnitTests"; server.url TBD; sfProcessHost "client"; properties [ ["server.url",server.url], ["cactus.contextURL",server.url] ]; classes [ "EventFeedTest", "HappyTagTest", "IndexTest" ]; }

29. 29 Distributed Testing: methodology Allocate & configure test systems Deploy application across nodes Deploy tests on other nodes Collect and correlate results Try to understand what went wrong

30. 30 Test scenarios Application Single NodeMultiple Nodes Test Single Node Traditional testing client/server Multiple Nodes Write once, test everywhere -client side apps fully distributed application and tests

31. 31 Example: “Enterprise” Web Server

32. 32 Example: Interop testing of SOAP Systems

33. 33 How we test SmartFrog itself - SmartFrog daemon One ring of SmartFrog daemons deploys a new set Securityclass loading   

34. 34 Test Components

35. 35 Scalable Webserver

36. Programmable utility Built using Smart Framework for Object Groups (SmartFrog) SmartFrog describes, deploys and orchestrates distributed, component- based applications Service Portal allocates resources among multiple services Service Manager manages service on behalf of the Service User Service User Service Manager Service Portal Firewall Service Instance

37. 37 Focus areas The programmable utility −Service creation and management on utility infrastructure −Flexible and repeatable Service packaging −Service packages encapsulate and fully describe services −Automatic service deployment and ignition from packages −Service management and cleanup Adaptive behaviour −Services are designed for adaptive reconfiguration −Adaptation to workload changes and failures (for example) 2 example services: “Scalable Web Server” and “Frame Factory”

38. Service packaging Service Package includes: −software components −configuration −lifecycle −resource requirements −specific management interface Service Portal: −allocates required resources −configures resources −deploys service −manages service lifecycle −re-purposes resources after service termination WebS Service Manager Service Portal Firewall Dynamic Webserver Service WebS Service Package

39. Adaptive behaviour Initial number of resources Minimal number of resources Automatic replacement of failed resources Number of Web Server Nodes adjusted to meet demand Min/Max thresholds Web Server Manager Web Server Service Package Submit Deploy Service Utility Resource Manager Utility Provider

40. Demo Setup CPU-M controller management node web server nodes service template web template automatically deploy and terminate as required load generator template load balancer node r001s007e01 sfdemolaptop r001s006e01 r001s008e01 r001s009e01 load generator node

41. The Components : key CPU-M controller a graphing component – polls for data optionally with threshold lines a bar-chart component – polls for data a data source component wrapping Linux “top” application, extracting CPU load Apache installation workflow Apache management component starts and monitors apache threads providing a thread-count data source a load-balancer component provides a data source of connections handled in a specific period a main controller component manages instances of template based on measurements and thresholds a load generator – creating http requests according to configured settings a slide-bar component, can be used to control the load-generator

42. Utility Rendering: Frame Factory SE3D

43. page 43 Adaptive Services Automated service creation on utility infrastructure Automated service adaptation −Workload variation −Resource variation −Failures Driven by business objectives Complete service lifecycle management −Creation −Adaptation −Evolution −Removal

44. 44 Benefits Reduce time-to-service Reduce human error Reduce operator intervention Reduce impact of failures Increase resource utilization Increase return on IT Increase business agility Align resource allocation with business priorities

45. 45 Demo services Frame Factory Service Package Submit Services Deploy/Ignite Services Web Server Service Package Digital AnimatorsUtility Provider

46. 46 Service 1: Frame Factory Frame Factory Service User Frame Factory Service Manager Web Server Manager Frame Factory Service Package Submit Deploy Service Return Running Service Handle Utility Resource Manager Digital Animators Utility Provider

47. User’s Perspective 1. Create input data in Maya 2. Upload data to FF 3. Manage multiple snapshots of input data 4. Submit rendering jobs against snapshots 5.With Mercado purchase and manage rendering power 6. Download rendered frames Repeat until your film is perfect!

48. page 48April 2003 The Programmable Utility utility computing for animated film production | hp labs

49. Origins Frame Factory Elephant Store Market-Based Resource Allocation Management by Business Objectives SmartFrog/Anubis Security RIT Projects 2003 2004/5 Programs ‘The Painter’

50. HP © 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice Q&A http://www.smartfrog.org For more information:

51. HP © 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice SmartFrog: description language BACK >>

52. 5221 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. SmartFrog language a declarative data description language configuration data is organised as hierarchical attribute-value pairs supports templates: −inheritance (through prototyping) −extend, override, combine supports parameterisation: −To fill in templates with real values supports flexible variable linking supports component binding supports a ‘transformation framework’ to extend the core language –functions, validations, …

53. 5321 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. language example webServerTemplate extends { sfProcessHost “localhost”; port 80; useDB; } system extends { ws1 extends webServerTemplate { sfProcessHost “15.144.59.34”; } ws2 extends webServerTemplate { sfProcessHost “15.144.59.64”; port 8088; } db extends dbTemplate { userTable:rows 6; } dbTemplate extends { userTable extends { columns 4; rows 3; } dataTable extends { columns 2; rows 5; }

54. 54 Language example: prototyping dbTemplate extends { userTable extends { columns 4; rows 3; } dataTable extends { columns 2; rows 5; } db.sf webServerTemplate extends { sfProcessHost “localhost”; port 80; useDB; } webserver.sf system extends { ws1 extends webServerTemplate { } ws2 extends webServerTemplate { } db extends dbTemplate { } to build your own system you can use existing templates by: - extending them by adding attributes and values - extending them by overwriting prior values #include “db.sf”; #include “webserver.sf”; sfProcessHost “15.144.59.34”; sfProcessHost “15.144.59.67”; port 8088; userTable:rows 6; - importing them, new values type “backup”; version “Oracle8.2”; new attributes and using them in your description your own description is now also a prototype system.sf … system extends { … }

55. 55 simple, containment naming tree based linking “LAZY” means “resolve after deployment” system extends { commonPort 8088; ws1 extends webServerTemplate { sfProcessHost “15.144.59.34”; port ATTRIB commonPort; useDB LAZY ATTRIB db; } ws2 extends webServerTemplate { sfProcessHost “15.144.59.64”; port ATTRIB commonPort; } db extends dbTemplate { fooTables:rows 6; } Language example: linking and binding additional mechanisms are provided to link attributes (host-based, slp) new mechanisms may be added (jini, jndi, …)

56. 5621 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. notation: example - transformations fetch extends { phase.function “com.hp.openview.Fetch”; url; // set this to the url of the text } concat extends { phase.function “com.hp.SmartFrog.Parser.Functions.concatenate”; } system extends { server “serrano.hpl.hp.com”; file “config.sf”; } text extends fetch { url extends concat { -- “http://”; -- ATTRIB server; -- “/”; -- ATTRIB file; } text extends fetch { url extends concat { -- “http://”; -- “serrano.hpl.hp.com”; -- “/”; -- ATTRIB file; } text extends fetch { url extends concat { -- “http://”; -- “serrano.hpl.hp.com”; -- “/”; -- “config.sf”; } text extends fetch { url “http://serrano.hpl.hp.com/config.sf”; } text “whatever text was in the remote file…”;

57. HP © 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice SmartFrog: deployment engine BACK >>

58. 58 deploying a single component Server extends Compound { sfClass “MyServer”; port 2861; database extends DBPrim { … } … } MyServer.class server.sf the deployment process is handled by an instance of a ComponentDeployer (by default) the component deployer will create an instance of MyServer.class according to the sf description

59. 5921 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Host A SF Daemon (root) theServer deploying a single component smartfrog allows the user to deploy components in multiple ways, without having to change the components’ code by default a component is deployed in the local root daemon #include “server.sf”; sfConfig extends Compound { theServer extends Server { port 2867; …… }

60. 6021 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Host A serverDaemon SF Daemon (root) theServer deploying a single component SmartFrog allows the user to deploy components in multiple ways, without having to change the components’ code #include “server.sf”; sfConfig extends Compound { theServer extends Server { sfProcessName “serverDaemon”; port 2867; } by default a component is deployed in the local root daemon but you can ask for it to be deployed in a separate daemon ( / process / jvm)

61. 6121 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. deploying a single component smartfrog allows the user to deploy components in multiple ways, without having to change the components’ code by default a component is deployed in the local root daemon but you can ask for it to be deployed in a separate daemon #include “server.sf”; sfConfig extends Compound { theServer extends Server { sfProcessHost “hostB”; port 2867; } or on a separate host (in which case although the deployment was requested on hostA, the description creates a server on hostB) Host A SF Daemon (root) Host B SF Daemon (root) theServer

62. 6221 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. binding components with smartfrog smartfrog allows the user to bind components in a flexible and transparent manner write your components once, bind them any way you like afterwards an example : a client – server pair of components Client extends Prim { sfClass “MyClient”; port 4562; … } MyClient.class Server extends Compound{ sfClass “MyServer”; myDatabase extends DBPrim { … } … } in the client description, the link to the server can be specified in different ways. //myServer LAZY ATTRIB … ; client.sf server.sf MyServer.class

63. 6321 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. binding components with smartfrog Client extends Prim { sfClass “MyClient”; port 4562; … } MyClient.class Server extends Compound{ sfClass “MyServer”; myDatabase extends DBPrim { … } … } MyServer.class at runtime (for example during its deploy phase) the client asks the smartfrog daemon for a link to the server //myServer LAZY ATTRIB … ; MyServer server = (ServerIF) sfFind (“myServer”) ; smartfrog allows the user to bind components in a flexible and transparent manner –write your components once, bind them any way you like afterwards an example : a client – server pair of components

64. 6421 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Host A SF Daemon theServer theClient myServer direct binding the two components may be directly bound in a single description: #include “client.sf”; #include “server.sf”; sfConfig extends Compound{ theServer extends Server; theClient extends Client { myServer LAZY ATTRIB theServer; } the binding is exactly the same, and the runtime code of the client does not change,

65. 6521 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Host A SF DaemonB SF DaemonA theServer theClient myServer direct binding #include “client.sf”; #include “server.sf”; sfConfig extends Compound{ theServer extends Server { sfProcessName “DaemonA”; } theClient extends Client { sfProcessName “DaemonB”; myServer LAZY ATTRIB theServer; } the binding is exactly the same, and the runtime code of the client does not change, even if the components are deployed in different daemons … the two components may be directly bound in a single description:

66. 6621 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Host B Host A SF Daemon theServer theClient myServer direct binding #include “client.sf”; #include “server.sf”; sfConfig extends Compound{ theServer extends Server { sfProcessHost “hostA”; } theClient extends Client { sfProcessHost “hostB”; myServer LAZY ATTRIB theServer; } the binding is exactly the same, and the runtime code of the client does not change, even if the components are deployed in different hosts … the two components may be directly bound in a single description:

67. 6721 May, 2015 SmartFrog © 2007 Hewlett-Packard Development Company, L.P. Host B Host A SF Daemon theClient myServer decoupled binding if the server and client are not part of the same description, only the link has to change #include “server.sf”; sfConfig extends Compound { sfProcessComponentName “myServer”; theServer extends Server; } #include “client.sf”; sfConfig extends Compound { theClient extends Client { myServer LAZY HOST hostA:myServer:theServer; } SF Daemon theServer the server is the first component deployed the client is deployed afterwards; only the value of the link has changed, the client’s code remains unchanged

68. HP © 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice SmartFrog: end BACK >> http://www.smartfrog.org For more information: