1. Aake Edlund, PhD KTH and SNIC, Sweden NordicCloud (Project Manager) NEW BalticGrid (Project Director) BalticGrid Innovation Lab and BalticCloud (Initiator, Manager) Baltic Cloud 1 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

2. 2 – Started 1 st of May, 2008, duration - 2 years – Continues the BalticGrid project (2005-2008) – 13 partners from Baltic States, Belarus, Poland, Sweden, Switzerland – Financing – 3 MEUR – Collaborating project for EGEE – Supporting NGIs and their collaboration Normalized computing hours for the BalticGrid project and that of the first six months of BalticGrid-II 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud BalticGrid - the Foundation

3. Motivation for BalticCloud When we started, cloud was considered immature, and only for startups and possibly prototyping. This suited us well, knowing that we had not been very successful in attracting bigger companies. Startup and prototyping focus 3 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

4. Initiated in January 20, 2009 A program for early stage companies in our region, by inviting them to a hands-on course in grid and cloud computing. – A course prepared and run by us, on our resources as well as on donated dito from e.g. Amazon. The aim with the course is to help early stage high-tech Internet based companies to try their services on new platforms, resulting in early proof of concepts and later exploitation of grid and cloud in the region. On top of the course we'll build a network of innovative companies in the region.” 4 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud The Vision A new way to attract startups bgin.wordpress.com

5. Motivation for BalticCloud -Low resource usage in BG. Need of elasticy - volume on demand - a way to handle capacity on the fly, to balance the usage -Too steep learning curve for grid users -Low acceptance within industry – so far -Complicated to adapt new applications, typically, need a “project” to do that Complement to current grid services 5 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

6. Motivation for BalticCloud One more, hopefully, easier access point to our BalticGrid resources. So, one more access point 6 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

7. Baltic Cloud How it all fits together BalticGrid Planned activities SA1-3, NA1- 4, JRA BGi - BalticGrid Innovation Lab BalticCloudCourses SME connectivity BC 7 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

8. BalticGrid asked one center per country to install a cloud instance SNIC (Swedish National Infrastructure for Computing) supported the coordination of the work Learning by doing - what's in it for eScience, really? Is industry interested? Started information gathering and sharing through our site; went to cloud expos and conferences Took early contact with open source cloud vendors, decided to focus on Eucalyptus 8 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud Practicalities & Learning by doing

9. First users First usage: 1.Virtual world installations (Wonderland), for the Immersive Education organization 2.Film rendering, for Estonian film company Virtual worlds and movies July 17, 20099ISSGC09 - Sophia Antipolis

10. Open source “cloud in a box” Our (main) choice: -Lead by Rich Wolski, UCSB. -Integrated with 10 BC 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

11. Will the open source alternatives be supported in a satisfactory way? “Not invented here” easily becomes “Not supported here” Early success open source alternatives branching off a private version – think Xen, Eucalyptus, Hadoop – which branch will get the best support? De facto lock-in, the bidding-on-a-dollar syndrome Major Concerns 11 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

12. Basics Cloud Interface Cloud Controller Cluster Controller Node Controller 12 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

13. Cloud controller Web service – Java Handles distribution of VM instance control events to clusters of resources Cloud controller SLA Engine Instance Scheduler Other Schedulers? 13 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud Cloud Interface Cloud Controller Cluster Controller Node Controller

14. Cluster controller Web service – C based Handles distribution of VM instance control events to individual of resources Investigation: what are the performance, invasiveness and functionality tradeoffs of various cloud networking strategies? Cluster controller Virtual Network Instance Scheduler Other Network/Scheduler solutions? 14 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud Cloud Interface Cloud Controller Cluster Controller Node Controller

15. Cloud interface Web service based – Written in Java Handles user event to cloud VM instance control translation Experiment: instance creation time Cloud interface EC2 compatible user interface Web-based admin interface Other interfaces? 15 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud Cloud Interface Cloud Controller Cluster Controller Node Controller

16. Status – Baltic Cloud 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud 16 See http://cloud.balticgrid.eu

17. Status – Estonia - NICPB (1/3) KVM-based setup with Eucalyptus interface Evaluating: other virtualisation technologies, e.g. vmware solution creation of the virtual private networks using Eucalyptus, OpenVPN and dynamic DNS - to be able to provide something like the recently rolled out Amazon Virtual Private Cloud (http://aws.amazon.com/vpc/) Usage: primarily NICPB ImmersiveEducation project with the developement and demo resources for running the Project Wonderland-based virtual worlds 17 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

18. Status – Estonia - NICPB (2/3) The KVM infrastructure is also used as a platform for the ongoing SME collaboration - on-demand render farm creation. The plans are to make it controllable by Eucalyptus, but so far it is not capable of supporting Windows machines out of the box (ongoing). Investigating the creation of the private virtual networks (using OpenVPN and dynamic DNS solutions) to be able to provide something like the recently rolled out Amazon Virtual Private Cloud (http://aws.amazon.com/vpc/). 18 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

19. Status – Estonia - NICPB (3/3) Storage: – testbed installation of the hadoopFS in NICPB with SRM interfaces. Trying to migrate the storage infrastructure from dCache to HDFS. – Wins: easier administration, cloud flavor - can run hadoop jobs much more efficiently. Problems: – Eucalyptus sw is still too alpha/pre-beta. The 1.6 _should_ be better, incl. support for multiple data centers (availability zones) and more robust operations, but it's not out yet. – Accounting of consumed resources is complicated – the virtualisation tools are not making an accent on that. 19 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

20. Status - Latvia - RTU Installation: Eucalyptus installation, xen, 4 cpu for users. HW updates expected (user request). Usage: Submitted simple tests, including students testing cloud Issues: System not quite stable. “imho so far not usable for serious project”. 20 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

21. Status - Belarus – UIIP NASB (1/3) Installation: – HW: 6 CPU, 24 cores (enough for present tasks); – Virtualisation: Xen; – Cloud technolgies: Eucalyptus, Walrus; – How to reach: https://80.94.168.123:8443/https://80.94.168.123:8443/ – Status: 3 GNU/Linux images available, commonly running from 2 to 10 instances. Usage: -verified by regular manual monitoring of hosts and services running within the cloud; -used by UIIP NASB grid team for designing dynamic grid resource reservation and adjustment mechanisms for UNICORE sites 21 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

22. Status - Belarus – UIIP NASB (2/3) Usage: pilot project of many users -we have prepared a pilot project proposal on deploying one of the advanced information systems of the State Committee for Science and Technologies of the Republic of Belarus (SCST) into the cloud infrastructure provided by the datacenter of the UIIP NASB. -The information system to be cloudified is created and administered on behalf of the SCST by Republican Centre for Technology Transfer (http://ictt.by). The system has thousands of visitors and hundreds of active users who regularly post and update their Technology Offers and Requests, disseminate news, upload technology transfer related applications, graphical and video data. -The primary goal of the pilot project is proving the concept of cloud computing being a good way to go not only for SMEs but for some governmental agencies as well. 22 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

23. Status - Belarus – UIIP NASB (3/3) Plans: UIIP NASB have plans for cloud usage growth, expansion, connection to the outside world and to other countries. [e.g. right now we are buying hardware for the expansion of our cloud infrastructure ] Issues: To increase cloud infrastructure stability we use custom kernel builds for Xen virtual machine images deployed in the cloud. 23 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

24. Status - Lithuania – VGTU Eucalyptus Cloud Scheme 24 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

25. Cluster installation is done using Rocks Clusters 5.2 (Chimichanga) VGTU cloud instance is available at https://euca.cloud.vgtu.lt:8443https://euca.cloud.vgtu.lt:8443 Eucalyptus v1.5.2 is installed XEN hypervisor is used for paravirtualization MANAGED network mode is configured i.e. machine instances can be accessed directly from internet Available two virtual machine images compiled manually: CentOS 5.3 (Base setup) SLC 53 (Scientific Linux Cern) (Base setup) Status - Lithuania – VGTU Cloud cluster setup 25 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

26. Frontend (Cluster controler, Cloud controler): 1 x Pentium 4 CPU, 2 GB RAM, 200 GB HDD, 1 Gbps Nodes (Node controler) 4 x Pentium 4 CPU, 2 GB RAM, 200 GB HDD, 1 Gbps 2 x Quad Qore CPU, 4 GB RAM, 320 GB HDD, 1 Gbps Availability zone DEBESELIS AVAILABILITYZONEDEBESELIS UP euca.cloud.vgtu.lt AVAILABILITYZONE|- vm typesfree / max cpu ram disk AVAILABILITYZONE|- m1.small0016 / 0016 1 128 3 AVAILABILITYZONE|- c1.medium0016 / 0016 1 256 5 AVAILABILITYZONE|- m1.large0016 / 0016 1 512 5 AVAILABILITYZONE|- m1.xlarge0016 / 0016 1 768 5 AVAILABILITYZONE|- c1.xlarge0008 / 0008 2 1024 5 Status - Lithuania – VGTU Available Resources Total 12 Cores 26 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

27. Status - Lithuania – VGTU Technical ISSUES (problems and limitations) In recent eucalyptus version 1.5.2 famous error: “Error 500 message expired” that prevented graphical tools like Elasticfox, Rightscale, Ylastic, EC2Dream from working with eucalyptus instance, was not fixed We have fixed this issue manually: Added this code to: HMACQuerySecurityHandler.java as a temporary fix -: tsFormat.setTimeZone(TimeZone.getTimeZone("EEST")); // Add this Line and import TimeZone class ts.setTime( tsFormat.parse( timestamp ) ); And then Eucalyptus rpms were rebuilt. 27 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

28. Status - Lithuania – VGTU We have installed and use ElasticFox for Eucalyptus management. 28 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

29. We have tried EC2Dream for Eucalyptus management as well. 29 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud Status - Lithuania – VGTU

30. “Cloud computing” activity is included into LitGrid programme 2009. Walrus based storage will be installed. Status - Sweden – PDC Installation (ongoing): 4 quad core machines, Eucalyptus, XEN Usage: for testing and learning, to prepare larger installation based on older HPC resources 30 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud Status - Lithuania – VGTU Nearest future

31. Status - Lithuania - VU Installation: Small eucaluptus intallation (might be increased if demand arrises), xen based, hardware is only 32 bit. Doing some tests with vmware and MS hyper-v Usage: Small tests, nothing in production Plans: Thinking of expanding (have to think more about which technology), maybe some services to some other institutions Issues: Current system is not very stable, some accidental crashes and system hangings were noticed. 31 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

32. Status - Poland – PSNC Installation: – two dual CPU machines (one for cloud controller one for cloud nodes), 2x dual core Intel Xeon, 8GM ram/each – 147 GB SAS hdd – two 1Gbit Eth network card/each virtualization : kvm or xen ( on demand ) – cloud stack : eucalyptus 1.5.2 – url : https://bgcloud.man.poznan.pl:8443, currently available only from PSNC network Usage: test stage Plans for usage and expansion : – as part of Baltic Cloud – for local users (PSNC developers and research team), education institution in Poznan – expansion according to load and utilization we can provide additional cloudnode controllers 32 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

33. Educational efforts (overview) VGTU: Bachelor theses on cloud computing (2009): “Deployment of Service Oriented IT Infrastructure” VGTU: Cloud computing is part of the MSc course “High performance systems” at VGTU BG: general presentations, e.g. at ISSGC09 NICPB: A course in IT college (in Tallinn) about the distributed systems. Cloud computing is part of the course, labs will be run on the BC resources NICPB: Tartu cloud and startup school – May, 2009 33 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

34. Expectations on 2010 Connecting the Baltic Cloud with the Nordic Cloud (Sweden, Norway, Denmark, Estonia, Latvia, Lithuania, Belarus, Poland) + Iceland(?) + … Availability zones – connecting the clouds Connecting with other clouds, e.g. in the Netherlands Increased user base – learning what is needed New examples of industry usage BC 34 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

35. ? 35 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud

36. Thank you! Aake Edlund ake.edlund@gmail.com www.pdc.kth.se/Members/edlund www.balticgrid.eu (Baltic Grid) cloud.balticgrid.eu (Baltic Cloud) bgin.wordpress.com (BG Innovation Lab) 36 4th Terena TF Storage Meeting, Copenhagen, 2009.Sept.15, BalticCloud