1. GORDA Kickoff meeting INRIA – Sardes project Emmanuel Cecchet Sara Bouchenak

2. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 2 - 07/10/2004 Outline è INRIA, ObjectWeb & Sardes è GORDA

3. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 3 - 07/10/2004 Budget: 120 M€ (tax not incl.) 900 permanent staff 400 researchers 500 engineers, technical and administrative 450 researchers from other organizations 700 Ph.D students 200 external collaborators 750 trainees, post-doctoral students, visiting researchers from abroad (universities or industry)) 6 Research Units A scientific force of 3,000 INRIA key figures Jan. 2003 A public scientific and technological research institute in computer science and control under the dual authority of the Ministry of Research and the Ministry of Industry INRIA Rhône-Alpes

4. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 4 - 07/10/2004 Itanium-2 processors 104 nodes (Dual 64 bits 900 MHz processors, 3 GB memory, 72 GB local disk) connected through a Myrinet network 208 processors, 312 GB memory, 7.5 TB disk Connected to the GRID Linux OS (RedHat Advanced Server) First Linpack experiments at INRIA (Aug. 03) have reached 560 GFlop/s Applications : Grid computing, classical scientific computing, high performance Internet servers, … iCluster 2

5. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 5 - 07/10/2004 ObjecWeb key figures è Open source middleware development è Based on open standard  J2EE, CORBA, OSGi… è International consortium  Founded by INRIA, Bull and FT R&D in 2001 è Academic partners  European universities and research centers è Industrial partners  RedHat, Suse, MySQL, …  NEC, Bull, France Telecom, Dassault, Cap Gemini, …

6. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 6 - 07/10/2004 Common Software & Architecture for Component Based Development OSGi Hardware & OS Applications J2EECCM... Dedicated Middleware Platform Networked Resources JVM TransactionsPersistence Deployment... Composition... Application Servers Database Connectivity Interoperability Tools FrameworksComponents... JOnAS OpenCCM OSCAR ProActive JORM/MEDOR JOTMOSCAR Fractal Think JORAM DotNetJ C-JDBC RmiJdbc Octopus Enhydra Zeus Kilim CAROL Jonathan Speedo RUBiS Bonita JAWE XMLC JMOB

7. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 7 - 07/10/2004 Sardes project è Distributed Systems group è Main research themes  Reflective component technology  Autonomous systems management è Applications areas  high-availability J2EE servers  dynamic monitoring, configuration and resource management in large scale distributed systems  (embedded system networks, ubiquitous computing) è Result dissemination by ObjectWeb

8. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 8 - 07/10/2004 Outline è INRIA, ObjectWeb & Sardes è GORDA

9. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 9 - 07/10/2004 Sardes experiences è Component-based open source middleware  ObjectWeb (http://www.objectweb.org) è J2EE application servers  JOnAS clustering (http://jonas.objectweb.org) è Database replication middleware  C-JDBC (http://c-jdbc.objectweb.org) è Benchmarking  RUBiS (http://rubis.objectweb.org)  TPC-W (http://jmob.objectweb.org)  CLIF (http://clif.objectweb.org) è Monitoring  LeWYS (http://lewys.objectweb.org)

10. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 10 - 07/10/2004 Common scalability practice Internet Web frontend App. server Well-known database vendor here Database Well-known hardware + database vendors here è Cons  Cost  Scalability limit

11. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 11 - 07/10/2004 Replication with shared disks Internet Web frontend App. server Database Disks Another well-known database vendor è Cons  still expensive hardware  availability

12. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 12 - 07/10/2004 Master/Slave replication Internet Web frontend App. server è Cons  consistency  failover time on master failure  scalability Master

13. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 13 - 07/10/2004 Internet è Database tier should be  scalable  highly available  without modifying the client application  database vendor independent  on commodity hardware Atomic broadcast-based replication Atomic broadcast

14. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 14 - 07/10/2004 Internet è JDBC compliant (no client application modification) è database vendor independent  JDBC driver required  heterogeneity support è no 2PC, no group communication between databases è group communication for controller replication only C-JDBC JDBC

15. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 15 - 07/10/2004 RAIDb - Definition è Redundant Array of Inexpensive Databases è better performance and fault tolerance than a single database, at a low cost, by combining multiple database instances into an array of databases è RAIDb levels offers various tradeoff of performance and fault tolerance

16. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 16 - 07/10/2004 RAIDb è Redundant Array of Inexpensive Databases  better performance and fault tolerance than a single database, at a low cost, by combining multiple database instances into an array of databases è RAIDb controller  gives the view of a single database to the client  balance the load on the database backends è RAIDb levels  RAIDb-0: full partitioning  RAIDb-1: full mirroring  RAIDb-2: partial replication  composition possible

17. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 17 - 07/10/2004 C-JDBC – Key ideas è Middleware implementing RAIDb è Two components  generic JDBC 2.0 driver (C-JDBC driver)  C-JDBC Controller è C-JDBC Controller provides  performance scalability  high availability  failover  caching, logging, monitoring, … è Supports heterogeneous databases

18. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 18 - 07/10/2004 C-JDBC – Overview

19. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 19 - 07/10/2004 Heterogeneity support è unload a single Oracle DB with several MySQL è RAIDb-2 for partial replication

20. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 20 - 07/10/2004 Inside the C-JDBC Controller Sockets JMX

21. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 21 - 07/10/2004 C-JDBC features è unified authentication management è tunable concurrency control è automatic schema detection è tunable replication: full partitioning, partial replication, full replication è caching: metadata, parsing, result with various invalidation granularities è various load balancing strategies è on-the-fly query rewriting for macros and heterogeneity support è recovery log for dynamic backend adding and failure recovery è database backup/restore using Octopus è JMX based monitoring and administration è graphical administration console

22. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 22 - 07/10/2004 Functional overview connect myDB connect login, password execute SELECT * FROM t

23. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 23 - 07/10/2004 Functional overview execute INSERT INTO t …

24. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 24 - 07/10/2004 Failures è No 2 phase- commit  parallel transactions  failed nodes are automatically disabled execute INSERT INTO t …

25. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 25 - 07/10/2004 Controller replication è Prevent the controller from being a single point of failure è Group communication for controller synchronization è C-JDBC driver supports multiple controllers with automatic failover jdbc:c-jdbc://node1:25322,node2:12345/myDB

26. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 26 - 07/10/2004 Controller replication jdbc:cjdbc://node1,node2/myDB Uniform total order reliable multicast

27. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 27 - 07/10/2004 Mixing horizontal & vertical scalability

28. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 28 - 07/10/2004 Lessons learned è SQL parsing cannot be generic è many discrepancies in JDBC implementations è minimize the use of group communications  IP multicast does not scale è notification infrastructure needed è users want  no single point of failure  control (monitoring, plug-able recovery policies, …)  no database vendor locking  no database modification è need for an exhaustive test suite è benchmarking accurately is very difficult  load injection requires resources  monitoring and exploiting results is tricky

29. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 29 - 07/10/2004 Sardes role in GORDA è provide input  GORDA APIs  group communication requirements  monitoring and management requirements è middleware implementation based on C-JDBC è dissemination effort  ObjectWeb  possible participation to JCP for JDBC extensions è hardware resources for experiments è eCommerce benchmarks

30. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 30 - 07/10/2004 Other interests è LeWYS (http://lewys.objectweb.org)  monitoring infrastructure  generic hardware/kernel probes for Linux/Windows  software probes: JMX, SNMP, …  monitoring repository è autonomic behavior  building supervision loops  self-healing clusters  self-sizing (expand or shrink)  SLAs

31. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 31 - 07/10/2004 Q without A è do we consider distributed query execution? è XA support? è cluster size targeted? è do we target grids or cluster of clusters?  reconciliation  consistency/caching  network architecture considered? è are relaxed or loose consistency options? è what will really cover the GRI?  do we impose a specific way of doing replication?  access to read-set/write-set difficult to implement with legacy databases è which workloads are considered? è which WP deals with backup/recovery? è licensing issues?

32. Q&A _________ Thanks to all users and contributors...

33. Bonus slides

34. INTERNALS

35. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 35 - 07/10/2004 Virtual Database è gives the view of a single database è establishes the mapping between the database name used by the application and the backend specific settings è backends can be added and removed dynamically è configured using an XML configuration file

36. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 36 - 07/10/2004 Authentication Manager è Matches real login/password used by the application with backend specific login/ password è Administrator login to manage the virtual database

37. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 37 - 07/10/2004 Scheduler è Manages concurrency control è Specific implementations for Single DB, RAIDb 0, 1 and 2 è Query-level è Optimistic and pessimistic transaction level  uses the database schema that is automatically fetched from backends

38. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 38 - 07/10/2004 Request cache è caches results from SQL requests è improved SQL statement analysis to limit cache invalidations  table based invalidations  column based invalidations  single-row SELECT optimization è request parsing possible in the C-JDBC driver  offload the controller  parsing caching in the driver

39. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 39 - 07/10/2004 Load balancer 1/2 è RAIDb-0  query directed to the backend having the needed tables è RAIDb-1  read executed by current thread  write executed in parallel by a dedicated thread per backend  result returned if one, majority or all commit  if one node fails but others succeed, failing node is disabled è RAIDb-2  same as RAIDb-1 except that writes are sent only to nodes owning the written table

40. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 40 - 07/10/2004 Load balancer 2/2 è Static load balancing policies  Round-Robin (RR)  Weighted Round-Robin (WRR) è Least Pending Requests First (LPRF)  request sent to the node that has the shortest pending request queue  efficient if backends are homogeneous in terms of performance

41. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 41 - 07/10/2004 Connection Manager è Connection pooling for a backend  Simple : no pooling  RandomWait : blocking pool  FailFast : non-blocking pool  VariablePool : dynamic pool è Connection pools defined on a per login basis  resource management per login  dedicated connections for admin

42. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 42 - 07/10/2004 Recovery Log è Checkpoints are associated with database dumps è Record all updates and transaction markers since a checkpoint è Used to resynchronize a database from a checkpoint è JDBCRecoveryLog  store information in a database  can be re-injected in a C-JDBC cluster for fault tolerance

43. SCALABILITY

44. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 44 - 07/10/2004 C-JDBC scalability è Horizontal scalability  prevents the controller to be a Single Point Of Failure (SPOF)  distributes the load among several controllers  uses group communications for synchronization è C-JDBC Driver  multiple controllers automatic failover jdbc:c-jdbc://node1:25322,node2:12345/myDB  connection caching  URL parsing/controller lookup caching

45. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 45 - 07/10/2004 C-JDBC scalability è Vertical scalability  allows nested RAIDb levels  allows tree architecture for scalable write broadcast  necessary with large number of backends  C-JDBC driver re-injected in C-JDBC controller

46. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 46 - 07/10/2004 C-JDBC vertical scalability è RAIDb-1-1 with C-JDBC è no limit to composition deepness

47. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 47 - 07/10/2004 C-JDBC vertical scalability è RAIDb-0-1 with C-JDBC

48. CHECKPOINTING

49. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 49 - 07/10/2004 Fault tolerant recovery log UPDATE statement

50. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 50 - 07/10/2004 Checkpointing è Octopus is an ETL tool è Use Octopus to store a dump of the initial database state è Currently done by the user using the database specific dump tool

51. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 51 - 07/10/2004 Checkpointing è Backend is enabled è All database updates are logged (SQL statement, user, transaction, …)

52. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 52 - 07/10/2004 Checkpointing è Add new backends while system online è Restore dump corresponding to initial checkpoint with Octopus

53. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 53 - 07/10/2004 Checkpointing è Replay updates from the log

54. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 54 - 07/10/2004 Checkpointing è Enable backends when done

55. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 55 - 07/10/2004 Making new checkpoints è Disable one backend to have a coherent snapshot è Mark the new checkpoint entry in the log è Use Octopus to store the dump

56. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 56 - 07/10/2004 Making new checkpoints è Replay missing updates from log

57. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 57 - 07/10/2004 Making new checkpoints è Re-enable backend when done

58. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 58 - 07/10/2004 Recovery è A node fails! è Automatically disabled but should be fixed or changed by administrator

59. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 59 - 07/10/2004 Recovery è Restore latest dump with Octopus

60. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 60 - 07/10/2004 Recovery è Replay missing updates from log

61. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 61 - 07/10/2004 Recovery è Re-enable backend when done

62. HORIZONTAL SCALABILITY

63. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 63 - 07/10/2004 Horizontal scalability è JGroups for controller synchronization è Groups messages for writes only

64. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 64 - 07/10/2004 Horizontal scalability è Centralized write approach issues è Issues with transactions assigned to connections

65. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 65 - 07/10/2004 Horizontal scalability è General case for a write query  3 multicast + 2n unicast

66. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 66 - 07/10/2004 Horizontal scalability è Solution: No backend sharing  1 multicast + n unicast [+ 1 multicast]

67. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 67 - 07/10/2004 Horizontal scalability è Issues with JGroups  resources needed by a channel  instability of throughput with UDP  performance scalability è TCP better than UDP but  unable to disable reliability on top of TCP  unable to disable garbage collection  ordering implementation is sub-optimal è Need for a new group communication layer optimized for cluster

68. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 68 - 07/10/2004 Horizontal scalability è JGroups performance on UDP/FastEthernet

69. USE CASES

70. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 70 - 07/10/2004 Budget High Availability è High availability infrastructure “on a budget” è Typical eCommerce setup è http://www.budget-ha.com

71. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 71 - 07/10/2004 OpenUSS: University Support System è eLearning è High availability è Portability  Linux, HP-UX, Windows  InterBase, Firebird, PostgreSQL, HypersonicSQL è http://openuss.sourceforge.net

72. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 72 - 07/10/2004 Flood alert system è Disaster recovery è Independent nodes synchronized with C-JDBC è VPN for security issues è http://floodalert.org

73. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 73 - 07/10/2004 J2EE benchmarking è Large scale J2EE clusters è http://jmob.objectweb.org Interne t emulated users

74. PERFORMANCE

75. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 75 - 07/10/2004 TPC-W è Browsing mix performance

76. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 76 - 07/10/2004 TPC-W è Shopping mix performance

77. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 77 - 07/10/2004 TPC-W è Ordering mix performance

78. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 78 - 07/10/2004 Result cache è Cache contains a list of SQL->ResultSet è Policy defined by queryPattern->Policy è 3 policies  EagerCaching: variable granularities for invalidations  RelaxedCaching: invalidations based on timeout  NoCaching: never cached RUBiS bidding mix with 450 clients No cacheCoherent cache Relaxed cache Throughput (rq/min)389241844215 Avg response time801 ms284 ms134 ms Database CPU load100%85%20% C-JDBC CPU load-15%7%

79. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 79 - 07/10/2004 Outline è Motivations è RAIDb è C-JDBC è Performance è Lessons learned è Conclusion

80. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 80 - 07/10/2004 Open problems è Partition of clusters è Users want control on failure policy è Reconciliation must also be user controlled

81. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 81 - 07/10/2004 LeWYS overview Monitoring Pump CPU probe disk probe pump thread DREAM Monitoring Pump CPU probe memory probe pump thread DREAM Monitoring Pump pump thread CPU probe network probe Observer Monitoring repository

82. LeWYS

83. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 83 - 07/10/2004 LeWYS components è Library of probes  hardware resources: cpu, memory, disk, network  generic sensors: SNMP, JMX, JVMPI, … è Monitoring pump  dynamic deployment of sensors  manages monitoring leases è Event channels  propagate monitored events to interested observers  allows for filtering, aggregation, content-based processing, … è Optional monitoring repository

84. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 84 - 07/10/2004 LeWYS design choices è Component-based framework  probes, monitoring pump, event channels  provides (re)configurability capabilities è Minimize intrusiveness on monitored nodes è No global clock  timestamp generated locally by pump è Information processing in DREAM channels

85. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 85 - 07/10/2004 Centralized monitoring using a monitoring repository (1) Monitoring Pump CPU probe memory probe pump thread DREAM Monitoring Pump CPU probe memory probe pump thread DREAM Monitoring Repository Monitoring DB storage thread query threads Monitoring Console event subscribe service

86. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 86 - 07/10/2004 Centralized monitoring using a monitoring repository (2) è Monitoring repository  stores monitoring information  service to retrieve monitoring information è Pros  DB allows for storing large amount of data  powerful queries correlate data from various probes at different locations resynchronize clocks  browsing history to diagnose failures  use history for system provisioning è Cons  requires a DB (heavy weight solution)

87. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 87 - 07/10/2004 Outline è J2EE Cluster è Group communications è Monitoring  motivations  LeWYS  implementation è Status & Perspectives

88. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 88 - 07/10/2004 Monitoring pump implemention Component ProbeFactory Monitoring Pump Thread Probe MonitoringPump Manager ChannelOut ProbeManager Binding Controller Probe Repository ProbeManager OutputManager Component PullPush Multiplexer MonitoringMumpManager OutputManager TimeStamp Probe Cache CachedProbe RMI

89. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 89 - 07/10/2004 Hardware Probes è Pure Java probes  using /proc  cost: 0.01ms/call (Linux) LinuxSolaris /proc Hardware resources Windows.DLLCC Linux C JNI cpu, mem, disk, net, kernel, …probes cpu, … probes ……

90. http://c-jdbc.objectweb.org/ - c-jdbc@objectweb.org 90 - 07/10/2004 Software Probes è Application level monitoring  JMX  ad-hoc è JVM LinuxSolaris JVM Hardware resources WindowsLinux JMXJVMPI SNMP, ad-hoc, … probes JVM probes JMX based probes