1. emmanuel.cecchet@inria.fr ObjectWeb Architecture Meeting, 25 sept 03 Our experience of open source

2. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr2 Outline è Why and how? è C-JDBC community è Architectural concerns è Conclusion

3. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr3 Why did we design ? è Scalability evaluation of J2EE servers  performance bounded by database even with a single server  how to compare middleware performance ?  how to evaluate clustering features in J2EE servers ? è Solutions  Large SMP machine: too expensive  Open source solution: do it yourself! è Features we wanted ordered by priority  scalability  on commodity hardware  using open source databases  fault tolerance (high availability + failover)  without modifying the client application

4. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr4 Internet EJBServlets Apache Databases How do we want to use ? è From small dynamic content web sites using a centralized open source database è To an end-to-end open source solution for large scale J2EE clustered application servers

5. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr5 Why users are using ? è JDBC standard è Open source solution è Features they want ordered by priority  fault tolerance (high availability + failover) [was 4/5]  using open source existing databases [was 3/5]  on commodity hardware [was 2/5]  administration tools [was not]  security [was not]  scalability [was 1/5]  without modifying the client application [was 5/5]

6. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr6 How users are using ? è Hard to really know è Just default settings! è Most common usage  existing applications (Tomcat/JBoss/JOnAS) with one MySQL/Postgres backend  add a second backend for fault tolerance and scalability è For things it was not designed for  write mostly workloads  distributed databases  hosting centers (administration tools missing)

7. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr7 Lessons learned è Users do not use it for what it was first designed for  advanced features are never used  concerned about ease of use and TCO è Default settings are important è Good technology is necessary but not sufficient  [administration] tools are needed  bugs are ok

8. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr8 Outline è Why and how? è C-JDBC community è Architectural concerns è Conclusion

9. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr9 Stats as of Sep 25, 03 è Downloads  total : > 5200 downloads since may 2003  last 30 days : > 1700 downloads  2nd most downloaded ObjectWeb project è Mailing lists  c-jdbc@objectweb.org: 93 subscribers c-jdbc@objectweb.org  c-jdbc-commits@objectweb.org: 16 subscribers c-jdbc-commits@objectweb.org è Web site  > 100.000 hits for september 03  US 35%, EU 25%, Canada 5%, Australia 5%, China 3%, Brazil 2%, India 2%, Japan 2%, …

10. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr10 Freshmeat.net è Links to projects è Users can subscribe to be notified of new releases è Need to register project in all possible categories to have good visibility è One release per week is a good timing è 3 new subscribers with every release Slashdot Weekly release Friday release out of town

11. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr11 How do we build a community? è Necessary features (but not sufficient)  open source  standard API  responsiveness on the mailing list è Visibility  Web: slashdot, TheServerSide, freshmeat, …  Conferences: JAX, Middleware, LinuxWorld, ICAR, … è Our weak points  no detailed design documentation  beta phase

12. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr12 How do we interact with the user community? è only one mailing list è being very responsive on the mailing list  reply even if we don’t have a response yet  no direct communication with team but share everything on the mailing list è benefit from engineers who work 1 week full- time to evaluate C-JDBC for their corporation è plan every feature request in the task list

13. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr13 How do we interact with the developer community? è single user/developer mailing list è post all design questions/choices on the mailing list  most users use default settings  hard to get feedback about usage è very permissive to accept new commiters  8 committers (3 outside ObjectWeb – 2 full time)  2 contributors who didn’t want to become commiters  no problem so far è involve people in testing

14. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr14 Lessons learned è Visibility  perpetual involvement  time consuming but necessary è Responsiveness to user queries  always on the mailing list  makes first impression for many users è Involve users in all decisions è Be open  source, CVS, contributions, patches, …

15. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr15 Outline è Why and how? è C-JDBC community è Architectural concerns  C-JDBC overview  Octopus integration  Clustering  Open problems è Conclusion

16. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr16 C-JDBC è Redundant Array of Inexpensive Databases (RAIDb)  RAIDb-0: full partitioning  RAIDb-1: full replication  RAIDb-2: partial replication è Two components  generic JDBC 2.0 driver (C-JDBC driver)  C-JDBC Controller è C-JDBC Controller provides  scheduler for concurrency control  load balancer for performance  recovery log for fault tolerance è Supports heterogeneous databases quick facts

17. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr17 Overview of

18. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr18 TPC-W Performance

19. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr19 Outline è Why and how? è C-JDBC community è Architectural concerns  C-JDBC overview  Octopus integration  Clustering  Open problems è Conclusion

20. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr20 Octopus integration è Use Octopus to store a dump of the initial database state è Currently done by the user using the database specific dump tool

21. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr21 Octopus integration è Backend is enabled è All database updates are logged (SQL statement, user, transaction, …)

22. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr22 Octopus integration è Add new backends while being online è Restore dump corresponding to initial checkpoint with Octopus

23. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr23 Octopus integration è Replay updates from the log

24. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr24 Octopus integration è Enable backends when done

25. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr25 Making new checkpoints è Disable one backend to have a coherent snapshot è Mark the new checkpoint entry in the log è Use Octopus to store the dump

26. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr26 Making new checkpoints è Replay missing updates from log

27. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr27 Making new checkpoints è Re-enable backend when done

28. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr28 Fault tolerance è A node fails! è Automatically disabled but should be fixed or changed by administrator

29. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr29 Fault tolerance è Restore latest dump with Octopus

30. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr30 Fault tolerance è Replay missing updates from log

31. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr31 Fault tolerance è Re-enable backend when done

32. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr32 Outline è Why and how? è C-JDBC community è Architectural concerns  C-JDBC overview  Octopus integration  Clustering  Open problems è Conclusion

33. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr33 Clustering è JavaGroups everywhere  performance and scalability study needed è Administration  integration of various consoles (Tomcat, JOnAS, …)  common vocabulary needed  JMX is standard Internet EJBServlets Apache Databases

34. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr34 Clustering continued … è Monitoring framework needed  health of the system  building efficient load balancing algorithms è JMX performance impact?  resource usage  design patterns  results to be published in JMOB è User console  text, Swing and/or HTTP ?  SQuirreL SQL or iSQL for JDBC console

35. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr35 Outline è Why and how? è C-JDBC community è Architectural concerns  C-JDBC overview  Octopus integration  Clustering  Open problems è Conclusion

36. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr36 Open problems è Partition of clusters è Users want control on failure policy è Reconciliation must also be user controlled

37. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr37 Open problems è Opening the architecture to the users  user defined strategies when a fault or exception occurs  which interfaces/callbacks to provide ? è Monitoring  needed for more accurate load balancing algorithms è Benchmarking  need automatic evaluation of clustered servers  platform available: new INRIA 208 itanium-2 cluster è Sun Test Suite  should help strenghening C-JDBC code  interoperability with J2EE servers

38. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr38 Fractal ? è Fractal version of C-JDBC è Benefits for C-JDBC  short term: good question !  long term: resource usage tracing è Benefits for Fractal  feedback  performance comparison with POJO version

39. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr39 Outline è Why and how? è C-JDBC community è Architectural concerns è Conclusion

40. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr40 Conclusion è Integration of “coarse grain” ObjectWeb components è Standard APIs (not OW defined) allows easy interaction with other components (OW or not) è Common tools needed for administration and monitoring

41. ObjectWeb Architecture Meeting – 25 sep. 2003 – Emmanuel.Cecchet@inria.fr41 Heterogeneity support è application already written for a specific [commercial] database è user defined rules for on-the-fly query rewriting to execute on heterogeneous backends