Java and Databases PPJDG, May 2004 Chris Smith, MindIQ

Overview The Persistence Problem The Persistence Problem Object/Relational Mapping Object/Relational Mapping Persistence in EJBs Persistence in EJBs Hibernate Hibernate JDO JDO Object Databases Object Databases Wrap-up Wrap-up

The Persistence Problem

Why Worry About Persistence? Persistence is one of the most common requirements of modern applications. Persistence is one of the most common requirements of modern applications. More money is often spent on persistence than the rest of the application put together! More money is often spent on persistence than the rest of the application put together! Persistence is generally the major performance limiter for enterprise applications. Persistence is generally the major performance limiter for enterprise applications. Java developers spend a lot of time doing the same thing over and over. Java developers spend a lot of time doing the same thing over and over.

Basic Persistence Approaches Persistence can be solved by: Persistence can be solved by: Avoiding a database Avoiding a database Can be made easy to do (serialization) Can be made easy to do (serialization) Fine for transient local data, but doesnt scale Fine for transient local data, but doesnt scale Writing JDBC by hand Writing JDBC by hand A lot of repetition in CRUD operations A lot of repetition in CRUD operations Possible ad-hoc automation of common tasks Possible ad-hoc automation of common tasks Third-party frameworks and services Third-party frameworks and services Easier, and often a better theoretical basis Easier, and often a better theoretical basis Often some performance cost, but not much Often some performance cost, but not much

Persistence Frameworks Born of the concept that much persistence work can be automated… Born of the concept that much persistence work can be automated… Buyer (or free software user) beware: Buyer (or free software user) beware: Easy to write a 90% functional framework Easy to write a 90% functional framework Easy to think you can solve this problem Easy to think you can solve this problem Hard to really solve persistence problems Hard to really solve persistence problems

Goals and Concerns Performance Performance Scalability Scalability Flexibility Flexibility Transparency Transparency Fidelity to Relational Model Fidelity to Relational Model Fidelity to Object Model Fidelity to Object Model Legacy Data Legacy Data Cross-application and cross-platform access Cross-application and cross-platform access

Enabling Technologies Explicit Persistence Explicit Persistence Runtime Introspection of Code (Reflection) Runtime Introspection of Code (Reflection) Code Generation Code Generation Bytecode Postprocessing Bytecode Postprocessing Proxies and Polymorphism Proxies and Polymorphism

Object/Relational Mapping

Definition: O/R Mapper An O/R mapper bridges between the relational and object models of data. An O/R mapper bridges between the relational and object models of data. Loads from relational database to objects Loads from relational database to objects Saves from objects to relational database Saves from objects to relational database Relational Model: A strict mathematical model of information used by most DBMS packages. Relational Model: A strict mathematical model of information used by most DBMS packages. Object Model: A looser, more familiar information model found in application programming languages. Object Model: A looser, more familiar information model found in application programming languages.

The Relational Model - Basics Two kinds of relational terminology: Two kinds of relational terminology: Snobby pretentious words Snobby pretentious words Practical everyday usage words Practical everyday usage words Snobby Word Normal Word RelationTable TupleRow Attribute Column or Field DomainType

The Relational Model - Joins Relational entities are associated by joining tables. Relational entities are associated by joining tables. Fast for arbitrary searching of data Fast for arbitrary searching of data Slow for navigating intrinsic associations Slow for navigating intrinsic associations This favors a very course entity model This favors a very course entity model In other words, keep as much information as possible in one table. In other words, keep as much information as possible in one table.

The Object Model Object models vary between languages. Object models vary between languages. Not as strict as the relational model. Not as strict as the relational model. In general, though, objects have: In general, though, objects have: Identity Identity State State Behavior Behavior Dont worry about modeling behavior, but identity and state are both critical! Dont worry about modeling behavior, but identity and state are both critical!

Mapping State In general, its easiest to map an objects state to fields in a table. In general, its easiest to map an objects state to fields in a table. Potential problems: Potential problems: Granularity: object models should generally be finer-grained than relational models Granularity: object models should generally be finer-grained than relational models Associations: relationships between entities are expressed very differently Associations: relationships between entities are expressed very differently Collections or Foreign Keys Collections or Foreign Keys

Mapping Identity Objects have identity thats independent of their attributes or fields. Objects have identity thats independent of their attributes or fields. This is foreign to the relational world. This is foreign to the relational world. Solutions: Solutions: Give up identity, and use a natural key Give up identity, and use a natural key Invent a surrogate key to represent identity Invent a surrogate key to represent identity

Another Stick in the Gears A very important distinction: A very important distinction: Databases model a complete set of data. Databases model a complete set of data. Applications use a subset of data. Applications use a subset of data. A good O/R mapper will let you build a simplified map to just that data that your application needs. A good O/R mapper will let you build a simplified map to just that data that your application needs.

Persistence in EJBs

Entity Bean Persistence EJB provides two options for persisting entity beans: EJB provides two options for persisting entity beans: BMP - Bean Managed Persistence BMP - Bean Managed Persistence a.k.a., If you want it done right… a.k.a., If you want it done right… Not a persistence framework Not a persistence framework CMP – Container Managed Persistence CMP – Container Managed Persistence A somewhat rigid and inflexible persistence framework for entity beans. A somewhat rigid and inflexible persistence framework for entity beans.

EJB-CMP Vital Signs Persistence only for EJB entity beans Persistence only for EJB entity beans Concrete classes are generated at runtime from abstract bean classes Concrete classes are generated at runtime from abstract bean classes Associations declared in deployment descriptors and managed with CMR Associations declared in deployment descriptors and managed with CMR EJB-QL query language for data EJB-QL query language for data

EJB-CMP Advantages Easy to use, if youre already in an entity bean environment. Easy to use, if youre already in an entity bean environment. Integrated into your application server. Integrated into your application server. No redundant declarations for persisting bean relationships. No redundant declarations for persisting bean relationships. Able to take advantage of container-private knowledge about transaction management. Able to take advantage of container-private knowledge about transaction management. If youve already bought WebLogic, you may as well use it! If youve already bought WebLogic, you may as well use it!

EJB-CMP Disadvantages No standard way to map beans to legacy data from an existing database. No standard way to map beans to legacy data from an existing database. Some containers provide this ability Some containers provide this ability Some containers dont Some containers dont No inheritance/polymorphic relationships No inheritance/polymorphic relationships Portability issues moving between application servers. Portability issues moving between application servers.

EJB-CMP Disadvantages II Little flexibility in data representation. Little flexibility in data representation. And the biggie: And the biggie: You have to be writing an EJB entity bean! You have to be writing an EJB entity bean!

EJB-CMP Performance This is a raging debate. This is a raging debate. App Server vendors claim (unsurprisingly) that CMP performs fine. BEA says better than is possible with BMP. (huh?!?) App Server vendors claim (unsurprisingly) that CMP performs fine. BEA says better than is possible with BMP. (huh?!?) General experience is that CMP performance is bad, but thats possibly related to memories of earlier versions. General experience is that CMP performance is bad, but thats possibly related to memories of earlier versions. As always, the truth is in the middle. As always, the truth is in the middle. And will depend on your app server! And will depend on your app server!

EJB-BMP Persistence Bean Managed Persistence is the other option for EJB entity beans. Bean Managed Persistence is the other option for EJB entity beans. BMP is not a persistence framework. You can: BMP is not a persistence framework. You can: Write your own JDBC by hand Write your own JDBC by hand Use another persistence framework that works with EJB-BMP (such as Hibernate) Use another persistence framework that works with EJB-BMP (such as Hibernate) The latter approach has a nice separation- of-concerns ring to it. The latter approach has a nice separation- of-concerns ring to it.

Hibernate

Hibernate Vital Signs Data persistence service for ordinary Java objects and classes. Data persistence service for ordinary Java objects and classes. Associations via Java Collections API Associations via Java Collections API O/R mapping defined in an XML file O/R mapping defined in an XML file Persistent operations via an external net.sf.hibernate.Session object Persistent operations via an external net.sf.hibernate.Session object HQL – An object-relational query language HQL – An object-relational query language Basic tool set for common tasks Basic tool set for common tasks

Persistent Classes Look a lot like ordinary Java classes. Look a lot like ordinary Java classes. Some special considerations, though. Some special considerations, though. Requirements: Requirements: JavaBeans accessors and mutators JavaBeans accessors and mutators No-argument constructor No-argument constructor Use of Collections interfaces Use of Collections interfaces List, not ArrayList List, not ArrayList Dont rely on null collections or elements Dont rely on null collections or elements

Entities and Components Entities in Hibernate represent entity tables in the database. Entities in Hibernate represent entity tables in the database. Recall that the relational model encourages entities to be course-grained. Recall that the relational model encourages entities to be course-grained. Components are dependent objects Components are dependent objects A customer record in a database may have columns called address, city, state, etc. A customer record in a database may have columns called address, city, state, etc. The Hibernate mapping may use a component class called Address to encapsulate those fields. The Hibernate mapping may use a component class called Address to encapsulate those fields.

Collections for Associations Many kinds of collections: Many kinds of collections: Set, Map, List, array, and Bag (using List) Set, Map, List, array, and Bag (using List) Even SortedSet and SortedMap Even SortedSet and SortedMap All implemented with foreign keys and/or association tables in the database All implemented with foreign keys and/or association tables in the database Hibernate is strict about collections: Hibernate is strict about collections: Sometimes common to use List as a convenience for unordered collections Sometimes common to use List as a convenience for unordered collections This wont work in Hibernate; Set is better This wont work in Hibernate; Set is better

Mapping with XML Documents Each Hibernate persistent class should have an XML mapping document. Each Hibernate persistent class should have an XML mapping document. Defines: Defines: Persistent properties Persistent properties Relationships with other persistent classes Relationships with other persistent classes Writing these mapping files is the core Hibernate persistence task. Writing these mapping files is the core Hibernate persistence task. … but there are tools for other approaches … but there are tools for other approaches

Using the Session Persistent operations are available via a net.sf.hibernate.Session object. Persistent operations are available via a net.sf.hibernate.Session object. Session.load Loads an object from the database Session.get Same, but the object may not exist Session.save Adds a new object to the database Session.delete Deletes an object from the database Session.find Search for objects by an HQL query Session.filter Get a subset of some collection Session.flush Flush local changes out to the database

HQL – Hibernate Query Language HQL is used for building queries to find or filter data from the database. HQL is used for building queries to find or filter data from the database. Looks like SQLs SELECT at first glance Looks like SQLs SELECT at first glance Differences from SQL: Differences from SQL: Its only used for searching, not updating. Its only used for searching, not updating. It understands inheritance polymorphism, and object-oriented ownership of associations. It understands inheritance polymorphism, and object-oriented ownership of associations. Most pieces (even the SELECT clause) are optional in at least some situations! Most pieces (even the SELECT clause) are optional in at least some situations!

HQL - Basics Simplest possible HQL query: Simplest possible HQL query: from Employee from Employee Returns all employees in the database Returns all employees in the database HQL implements the four ANSI join types, plus Oracle-style Cartesian joins. HQL implements the four ANSI join types, plus Oracle-style Cartesian joins. Clauses: Clauses: SELECT (optional) SELECT (optional) FROM (required, except with Session.filter ) FROM (required, except with Session.filter ) WHERE (optional) WHERE (optional) Other: ORDER BY, GROUP BY, HAVING, … Other: ORDER BY, GROUP BY, HAVING, …

HQL – Complex Queries HQL supports subqueries and correlated subqueries… HQL supports subqueries and correlated subqueries… If the underlying database does. If the underlying database does. That means no MySQL (for now?) That means no MySQL (for now?) Named or positional parameters: Named or positional parameters: Use createQuery to build the query Use createQuery to build the query Methods on Query set the parameters Methods on Query set the parameters from Employee where salary > :minSalary from Employee where salary > :minSalary

Hibernate Tools Plain Hibernate means writing XML mapping. Plain Hibernate means writing XML mapping. Other options include: Other options include: hbm2ddl Generate DDL for database directly from Hibernate mapping files hbm2java Generate Java source file directly from Hibernate mapping files Middlegen Generate Hibernate mapping from an existing database schema AndroMDA Generate Hibernate mapping from UML XDoclet Generate Hibernate mapping from annotations in source code

Hibernate Sample

Hibernate Summary XML files define the O/R mapping. XML files define the O/R mapping. Ordinary objects, but with some rules. Ordinary objects, but with some rules. Associations via the Collections API. Associations via the Collections API. HQL language for queries. HQL language for queries.

Java Data Objects (JDO)

JDO Vital Signs API for ordinary Java object persistence API for ordinary Java object persistence Successor to ODMG API Successor to ODMG API Generally uses bytecode enhancement to provide persistence (with controversy…) Generally uses bytecode enhancement to provide persistence (with controversy…) Persistence provided through a PersistenceManager instance Persistence provided through a PersistenceManager instance Managed (via JCA) or local environments Managed (via JCA) or local environments Mapping to database is not specified! Mapping to database is not specified!

JDO Status JDO is an official Java Specification JDO is an official Java Specification JSR-12: Java Data Object (JDO) JSR-12: Java Data Object (JDO) But, not adopted by database vendors But, not adopted by database vendors Limited to ODBMS vendors only Limited to ODBMS vendors only Most deployments use third-party O/R mappers Most deployments use third-party O/R mappers IBM, BEA, and Oracle recently banded together to kill the 2.0 release of the spec. IBM, BEA, and Oracle recently banded together to kill the 2.0 release of the spec. But they failed But they failed

JDO Step-by-Step Write persistent classes to represent data Write persistent classes to represent data Enhance the classes with a JDO enhancer Enhance the classes with a JDO enhancer Obtain an instance of PersistenceManager Obtain an instance of PersistenceManager Obtain PersistenceCapable instances Obtain PersistenceCapable instances Modify JDO objects obtained Modify JDO objects obtained Store and commit results Store and commit results

Persistent Classes Implement the PersistenceCapable interface, either: Implement the PersistenceCapable interface, either: Because they were enhanced, OR Because they were enhanced, OR Because they were written that way (rare) Because they were written that way (rare) Cant contain references to classes that are not persistence-capable Cant contain references to classes that are not persistence-capable Certain system classes are considered capable, though. Certain system classes are considered capable, though. May use Set – but List and Map collection types are optional for the implementation! May use Set – but List and Map collection types are optional for the implementation!

First/Second Class Objects First-class objects (FCOs): First-class objects (FCOs): Are independently persistent entities Are independently persistent entities Can exist on their own in the database Can exist on their own in the database Can be associated with several other entities Can be associated with several other entities Second-class objects (SCOs): Second-class objects (SCOs): Are part of their containing FCO Are part of their containing FCO Cannot exist on their own on the database Cannot exist on their own on the database Must belong to one specific FCO Must belong to one specific FCO Reminds you of Hibernates components? Reminds you of Hibernates components? Often theres no guarantee of SCO vs. FCO Often theres no guarantee of SCO vs. FCO

Persistent Fields Fields are persisted – not JavaBeans properties as in Hibernate. Fields are persisted – not JavaBeans properties as in Hibernate. May be FCOs or SCOs. May be FCOs or SCOs. May be primitives, class-typed or interface-typed references. May be primitives, class-typed or interface-typed references. References must be to persistence- capable instances… References must be to persistence- capable instances… including system classes that are persistence- capable from the JDO implementation. including system classes that are persistence- capable from the JDO implementation.

Bytecode Enhancement Classes are made persistence-capable is bytecode enhancement. Classes are made persistence-capable is bytecode enhancement. Helps objects determine what fields changed. Helps objects determine what fields changed. Provides support for instance-pooling. Provides support for instance-pooling. And more, of course. And more, of course. There are alternatives: There are alternatives: Preprocessing: enhance before compiling Preprocessing: enhance before compiling Code generation: create pre-enhanced code Code generation: create pre-enhanced code Enhanced classes implement PersistenceCapable Enhanced classes implement PersistenceCapable

Getting a PersistenceManager A JDO application starts by obtaining a persistence manager instance. A JDO application starts by obtaining a persistence manager instance. This is done via a PersistenceManagerFactory. This is done via a PersistenceManagerFactory. Can be obtained with JDOHelpers getPersistenceManagerFactory method Can be obtained with JDOHelpers getPersistenceManagerFactory method Pass a properties object with parameters Pass a properties object with parameters Required parameter: javax.jdo.PersistenceManagerFactoryClass Required parameter: javax.jdo.PersistenceManagerFactoryClass

The PersistenceManager The entry point to persistence operations The entry point to persistence operations Used to: Used to: Add/delete/detach objects from the database Add/delete/detach objects from the database Retrieve specific objects by ID Retrieve specific objects by ID Build extents of objects from the database Build extents of objects from the database Build queries to filter objects from extents Build queries to filter objects from extents Obtain the current transaction for operations Obtain the current transaction for operations Manage the persistent object cache Manage the persistent object cache

Using the PersistenceManager makePersistent(Object)deletePersistent(Object) getObjectById(Object,boolean)makeTransient(Object) getObjectId(Object)currentTransaction() refresh(Object)retrieve(Object) evict(Object)currentTransaction() getExtent(Class,boolean)newQuery() newQuery(Object)newQuery(Class) newQuery(…)close()

PersistenceCapable Interface PersistenceCapable is intended as an internal interface PersistenceCapable is intended as an internal interface not used by application not used by application Instead, use JDOHelper static methods: Instead, use JDOHelper static methods: getPersistenceManager getPersistenceManager makeDirty/isDirty makeDirty/isDirty getObjectId getObjectId isPersistent/isTransactional/isNew/isDeleted isPersistent/isTransactional/isNew/isDeleted Changes to objects saved automatically! Changes to objects saved automatically!

JDOQL JDO uses query language JDOQL. JDO uses query language JDOQL. Based more on Java than SQL… Based more on Java than SQL… But some SQL syntax is still there. But some SQL syntax is still there. JDOQL fragments fit in specific locations: JDOQL fragments fit in specific locations: Query.setFilter (like SQL WHERE clause) Query.setFilter (like SQL WHERE clause) Query.declareImports, declareVariables and declareParameters Query.declareImports, declareVariables and declareParameters Query.setOrdering (like SQL ORDER BY) Query.setOrdering (like SQL ORDER BY)

JDOQL Example Sample JDOQL with parameter: Sample JDOQL with parameter: Extent ext = pm.getExtent(BreakfastItem.class, false); Query q = pm.newQuery(ext, carbs <= maxc); q.declareParameters(Integer maxCarbs); Collection items = (Collection) q.execute( new Integer(15));

JDO Sample

JDO Summary O/R Mapping (and lots of functionality) is implementation-specific. O/R Mapping (and lots of functionality) is implementation-specific. Ordinary objects, with restrictions. Ordinary objects, with restrictions. JDOQL for query language. JDOQL for query language. Complex because of level of abstraction. Complex because of level of abstraction.

Object Databases

An alternative to relational databases… An alternative to relational databases… Object databases! Object databases! Not new; they have been around for ages Not new; they have been around for ages Not as much standardization as relational Not as much standardization as relational Sometimes lack scalability and data integrity of relational databases. Sometimes lack scalability and data integrity of relational databases. Interfaces are all over the board: Interfaces are all over the board: JDO is frequently used. JDO is frequently used. Proprietary interfaces are also common. Proprietary interfaces are also common.

Pick One For the purposes of this presentation, we choose one object database. For the purposes of this presentation, we choose one object database. Since youve already seen JDO, I choose one with a proprietary interface: Since youve already seen JDO, I choose one with a proprietary interface: db4o (= database for objects) db4o (= database for objects) Available for Java and.NET Available for Java and.NET

db4o Vital Signs Persistence for ordinary objects Persistence for ordinary objects The only solution that doesnt restrict your object model to fit database needs The only solution that doesnt restrict your object model to fit database needs Uses pure reflection for persistence Uses pure reflection for persistence Opaque non-relational database; no O/R mapping or schema management Opaque non-relational database; no O/R mapping or schema management Query-by-example or S.O.D.A. querying Query-by-example or S.O.D.A. querying

Weaknesses of db4o Doesnt scale well in my testing to arbitrary queries on very large data, as in data mining. Doesnt scale well in my testing to arbitrary queries on very large data, as in data mining. But its very fast for simple persistence needs But its very fast for simple persistence needs Perhaps 1000 times the speed of some databases Perhaps 1000 times the speed of some databases Poor referential integrity checking Poor referential integrity checking Deleting an elsewhere-referenced object doesnt give an error message, but causes database contents to become potentially invalid. Deleting an elsewhere-referenced object doesnt give an error message, but causes database contents to become potentially invalid. No provision for defining referential integrity constraints on the data. No provision for defining referential integrity constraints on the data. (Validation for non-referential constraints can happen in Java mutator methods.) (Validation for non-referential constraints can happen in Java mutator methods.)

Steps in Using db4o Create an ObjectContainer. Create an ObjectContainer. Use the ObjectContainer for data manipulation. Use the ObjectContainer for data manipulation. Commit between transactional boundaries. Commit between transactional boundaries. Close the ObjectContainer Close the ObjectContainer Careful – theres an automatic commit here! Careful – theres an automatic commit here!

Basic ObjectContainer Methods get(Object) Queries by example query() Creates a S.O.D.A. query set(Object) Stores and object in the database delete(Object) Deletes a database object activate(Object, int) Fills in fields of this object commit() Commits the current transaction rollback() Rolls back the current transaction ext() Accesses advanced functions close() Closes the database

Query by Example Perhaps the most unique aspect of db4o Perhaps the most unique aspect of db4o Fill in desired fields in an example object Fill in desired fields in an example object Including relationships and their properties Including relationships and their properties Null references or default primitives ignored Null references or default primitives ignored Call ObjectContainer.get(Object) Call ObjectContainer.get(Object) Nice but… Nice but… Only equality comparisons are possible Only equality comparisons are possible Cant look for null or default values Cant look for null or default values

S.O.D.A. Queries The alternative to query-by-example. The alternative to query-by-example. Simple Object Database Access Simple Object Database Access Queries are built using an object-based query graph. Queries are built using an object-based query graph. Nice for auto-generating queries Nice for auto-generating queries Hard to include queries in properties files! Hard to include queries in properties files!

S.O.D.A. Example Query q = db.query(); q.constrain(Employee.class); q.constrain(new Evaluation() { public void evaluate(Candidate c) { public void evaluate(Candidate c) { c.include(((Employee) c.getObject()) c.include(((Employee) c.getObject()).isRetired()).isRetired()) }});q.descend(department).descend(payPeriodDays).descend(payPeriodDays).constrain(new Integer(14));.constrain(new Integer(14)); ObjectSet results = q.execute();

Db4o Sample

Object Databases Summary Many object databases use JDO… Many object databases use JDO… But some are proprietary But some are proprietary db4o is an example of a proprietary object database system. db4o is an example of a proprietary object database system. Very close to the Java object model. Very close to the Java object model. Not suitable for large-scale systems or high reliability environments. Not suitable for large-scale systems or high reliability environments. Query by example or S.O.D.A. Query by example or S.O.D.A.

Concluding Thoughts

Putting It All Together Relational Object JDBC db4o JDO Hibernate EJB-CMP

Resources More about designing an object-relational database, but theres good info here. More about designing an object-relational database, but theres good info here

Resources Java Persistence for Relational Databases by Richard Sperko Java Persistence for Relational Databases by Richard Sperko Java Database Best Practices, George Reese Java Database Best Practices, George Reese White Papers by Scott Ambler White Papers by Scott Ambler

Other Options We cant cover everything in one night. We cant cover everything in one night. Oracle TopLink Oracle TopLink Castor JDO (confusingly not JDO-based) Castor JDO (confusingly not JDO-based) CocoBase CocoBase Apache OJB Apache OJB And more… And more…

Questions?