Archive

Archive for the ‘Persistence’ Category

Persistence testing in jBPM

27 October 2012 Leave a comment

In the last year or so, I’ve started 3 different persistence-related testing initiatives for jBPM. This is a quick summary of what’s already been created and what the plans are going forward.

Maven property injection cross-database testing

 
I’ve described the basic mechanisms and infrastructure of this testing infrastructure here:
jBPM 5 Database testing wiki page

The main jBPM project pom contains maven properties that are injected into resource files that are placed in (test) resource directories for which maven (property) filtering has been turned on.

In turn, when maven processes the test resources, the properties in the filtered files are replaced with the values placed in the pom. The resource files filtered are mostly a persistence.xml file and a datasource.properties file used to create data sources.

Unfortunately, there a still a couple problems:

  • To start with, this is only really completely implemented in the drools-persistence-jpa and jbpm-persistence-jpa modules. It needs to be fully implemented or “turned on” in a number of other important modules, such as jbpm-bam, jbpm-bpmn, jbpm-human-task-core and jbpm-test.
  • Some developers have encountered problems in their environments due to the fact that the persistence.xml file (in src/test/filtered-resources) contains properties (like ${maven.jdbc.url}) instead of real values. I’m not sure what’s going on there, but I’d like to fix that problem if I can.
  • Lastly, this infrastructure doesn’t help us test with different ORM frameworks. The problem here is that it’s practically impossible to test using a specific ORM framework (for example Hibernate 3.3) while the other ORM frameworks (Hibernate 4.1, OpenJPA) are in the classpath. But if you want to test with a specific ORM framework, the first thing you need to do is to have it in the classpath. So how do I test against multiple (“specific”) ORM frameworks?
    • While maven profiles are an answer to this, maven profiles add lots of complexity to a setup. I don’t want to make a maven profile for every different ORM framework that we need to test against, instead, I’d like to just make a maven profile that turns on the cross-database and cross-ORM framework testing
  • In the coming months, it looks like we’ll try to make sure that this framework is turned on and executed in all of the modules were it’s applicable. While I expect that I’ll eventually remove the maven filtering being used here, I think I’ll probably try to keep the property based control: being able to run the test suite on a different database simply by injecting (settings.xml) or otherwise changing (in the pom.xml) properties is valuable.

    Backwards compatible BLOB testing

     
    One new that I encountered when learning the jBPM 5 (and Drools) code is the use of BLOB’s in the database to store session and process instance state (and work item info). One of the great advantages of storing process instance state in a BLOB is that it avoids the complicated nest of tables that a BPM engine would otherwise bring with it — see the jBPM 3 database schema for a good example. :/

    Using a BLOB also has the advantage that changes can be made to the underlying data model without having an impact on the database schema that the (end) user will use while working with jBPM 5. This, more than any other reason, is really _the_ reason to use BLOBs. I still like to think about how much easier that makes the work of developers working on Drools and jBPM. (Considering the complexity of the Drools RETE network, it makes even more sense to use a BLOB.)

    However, when the serialization (or “marshalling”) code was originally developed, the choice was made for a hand-crafted serialization algorithm, instead of relying on pure Java serialization. This is also with good reason given that Java serialization is slow and the hand-crafted serialization algorithm was a lot faster.

    At the beginning of this year, for reasons having to do with the evolution of both Drools and jBPM 5, Edson (Tirelli) replaced all of the serialization code for Drools and jBPM 5 with protobuf. That was definitely an improvement, mostly because protobuf makes forward and backwards compatibility way easier.

    However, it made some of the “marshalling testing framework” work I had done up until that moment not usable anymore: the marshalling testing framework relies on generated databases from previous versions to be able to test for backwards and forwards compatibility. Switching to protobuf unfortunately broke all backwards compatibility at the cost of making sure that backwards compatibility from then on would be ensured.

    At the moment, what needs to be done is the following:

    • The databases for the different branches (5.2, 5.3, 5.4 and now 6.0) of jBPM need to be regenerated.
    • The code for the marshalling framework (as well as other persistence testing classes) needs to be cleaned up a little bit.

    Multiple ORM framework (ShrinkWrap based) testing

     
    I ran into a specific Hibernate 4 problem last month and it was unfortunately something that could also have affected compatibility with Hibernate 3. This meant that I needed to run the test suite multiple times with Hibernate 3 and then Hibernate 4 to check certain issues.

    As a result, I ended up building the framework in the org.jbpm.dependencies [github link] package. This framework creates a jar containing the tests to be run with the specified database settings (persistence.xml) as well as the specific ORM framework that the tests should be run with. It then runs the tests in the jar.

    This is eventually the framework that I want to have used within all persistence related jbpm modules. The code itself should probably be moved to the jbpm-persistence-jpa module.

Categories: jBPM, Persistence

The JTA manager lookup in Hibernate 4/AS7

21 September 2012 Leave a comment

I’m waiting for eclipse to start up again, so I’ll post a quick note about the JTA manager lookup mechanism that’s changed in Hibernate 4 and AS7.

Those of you who’ve read the How do I migrate my application from AS5 or AS6 to AS7 guide on the Jboss AS 7 documentation site have probably come across the following quote:

Configure the datasource for Hibernate or JPA

If your application uses JPA and currently bundles the Hibernate JARs, you may want to use the Hibernate that is included with AS7. You should remove the Hibernate jars from your application. You should also remove the “hibernate.transaction.manager_lookup_class” property from your persistence.xml as this is not needed.

Unfortunately, while you may not need the hibernate.transaction.manager_lookup_class property, you do need a property apparently.

If you don’t set any property (with regards to a JTA transaction manager), you’ll end up getting a stack trace that looks like this:

java.lang.NullPointerException
at org.hibernate.engine.transaction.internal.jta.JtaStatusHelper.getStatus(JtaStatusHelper.java:73)
at org.hibernate.engine.transaction.internal.jta.JtaStatusHelper.isActive(JtaStatusHelper.java:115)
at org.hibernate.engine.transaction.internal.jta.CMTTransaction.join(CMTTransaction.java:149)
at org.hibernate.ejb.AbstractEntityManagerImpl.joinTransaction(AbstractEntityManagerImpl.java:1208)

If you even go as far as googling the above stack trace, you’ll run into HHH-7109, which then mentions the hibernate.transaction.jta.platform. Oh, except that it’s a new, undocumented property”. Great..

Ah ha! But there is actually documentation for this! It can be found in the developer guide, specifically, here.

There is in fact, a whole chapter on transactions, for developers, here.

Categories: Persistence

The DB2 “result set is closed” error

1 July 2012 5 comments

Your (IBM) DB2 JDBC driver might throw the following error in some situations:

Caused by: com.ibm.db2.jcc.am.SqlException: [jcc][t4][10120][10898][4.12.55] 
Invalid operation: result set is closed. ERRORCODE=-4470, SQLSTATE=null

If you look around the internets, you’ll find pages like this, which state things like this:

The IBM Data Server Driver for JDBC and SQLJ automatically closes the cursor 
when all rows have been retrieved from a ResultSet. When ResultSet.next is 
executed after the cursor is closed the SQLException is thrown.

It might even suggest an eccentric solution, like:

Alternatively, you can may enable the allowNextOnExhaustedResultSet 
datasource property. When the allowNextOnExhaustedResultSet property is set 
to DB2BaseDataSource.YES (1), and a forward-only cursor is positioned after 
the last row of a result set, a call to ResultSet.next returns false, instead 
of throwing an SQLException.

Don’t listen. Okay, maybe, in your case, the above applies. In that case, congratulations on fixing your bug and please get the heck out of here. If you’re still stumped, listen up to my extrapolations:

You’re running a query somewhere, and you need to put a transaction around it. That’s what this unreadable, confusing error means.

Let’s start with cursors: cursors are also subject to transaction boundaries. In fact, the way I see it, cursors were made for programmatic SQL languages, such as Transact-SQL or PL/SQL — and, okay, for SQL. More info on wiki.

But cursors are, in essence, database-level constructs that are exposed to users so that they can take advantage of them. What I mean to say by that is that cursors are sometimes used when you don’t expect them to be — like in a normal query.

And cursors need transactions! Because it’s a non-atomic set of operations on a set of results — but we’re expecting it to act atomically: get records from database. Bam.

So go find the query that’s causing the error, and put a transaction around it. It’ll help — or at least, it helped me.

Categories: Persistence

JPA 2 with Drools and jBPM

3 April 2012 3 comments

At the moment, I’m finishing off making Drools/jBPM ready to be able use JPA 2 and Hibernate 4.x. This post is in some sense a draft for the official documentation, but again, see about for why I write blog posts.

 


Using jBPM with JPA 2

 

If you’re reading this, it’ll be because you want to use Drools or jBPM with JPA 2 and maybe even with a different ORM framework. Awesome! That means you’re using jBPM and that you’re using jBPM the way you want to and are not being forced by the software to make choices you don’t want to. I heartily approve of both.

First off, if you’re using JPA 2 with drools and JBPM, you’ll need to change the all the instances of "1.0" and ..._1_0.xsd to "2.0" and ..._2_0.xsd. Don’t forget those schemaLocation values that are off beyond the right edge of your screen.

Second off, you’ll have to modify your persistence.xml and change the mapping for the ProcessInstanceInfo class. In your persistence.xml, you’ll have to remove the following line:

  <mapping-file>META-INF/ProcessInstanceInfo.hbm.xml</mapping-file>

Once that’s done, you’ll have to add a JPA2 mapping for that class to your persistence unit. If you want to, you can just change the following line in your persistence.xml:

  <mapping-file>META-INF/JBPMorm.xml</mapping-file>

to

  <mapping-file>META-INF/JBPMorm-JPA2.xml</mapping-file>

The JBPMorm-JPA2.xml file is included with the jbpm-persistence-jpa jar: if you’re curious about the entity mapping, you can find it there.

Lastly, depending on your setup, you might have to change the TransactionManagerLookup class that you’re using. This line:

<property name=”hibernate.transaction.manager_lookup_class” value=”org.hibernate.transaction.BTMTransactionManagerLookup” />

needs to be changed to:

<property name=”hibernate.transaction.jta.platform” value=”org.hibernate.service.jta.platform.internal.BitronixJtaPlatform” />

And that’s it!


Developer bits and ranting

 

The following is mostly explaining how I got to the solution that I did get to. :)

In any case, regarding enabling and testing the JPA 2 functionality in Drools/jBPM and to start with: there are XML metadata overrides. I’ve known about XML overrides for persistence annotation data for a while, but while I’ve almost used them to solve a some problems, it never really happened. Part of this has to do with the fact that the configuration for jBPM is still evolving.

In the past year, the persistence.xml in jBPM has been extracted from the jars so that users can specify their own persistence units. This  means it’s also how users will be able to specify JPA 1.0 or 2.0 in their persistence units.

Another hurdle here has been the fact that Drools/jBPM is stuck with Hibernate 3.3.x and 3.4.x jars at the moment — and those versions do not support JPA 2. This means futzing around with dependencies and maven profiles. It’s not that big of a deal but also not that elegant — but I challenge anyone who says programming that gets things done should always be elegant. It’s a goal, not a requirement.

Also, the largest Hibernate 3.3/3.4 compatibility issue is by far the @CollectionOfElements annotation: ahead of it’s time, but it’s unfortunate that Hibernate didn’t elect to stay backwards compatible by translating @CollectionOfElements to @ElementCollection underwater once Hibernate became JPA 2 compliant.

So, what I’ve done is the following:

  • I’ve created a hibernate-4 profile in the relevant projects that use persistence (drools-persistence-jpa, jbpm-persistence-jpa, and jbpm-bam, to start with).
  • This hibernate-4 profile replaces the hibernate 3.x dependencies with hibernate 4.x dependencies, and since some hibernate jars changed names between versions, it adds the correct 4.x jars to replace the 3.x jars.
    • This means that, for example, the hibernate-annotations 3.x jar will be there when it’s not needed, but that’s not that big a deal. (It looks like hibernate-annotations disappeared as soon hibernate started supporting JPA 2.0).
    • Part of what I’m doing is also making sure that no hibernate jars are actually needed in the code — we want jBPM (and Drools) to be ORM framework independent and hard-baking hibernate dependencies in won’t help that.
  • This hibernate-4 profile also uses the maven-replacer-plugin to fix all kinds of things in the persistence.xml and orm.xml files.

Lastly, before I get to the meat of this post (using JPA 2 with Drools/jBPM), I have one more fact (or rant..). Why is the order of attribute types defined in entity-mappings? WHY?!? To give you an illustration, the following mapping is incorrect:

<entity class="forest.bee.hive">
  <element-collection name="honey" target-class="double" />
  <basic name="worker" />
  <version name="queen">
  <basic name="pupa" />
</entity class>

It’s invalid because all <basic> elements must come before all <version> elements which must come before all <element-collection> elements. WTF? I once had to mediate a ridiculously overheated discussion about XSD standards — and I do realize that it’s a complicated if arcane topic — but stil, wasn’t it possible to decouple the order? Okay, maybe it was impractical in terms of the parsing modifications then necessary.

In any case, if you’re writing your own JPA 2 XML entity-mappings, stop, and do it using annotations. Otherwise, if you really must know, the order (as specified in the xsd) is:

  1. description
  2. idorembedded-id
  3. basic
  4. version
  5. many-to-one
  6. one-to-many
  7. one-to-one
  8. many-to-many
  9. element-collection
  10. embedded
  11. transient

What was also interesting were the precise semantics of that metadata-complete attribute present in the <entity> element. If you do a websearch on “metatdata-complete jpa”, you’ll quickly learn that metadata-complete=true means that the annotation information for the class is ignored in this case and that the container/persistence-context will only use the XML information.

However, I wanted to do as little work as possible and was curious as to what would happen if metadata-complete=false. And the web was silent.. until I found this little sentence on a sun blog/technical post about JEE 6:

As is the case for web fragments, you use the <metadata-complete> element in the web.xml file to instruct the web container whether to look for annotations. If you set <metadata-complete> to false or do not specify the <metadata-complete> element in your file, then during deployment, the container must scan annotations as well as web fragments to build the effective metadata for the web application. However, if you set <metadata-complete> to true, the deployment descriptors provide all the configuration information for the web application. In this case, the web container does not search for annotations and web fragments.

This quote is from Introducing the JEE 6 Platform: Part 3 and the bold lettering has been added by me to emphasize my point: if metadata-complete=false then the container will use both the annotation data and the XML data. That means that if you want to override an annotation with XML information, you must use metadata-complete=true to ensure that the container does not use the annotation metadata. This must be partially why we can get away with using the ProcessInstanceInfo.hbm.xml mapping file and leaving a @PreUpdate annotation in the file.

Lastly, I’d also like to add that I’ve had to do some minor hacking to get all of this done. The problem, in short, is the following:

  1. The main problem is that jBPM 5 contained a class that used the Hibernate 3.x exclusive @CollectionOfElements annotation.
  2. Leaving this in the code meant that we could not compile the code with Hibernate 4 (since 4 doesn’t contain that class).
  3. Which means that using a ProcessInstanceInfo.hbm.xml file — a hibernate mapping file — to configure the ProcessInstanceInfo class. Because the mapping is now in the xml, the code will compile with Hibernate 4.
  4. However, this means we have to map all fields of ProcessInstanceInfo in the ProcessInstanceInfo.hbm.xml file: Hibernate will otherwise unfortunately complain about duplicate (hibernate) mappings if we use both (JPA or Hibernate) annotations and a Hibernate mapping file.
  5. Luckily, we can leave the @PreUpdate annotation in ProcessInstanceInfo, since Hibernate 3.x doesn’t have a simple annotation that is equivalent.
  6. But Hibernate 3.3/3.4 doesn’t support byte array blob objects. This means writing a BlobUserType class that extends the Hibernate UserType in order to be able to do this: we don’t want users to have to change their schema’s. Most annoyingly (and in slight contradiction to the quote from sun above), Hibernate will complain if we use JPA (field) annotations and a hbm.xml file on/for the same class.
  7. When users use JPA 2 (possibly with Hibernate 4), they then need to make sure not to use the ProcessInstanceInfo.hbm.xml and instead to include a JPA 2 entity mapping for the ProcessInstanceInfo class — which is currently commented out in the JBPMorm.xml file.

And that’s what has happened.

 

Categories: jBPM, Persistence

Human task and persistence: cross-cutting concerns

29 February 2012 Leave a comment

For those of you who aren’t familiar with the jBPM human task module, here’s a very brief introduction:

To start with, a “human task” describes two things:

  • In an otherwise automated business process, a “human task” is the concept that one of the steps in the process consists of and is strongly dependent on human input.
  • In a larger sense, a “human task” is a label that helps people who model business processes differentiate between system interactions and human interaction in a process.

Somewhat obviously, the “human task” aspect of BPM is problematic but important to model and formalize. Modelling human behaviour in a (computer) systems’ environment is not a simple task. There are even two specifications that describe how this should be done:

  1. To start with, we have the WS-HumanTask specification, which is the generally accepted specification for describing how a human task can be formally described.
  2. There’s also BPEL4People, which is essentially a BPEL extension for human tasks.

In any case, the jBPM human task module implements the WS-HumanTask specification. This module basically takes care of all the bookkeeping and administration specified for human tasks: who gets a task, who may get a task, who is working on a task, which tasks are closed, which content is associated with which tasks, etc.

If these were “computer tasks” instead of “human tasks”, then the BPM engine would call the computer or system responsible for the “computer task”. Then, a second or two later the computers respsonsible for the task would have completed the task and done all the bookkeeping or otherwise informed the BPM engine that an error occurred.

However, humans don’t work that fast. And they also need a human friendly method of letting the BPM engine know the status of the task. And lastly, given that computers are better at that bookkeeping thing (which is basically computation), humans also want the computers to do that. Thus, we get the WS-HumanTask specification and the jBPM implementation of that specification.


Phew, okay, now that we’ve gotten that out of the way, I can write about what I really want to.

Persistence in the jBPM human task code is a cross-cutting concern. Click on the link for wikipedia’s definition of the term, but in short, persistence has nothing to do with the human task code.

(In fact, persistence is almost always a cross-cutting concern, unless you’re working on something that only does persistence, like an ORM framework such as Hibernate.)

The easiest way to identify cross-cutting concerns is to think about how you would fix a bug in the code. For example, if you have a bug in the jbpm-human-task code because tasks aren’t being assigned to the correct users, then you look at the logic in the jbpm-human-task module dealing with task ownership. A persistence bug, however, will have nothing to do with the task ownership logic — or any of the other logic in jbpm-human-task that describes the human task bookkeeping and administration. A persistence bug will only have to do with the code in the jbpm-human-task module that well, deals with persistence. Well, at least, a persistence bug should only have to do with the persistence bug..

And that’s exactly my point: code deals with cross-cutting concerns should always be encapsulated and isolated from the rest of the code when possible. This isolation is a form of loose-coupling. Encapsulation is important so that other developers don’t mistakenly start using instances or methods that they shouldn’t — and of course, it makes finding bugs a lot easier. Loose-coupling is important because it allows developers to extend and further develop projects with relatively little problems. (Loose-coupling is one of several fundamental ideas that a developer should have hardwired into his or her brain, in fact).

Lastly, if you do not encapsulate or apply loose-coupling to your code, you are in for a world of hurt, as they say. This regardless of which cross-cutting concerns you’re actually dealing with. Lack of encapsulation or loose-coupling means project death, if left alone. I am really serious about this — and if you don’t believe me, or don’t understand how important this is, well.. then, I’m actually speechless (or I’m otherwise convinced that you haven’t studied computer science). If you then also happen to be one of my colleagues or work on my project, don’t worry, I already know who you are. <evil laugh>

Why am I writing all of this? So that I can refer to this post in the discussions that I unfortunately expect to have about this issue. And of course, as always, I’m writing this because that is what this blog is about.

Categories: Persistence

jBPM 5 database testing mechanisms

22 July 2011 Leave a comment

I recently finished making a mechanism for enabling the use of different databases with a set of unit tests.

A thorough explanation of the mechanism in its entirety can be found here:
http://community.jboss.org/wiki/JBPM5DatabaseTesting

The reason I’m also posting this here is because the mechanism that I describe can actually be built in to most maven projects that contain unit tests that use persistence.

In my case, the specifics are as follows:

  • The mechanisms are built into the jBPM framework (the code can be found here).
  • The unit tests are spefically using JTA (the Java Transaction API).
  • The Transaction layer is handled by the Bitronix transaction manager.

But none of these details mean that the same mechanism can’t be applied to any other project, including those that don’t do anything with JTA. Using JPA is to some degree intrinsic to the mechanism, although other java persistence frameworks that also use XML configuration could also be used, I think.

In short, it comes down to maven filtering + the Properties mechanism in Java + using XML configuration for JPA (persistence.xml, in this case).

Categories: CI/Buiilds, jBPM, Persistence Tags: