Archive for the ‘java’ Category

Batching JMS messages for performance; not so fast

Recently an idea had crossed my radar around speeding up performance of messaging producers; batching messages into a single transaction. The idea being that there is overhead in the standard behaviour of a message bus that can be optimised out if you group messages into a single transaction.

My initial thoughts about this were that it was unlikely; at best the performance would be the same as an untransacted client. The comments were framed in conversations of Spring’s JMSTemplate, which continually walks the object tree ConnectionFactory -> Connection -> Session -> MessageProducer, and back closing them off. Each close causes some network traffic between the library and the bus. My thoughts were that perhaps the discussion hadn’t accounted for caching these objects via a CachingConnectionFactory. I resolved to find out.

My other concern was that while it could be done in the general case, it didn’t make a lot of sense.

Generally speaking there are two categories of messages that will concern an application developer. Sending throwaway messages that aren’t critical is one thing – you can not persist them on the broker side and save 90% of the overhead. You can also do asynchronous sends, which won’t persist the messages and save further time by not giving your app confirmation that the broker received them. If you lose them, it won’t be a big deal.

The other type are the ones that you really care about. Things that reflect or affect a change in the real world – orders, instructions, etc. The value of a broker in this case comes from knowing that once it has the message, it will be dealt with.

In ActiveMQ, this guarantee comes via the message store. When you send a message the broker will not acknowledge receipt until the message has been persisted. If the broker goes down, it can be restarted, and any persisted messages may be consumed as though nothing happened, or another broker takes over transparently using the same message store (HA). As you can imagine, there is natural overhead. Two network traversals, and some I/O to disk. Keep this in mind.

I tested a couple of different scenarios.

  • JMSTemplate with a CachingConnectionFactory. This is the baseline.
  • An untransacted, persistent session. Each message gets sent off to the broker as they come in. The broker places the message in a persistent store. This is like the logic of the JMSTemplate, but where you control the lifecycles of JMS objects.
  • An untransacted, unpersisted session. Each message gets sent off to the broker as they come in. The broker stores the message in memory only for delivery.
  • Transacted sessions. Messages get bundled together into a transaction and committed.

Messages were posted to a queue with no consumers on a single broker with default setttings.

The results surprised me.

Average message send time

Average message throughput

Timings should be considered indicative only, and are only there for a relative comparison. Initial spikes should be considered part of a warm-up, and ignored.

Firstly, getting down to the JMS API was consistently faster than with JMSTemplate (not surprising in hindsight given its object walk). There was also indeed an upshot in performance from batching messages together for delivery in a transaction. The more messages there were in a batch, the better the performance (I repeated this with 10, 100 and 1000 messages per transaction). The results approached those of the non-persistent delivery mode, but never quite reached it. What accounts for this?

I did some digging, and came up with this description of transactions in ActiveMQ. It appears that there is a separate store in memory for uncommitted transactions. While a transaction is open messages are placed here, and are then copied over to the main store on commit.

So what to make of this? Should you forgo JMSTemplate and batch everything? Well, as with pretty much anything, it depends.

JMSTemplate is simple to configure and use – it also ties in nicely to the various Spring abstractions (TransactionManagers come to mind). There are certainly some upsides to getting down to the JMS API, however they come at a cost of time coding and losing those hooks; which you might regret later.

Batching itself is a funny one, in that it’s one of those things that could be easily misused. Transactions are not there for improving performance. They exist as a logical construct for making atomic changes. If you batch up messages in memory to send to the broker for a performance upshot that are unrelated; and your container goes down before you’ve had a chance to commit; you have violated the contract of “this business action has been completed”, and you lost your messages. Not something you want for important, change-the-world messages.

If you have messages that are relatively unimportant that won’t affect the correct operation of your system, there are better options available to you for improving performance such as dropping persistence.

If, however, you have related messages tied to a single operation, then perhaps getting down to the JMS API and batching these together under a single transaction is something you might want to consider. The key here is ensuring that you don’t lose the real meaning of transaction.

Update 1 (13/11): Replaced graphs with log scale to make the series clearer.

Update 2 (13/11):Thanks to James Strachan for an extended explanation:

The reason transactions are so much faster on the client site can be explained comparing it to the normal case, where the client blocks until the message has definitely gone to disk on the broker.

In a transaction the JMS client doesn’t need to wait until the broker confirms the message because it’s going straight into the transactional store (memory). Because the broker will complain on a commit if something has gone awry, the client can afford to not block at all doing a send or acknowledge (not even blocking for it to be sent over the socket), so it behaves like a true async operation. This means it’s very fast.

This explains why transactional performance approaches that of asych operations with the size of the batch – because for all intents and purposes it is nearly asynchronous. The only cost is that of the synchronous commit(), which is a flush from the transactional store to the journal.

You could argue that this doesn’t really affect the actual throughput – as the difference between batching & no transactions is just the client sitting around blocking. The benefit of the transactions is that it’s making each client much more effective. I.e. to get equivalent throughput you would need more clients running concurrently. In the case of the Camel, this could be achieved by configuring in a larger thread pool.

A brief outline of when to use transactions.

Get Functional

That was the message that was coming through the Devoxx conference presentations this year. The idea that it will help your code run in the brave new world of multi everything (multi-core, multi-thread etc.) is one that’s widely touted, but rarely the primary driver for its use. Instead, it’s about less code, that’s more easily understood. When you do get to scaling it, it won’t do any harm either.

As Guillaume Laforge tweeted, from 800 Java developers in his session, only 10 knew/used Scala, 3 Clojure, 20 Ruby, and 50 were on Groovy – which gives a nice gentle introduction to some of the constructs for those looking to wade in. Good stats to cut through they hype. So what of the roughly 90% slogging on without closures, does this mean that they have to miss out on this fun?

Quite simply, no. There’s heap of drop in libraries that you can add into a Java project for all manner of functional goodness, and which don’t change the syntax of the language. LambdaJ for example gives a nice functional way of dealing with collections. To steal an example directly from the website, the following typical Java code:

List<Person> sortedByAgePersons = new ArrayList<Person>(persons);
Collections.sort(sortedByAgePersons, new Comparator<Person>() {
        public int compare(Person p1, Person p2) {
           return Integer.valueOf(p1.getAge()).compareTo(p2.getAge());

is replaced with:

List<Person> sortedByAgePersons = sort(persons, on(Person.class).getAge());

Fancy a bit of map-reduce without a grid? Well, it comes stock-standard with the Fork Join (JSR166y) framework that will be added to the concurrency utilities in JDK 7. If you don’t fancy waiting until September 2010 (the latest expected date for the GA release), it’s downloadable here. As an aside, Doug Lea has written a really good paper on the FJ framework.

Don’t fancy loops in loops in loops to filter, aggregate, do set operations with all the null checking that Java programming typically entails? Well, the Google Collections library (soon to be integrated into Guava, a set of Google’s core libs), contains predicates and transform functions that make all of this a lot easier to write and reason about. Dick Wall had a great presentation about this showing just how much code can be reduced (heaps).

A thing I heard a number of times outside the sessions was, “I don’t know about all this stuff, surely as we get further from the metal, performance suffers”. Sure, it gets harder to reason about timings as the abstractions get weirder, but the environment gets better all the time, and the productivity gains more than outweigh performance in all but the most perf-intensive environments. Brian Goetz spoke about how the JVM supports this new multi-language world. Not something that I had ever really given much thought to, but the primary optimizations aren’t at the language compiler level (javac, scalac, groovyc etc.)- they’re are all done at runtime, when the JVM compiles the bytecode. The number of optimizations in HotSpot are massive (there was a striking slide showing 4 columns of individual techniques in a tiny font). Multiple man-centuries of effort have gone into it, and each new release tightens it up. If you’re not sure, then profile it and make up your own mind. JDK 7 will also see the VM with some goodness that will make dynamic languages really fly.

One thing that still sticks out like a sore thumb is Closures support in Java. It’s not a candidate for inclusion in JDK 7, and the proposed syntax shown at the conf by Mark Reinhold looks pretty ugly when compared to other langs (see the proposal by Neal Garter). Either way, not a sniff of actual implementation. I understand there’s some serious work on the VM to make any of this possible regardless of the syntax. Not holding my breath. [Closures will actually be in JDK7 – thanks Neal.]

All up, I’m pretty excited by all this, and can’t wait to get my hot little hands on some of these tools. The functional style yields code that’s much easier to read and reason about, and the fact that it’s essentially all Java syntax, means that there’s no reason not to apply it. If you’re already comfortable with using EasyMock on your team, you won’t find it a huge mind shift.

Deep Diff Pizza

There is nothing I love more than a proprietary, undocumented API. Call it an unfortunate fact of life, but weird object models that hang together by the skin of their teeth are out there. Most of the time there’s no validation logic to check that they’re semantically or syntatically correct until you send this tangle of objects to a system you’re integrating with. Having been burned badly in the past on this sort of work, I’ve been looking for a way to work effectively in these types of scenarios.

In my most recent case, I had some example code that built up these object graphs for various use cases. The code wasn’t transportable, but it did generate correct inputs. After reverse engineering the graph construction logic into a builder by working out of XStream dumps to console, I was then able to build up exploratory integration tests that triggered the various use case scenarios. So far, so good. Now to just remove the need to have the back-end system up.

What I needed was a way to compare a test object graph with a constructed one. I already had an XML representation thanks to XStream of the expected output, so I could reload it into memory as needed – so there’s the test data. The equals() and hashCode() methods on the model were unreliable, so that’s no good. I toyed with the idea of writing a general purpose deep diff mechanism for object graphs, but came to the conclusion that it wasn’t a good idea. Aside from not wanting to get into writing gnarly reflection code I realised that the problem was more difficult than it seemed. You get into questions like “What is equality?”, “How much change is acceptable?”, “Where do I stop? Primitives, java.util.*?”, and “How can I mark things as being acceptably different?”. It’s probably a good indication that noone had written this sort of thing already.

Then I realised that the answer was staring me in the face. Just diff the XML representation programatically! By loading up an expected input from a file next to the unit test, I could then dump the model under test out to another String and… Rock and Roll.

XMLUnit has a quite good diffing utility, which was about 80% of the way there. Some of the data in my model changed over time, so I needed a way to say “ignore these bits of the tree”. There’s a neat little interface in XMLUnit called DifferenceListener, that gets notified whenever the mechanism finds a diff. You can implement a method that decides whether to report the difference as different, the same, or similar (why you’d want to do that is beyond me – “this is different-ish”). I hacked up my own implementation that took some XPath expressions, mapping the nodes in the control graph that ought to be ignored and voila.

The more I code the more uses I find for XStream. In combination with XMLUnit, it’s like hazelnuts to chocolate. It’s an excellent option next time you need to diff object graphs.

A Better Builder

The builder pattern should be familiar to anyone who has needed to change data from one format to another. Often called assemblers, translators or similar, they’re found peppered throughout almost every project. The idea is pretty simple:

public class HouseBuilder {
    public House buildHouse(OtherHouseType otherHouse) {
        House house = new House();
        // copy a whole bunch of properties from the otherHouse....
        for (OtherWallType otherWall : otherHouse.getWalls()) {
        return house;

    private Wall buildWall(OtherWallType otherWall) {
        Wall wall = new Wall();
        // copy another whole bunch of other properties....
        // do something fancy every third Tuesday of the month...
        for (OtherWindowType otherWindow : otherWall.getWindows()) {
        return wall;

    private Window buildWindow(OtherWindowType otherWindow) {
        Window window = new Window();
        // you get the idea....
        return window;

It’s OK for translating small object graphs, but for non-trivial work, these things can turn into huge heaving masses of code. Because they’re pretty quick to knock out, they’re generally untested as the above code doesn’t really lend itself to it. Take the buildWall() method above – you can’t really exercise the “every third Tuesday” case easily without running through buildHouse(), and then calling buildWindow(). It becomes a pain to construct the data model to pump into test cases, and as we know, anything that’s not easy or highly visible doesn’t get tested. Did I just say that? Oops. Sorry, industry secret.

One strategy that’s applied is to write builders for each object type, i.e. HouseBuilder, WallBuilder, WindowBuilder. These are then either wired together via dependency injection (ugly as it exposes internal mechanics), or hard-coded in with a setter to substitute a dependency at test time. There’s also the drawback of a morass of tiny classes proliferating in a package somewhere. Not necessarily a bad thing, but when doing this sort of work there’s typically a good amount of refactoring, as well as use of utility classes to enrich data when moving between formats. Personally, not a big fan of this approach.

So, here’s a cool little trick:

public class MuchImprovedHouseBuilder {
    private MuchImprovedHouseBuilder delegate;

    public MuchImprovedHouseBuilder() {
        delegate = this;

    void substituteInternalBuilder(MuchImprovedHouseBuilder otherBuilder) {
        delegate = otherBuilder;

    public House buildHouse(OtherHouseType otherHouse) {
        House house = new House();
        // copy a whole bunch of properties from the otherHouse....
        for (OtherWallType otherWall : otherHouse.getWalls()) {
        return house;

    Wall buildWall(OtherWallType otherWall) {
        Wall wall = new Wall();
        // copy another whole bunch of other properties....
        // do something fancy every third Tuesday of the month...
        for (OtherWindowType otherWindow : otherWall.getWindows()) {
        return wall;

    Window buildWindow(OtherWindowType otherWindow) {
        Window window = new Window();
        // you get the idea....
        return window;

All of the translation is done in one builder class. On construction, a private field defines a “delegate”. Think of it as a placeholder that will do the “other bits” of the translation and set it to “this”. A package scoped setter allows a test to overwrite the builder with a mock. So with a bit of reference fiddling, we can then test one method at a time, without having to worry about the implementation details of the other methods in the class. All of the non-public methods are given package scope, so they can be tested straight out of your favourite test framework. So, you can now do this in live code:

HouseBuilder builder = new HouseBuilder();
House house =;

And at the same time write tests like this:

HouseBuilder builder = new HouseBuilder();
HouseBuilder mockBuilder = EasyMock.createMock(HouseBuilder.class); // the classextension version of EasyMock
EasyMock.expect(mockBuilder.buildWindow(isA(OtherWindowType.class)).andReturn(new Window());
Wall wall = builder.buildWall(otherWallType);

Simple, and you don’t have to stick in any new tools to achieve it.

Bob's girlfriend misunderstood when he said his diet needed more iron.

Would you like a toolbar with that platform upgrade?

It’s innocuous enough. “A new update of Java is available. Do you want to upgrade now?”

Sure, OK. Why not. Click. Dowload. Install.

Java installs Yahoo toolbar

WTF? What’s the story here? Are Sun that strapped for cash that they need to make a few bucks by installing Yahoo’s “value adds”?

When a Cache is more than a Cache

Recently I have come across a number of instances when I have needed to perform searches across data cached in memory. Standard cache implementations, however do not provide anything more than basic key-value lookups, which is a bit of a pain. One of my colleagues came up with a clever solution to this – keeping a seperate data structure in memory that acts as an index specific to the search at hand. Ehcache provides callback hooks that can be triggered when an object is about to be evicted. This way you can delete the object from the index when the eviction policy says that it’s time to go.

It’s a neat idea to get around the immediate requirement, but I am not convinced that it is the way to be doing things. The need to create supporting data structures and manage them on a per-search basis smells in a big way. Ideally what you want is a proper, searchable, in-memory database, like JavaDB/Derby/Cloudscape but without the hassle of needing to do O-R mapping on arbitrary pojos (updated: Berkely DB comes pretty close). So this has got me thinking that perhaps OO databases are the way forward here.

The last I heard of people I knew doing anything with them was nearly a decade ago, and the problems around their management were highlighted in my databases subject at university. Basically “don’t go there” was the general advice. I don’t know that the arguments around them apply any more. Integration databases (for people poking around in your data *ugh*)  aren’t as common these days thanks to a better awareness of integration options (Martin Fowler wrote about this recently). I suspect management tools have gotten better over a decade. If not, it won’t be long until scaffolding via Grails takes care of the mundane stuff for DB4O at least. Steve Zara writes about this on his blog. Steve also discusses using DB4O as a cache. It’s definitely worth taking a look at, especially as to how it might hook in with eviction policies. Perhaps used hand in hand with the callback hook technique above? DB4O might just be the peanut butter to Ehcache’s chocolate.

Hibernate/JPA Ternary Relationships

After much pain and suffering trying to get ternary relationships working correctly using the JPA annotations, I finally hit upon this post. The secret sauce: it makes use of the (rather poorly documented) @CollectionOfElements Hibernate annotation to annotate the set of link objects in the primary class, and makes the link class @Embeddable. No primary key class in the link.

JPA 2.0 supposedly manages to make this cryptic nonsense, including the ability to associate additional information with a link table, much easier.

Pimping Builds

From the Pimp My Build session by the Atlassian guys.

  • Use Ant imports. The imported stuff can check for preconditions and fail cleanly using the <fail unless=”…”> tag.
  • Use macros.
  • Don’t build stuff you don’t need using the <uptodate> task. Use <outofdate> from ant-contrib, which is even better.
  • You can use audio snippets to tell you when you screw it all up :)
  • You can filter messages in builds using the Unix shell to notify you of actually important stuff rather than the standard boiler plate.
  • Don’t be afraid to write tasks – everyone should know how the build works. Don’t be precious about it. If you have repetitive tasks, why not script it?
  • Use scripts. You can embed Javascript directly into your Ant build via a <[CDATA[..]]> block
  • Use conditional tasks (ant-contrib) <if> <then> <else>
  • Don’t do one-off analysis. PMD, Checkstyle and Findbugs can be scripted! I found this to be particularly useful. Much easier to find issues, especially if coupled with continuous integration.
  • Document your build! Ant targets have descriptions. You do it with your code, why not your build artefacts? Use the -target_name convention for private targets.
  • Use continuous integration. This has been an absolute life changing thing for me as a developer.
  • Test in your builds!!! JUnit, TestNG et al.
  • Maven tips:
    • Use a remote repository proxy – caches are good (Apache Archiva). Helps performance and stability – make sure you can run when the net goes down.
    • Create a local repository for private artifacts
    • Local repository for public artifacts – third party Jars or commercial stuff not available in public repositories

Oh yeah, Ivy is good when you aren’t using Maven.

JavaOne Flights Booked

Everything is finally booked and I am looking forward to hitting the shores of San Francisco next weekend. The lineup looks really good and I’m still having difficulty choosing between the sessions. I will also be at CommunityOne, which looks outstanding for a free event. 14 tracks!? Amazing! Hats must go off to the organizers. The only session that I have firmly fixed is on Java User Groups, but I have no doubt that the rest of the schedule will work itself out with ease.

I hope to be blogging live (ie. unedited notes and opinion) while there, but that all depends on how the laptop batteries manage to hold out. Fingers crossed!

All that and I get to sample Delta Airlines’ world famous hospitality too 😉

JavaOne Pre-booking

I finally got around to booking in onto some of the tech sessions for JavaOne in San Francisco next month. Gasp! The amount of stuff going on is incredible. From new languages on the JVM (Fortress, Scala, JRuby) to SOA, mobility and techniques in app development it’s pretty easy to book up 12 hours a day. My approach, lock in a full programme of stuff that looks good, and turn up if the brain is still functioning. It has taken the better part of an hour to read through the sessions for the Tuesday, so it’s no easy task. I can’t wait. All I have to do is get around to sorting out the minor detail of a flight from London 😉