Archive for the ‘osgi’ Category


Deep testing of integrations with Camel

One of the things that often comes up in client conversations about developing integration code with Camel is what test support is available, and more to the point appropriate, for testing integrations. There is a spectrum of test types that can be performed, ranging from fully automated unit tests to full-blown multi-system, user-based “click and see the flow through effects” tests. Camel came with comprehensive test support baked in from it’s very inception, but the mechanisms that are available can be used to go way beyond the standard unit test.

Unit tests

Without wanting to get academic about it, let’s define a unit test as being one that tests the logic encapsulated within a block of code without external side effects. Unit testing straightforward classes is trivial. If you want to make use of external service classes, these can be mocked using your favourite mocking library and injected into the class under test. Camel routes are a little different, in that what they define isn’t executed directly, but rather builds up a set of instructions that are handed to the Camel runtime for execution.

Camel has extensive support for testing routes defined using both the Java DSL as well as the Spring/Blueprints XML DSL. In general the pattern is:

  1. instantiate a RouteBuilder or Spring context containing the routes with a CamelContext, and start the context (this is handled for you by CamelTestSupport or CamelSpringTestSupport – see Camel testing). These should contain direct: endpoints as the inputs to the routes (consumers) and mock: endpoints as the outputs (producers).
  2. get a hold of the mock endpoints, and outline the expectations. A MockEndpoint itself uses a directed builder DSL to allow tou to define a suite of comprehensive expectation, ranging from checking the number of messages received to the details of an individual message. You can make full use of Camel expressions in these tests as well.
  3. create messages that you want to feed in to the route and send them to the direct: endpoint at the top of the route under test using a ProducerTemplate.
  4. assert that the mock endpoints received the expected messages.

An example of this approach can be seen in the RssConsumerRouteBuilderTest in the horo-app I blogged about yesterday.

There are a couple of things that you need to employ this approach successfully. If using Java, the RouteBuilder class that defines your routes should have the ability to have the route endpoint URIs injected and any beans that touch external resources into it – see RssConsumerRouteBuilder. The external beans can easily be mocked as in a standard unit test.

Using the Spring DSL, we can still employ the same general approach, but we need to jump through a couple of hoops to do it. Consider what you would need to do the equivalent. A simple route might be defined via:

    <route id="fileCopyRoute">
        <from uri="file:///some/directory"/>
        <to uri="file:///some/other/directory"/>
    </route>

You can externalise any URIs using Spring’s property support:

    <route id="fileCopyRoute">
        <from uri="${fileCopyRoute.input}"/>
        <to uri="${fileCopyRoute.output}"/>
    </route>

You could then define a PropertyPlaceHolderConfigurer with a properties file that defines these properties as

fileCopyRoute.input=file:///some/directory
fileCopyRoute.output=file:///some/other/directory

The definition of this class should be in a Spring context file seperate to that of your route definitions. For testing you would run the routes with another test XML file that defines a PropertyPlaceHolderConfigurer that points to a test file with the test URIs:

fileCopyRoute.input=direct:fileCopyRoute.in
fileCopyRoute.output=mock:fileCopyRoute.out

This is usually why Spring DM/Blueprints based bundle projects split the config across (a minimum of) two context files. One (META-INF/spring/spring-context-osgi.xml) contains all of the beans that touch the OSGi runtime including the properties mechanism, and the other (META-INF/spring/spring-context.xml) contains your physical routes. When testing you can easily switch out the OSGi bits via another config file. This allows you to inject in other bits during a unit test of the XML-based routes, or when using the camel-maven-plugin in order to run those routes off the command line without an OSGi container like ServiceMix.

Embedded integration tests

Sometimes, testing just the route logic isn’t enough. When I was building out the horo-app, I happily coded up my routes, tested tham and deployed, only to have them blow up immediately. What happened? The objects that I was expecting to receive from the RSS component didn’t match those the component actually sent out. So I changed tact. To engage the component as part of the route I needed a web server to serve the file that fed the test.

Integration testing is usually pretty problematic in that you need an external system servicing your tests – and when you are in an environment where the service changes, you can break the code of the other people working against the same system. But there is a solution! Sun’s Java 6 comes with an embeddable web server that you can start up as part of your integration tests.

The approach that I used was to spin up this server at the start of my test, and configure it programatically to serve up a response suitable for my test when a certain resource was consumed. The server was started on port 0, which means that it’s up to the runtime to assign an available port on the machine when the test runs. This is very important as it enables multiple instances of the same test to run at the same time, as is often the case on CI servers. Without it, tests would trip over each other. Similar approaches are possible using other embeddable server types, such as LDAP via ApacheDS, messaging via ActiveMQ, or databases via H2 or Derby.

Tests that require an external resource often start failing on large projects without any changes on the programmer’s side due to this exact reason – the underlying system dependencies changing. By embedding the server to test your integration against, you decouple yourself from that dependency.

The routes in your test then inject the URI to the embedded resource. In my case, I whipped up an integration test version of the original unit test (RssConsumerRouteBuilderITCase) to do exactly this. Integration tests can be wired in to a seperate part of the Maven build lifecycle using the maven-failsafe-plugin and use a different naming convention (*ITCase.java as opposed to *Test.java).

Usually the way the you structure your tests to avoid duplicating the lifecycle of these embedded backends ends up relying on a test class hierarchy, which may end up looking like:

  • CamelTestSupport
    • CamelTestSupportWithDatabase
    • CamelTestSupportWithWebserver

which I don’t really like, as you inevitably end up requiring two kinds of resource in a test. A much better option is to manage these extended resources using JUnit’s @Rule annotation. This treats any object that implements the org.junit.rules.ExternalResource interface as an aspect of the test, stopping and starting it as part of the test’s lifecycle. As such, you can compose your test of as many of these dependencies as you like – all without a rigid class hierarchy.

This approach allows you to test your integration code against a physical backend, without requiring that backend to be shared between developers. This decouples your development from the rest of the team and allows your integration tests to be run in a CI server. A huge win, as only tests which are deterministic end up being run and maintained in the long term.

#winning!

System Integrations as Plugins using Camel and ServiceMix

I recently had a client with a use case that I thought would be interesting to share (and they were happy for me to talk about – no names, industry changed). Sample code and full instructions for the solution available as always at FuseByExample on Github.

Imagine a system integration where the core logic is static, but the systems that participate in a particular process change over time. Take as an example, a travel booking system that accepts orders for buying flights. How do you go about adding new integration logic for a particular capability such as the booking of a flight with a new airline via its system? On top of this:

  • without changing any of the core logic around the main business process (the booking of a flight also includes taking payment)
  • with no application downtime (hot deployment)
  • enabling other development teams to define integrations for new airlines

Conceptually, these requirements could be satisfied via an application-level “plugin” solution. Your core flight booking application forms a platform alongside which you deploy capabilities specific to individual airlines. This may seem like a complicated set of requirements, but there is a set of tools that you can use to enable exactly this type of application.

Using Camel provides us with a way that to easily partition integration logic into a core process (flight bookings) and sub-processes (booking a ticket with a particular airline), and dynamically route to the right sub-process depending on the booking request. Routes can be connected within the one container by using the ServiceMix NMR to pass messages beteen bundles.

Deploying logic that is specific to an individual airline as an OSGi component enables the separation of code away from the core process, and provides the required dynamicity (and allows others to write that logic). The trick is in putting it all together.

Conceptual overview

OSGi bundles can be thought of as mini-applications. Basing a bundle on SpringDM/Blueprints (essentially defining a Spring-like config in a known location), we can embed a Camel context inside it and have routing logic start up when that bundle is deployed. This is a good candidate for our airline-specific booking code.

We then need to somehow advise the main application bundle that the new process is on-line and ready to do business by having messages routed to it. To achieve this part, we use of the OSGi service registry. To those unfamiliar with it, the registry acts as a conceptual whiteboard inside an OSGi container. Our plugin bundles can register beans inside the registry as implementors of an interface. The core application that wishes to use these services looks up the registry to get a handle on the implementations.

To advertise the availability of a sub-process and provide the name of the Camel endpoint that accepts bookings for a particular airline, we use an interface that indicates to the main application that a plugin bundle is available to take bookings. This is placed in its own bundle such that it could be implemented in the airline-specific bundles, and used in the main booking process.

public interface BookingProcessor {
        public String getAirlineCode();
        public String getBookingRouteUri();
}

Each airline bundle defines its own implementation that returns the airline code that it accepts booking messages for, and the URI for the endpoint of the Camel route that it would be listening for messages on.

Bundle dependencies

This object is then registered with the OSGi service registry:

<osgi:service ref="germanAirlinePlugin" 
        interface="com.fusesource.examples.booking.spi.BookingProcessor" />

The main process bundle is then able to get a dynamic proxy to a set of BookingProcessors that may come and go:

<osgi:set id="bookingProcessors" 
        interface="com.fusesource.examples.booking.spi.BookingProcessor" 
        cardinality="0..N"/>

This set can then be injected into a bean (BookingProcessorRegistry) that makes decisions such as:

  • Is this airline currently supported by the system?
  • What is the route that can be invoked to process this booking?

The Camel routing logic can then be really simple:

<route id="placeBooking">
	<from uri="jetty:http://0.0.0.0:9191/booking" />
	<transform>
		<simple>${headers.flightNumber}</simple>
	</transform>
	<choice>
		<when>
			<method bean="bookingProcessorRegistry" method="isAirlineSupported"/>
			<recipientList stopOnException="true" strategyRef="bookingResponseAggregationStrategy">
				<constant>direct:takePayment, direct:placeBookingWithAirline</constant>
			</recipientList>
		</when>
		<otherwise>
			<transform>
				<simple>Unable to book flight for ${body} - unsupported airline</simple>
			</transform>
		</otherwise>
	</choice>
</route>

<route id="placeBookingWithAirline">
	<from uri="direct:placeBookingWithAirline" />
	<!-- work out who to send the message to -->
	<setHeader headerName="bookingProcessor">
		<method bean="bookingProcessorRegistry" method="getBookingProcessorUri"/>
	</setHeader>
	<log message="Calling out to ${headers.bookingProcessor}"/>
	<recipientList>
		<header>bookingProcessor</header>
	</recipientList>
</route>

So to make a new airline available for bookings, you:

  1. create a new OSGi bundle
  2. register a BookingProcessor in the service registry that indicates the route that processes these bookings
  3. write the integration logic in a route that listens on that endpoint
  4. build the bundle and drop it into Servicemix alongside the main process.

Voila! An application-specific plugin system. You can then use the Karaf web console as a mechanism to make the bundle logic available via REST to a single container, or if you want to distribute it across a cluster – Fuse Fabric via Fuse ESB Enterprise.

Developing web services in ServiceMix

I have added a number of projects to FuseByExample/smx-ws-examples on GitHub that demonstrate how to go about developing a number of common web service use cases. The samples are designed to get you up and running quickly with SOAP based web services in an OSGi world.

The examples include:

  • a Maven project that generates all relevant Java code from a WSDL using the cxf-codegen-plugin, and wraps it in an OSGi bundle
  • a plain Java web service implementation of a web service using CXF
  • an implementation of the web service using a Camel CXF route
  • a web service proxy using a Camel route
  • a client based on a Camel route that uses the CXF component to invoke those web services; no Java code required

As usual, full documentation in the README. Enjoy!

Build-time integration testing OSGi bundles in ServiceMix

I have just added a new project to GitHub under FuseByExample/smx-integration-testing that you can use to verify at build time that your bundles and features will deploy and run as expected in a specific version of ServiceMix. The intention is that you can use it to kick start testing in your own projects.

To do this I hooked up the process described by Ioannis Canellos in Advanced integration testing with Pax Exam Karaf to start up and deploy your project artifacts into an embedded FuseSource distribution of ServiceMix (4.4.1-fuse-01.13 at time of writing), using the maven-failsafe-plugin for integration testing.

The result is a build that compiles and assembles your bundle or features project, unit tests it, packages it up for installation and then integration tests the binary using pax-exam-karaf and JUnit.

If your bundle fails either to install into the embedded ServiceMix container, or fails any of the criteria defined in your integration test, the build will fail and the build artifact won’t end up being installed to your Maven repo.

The sample demonstrates:

  • the wiring required to assemble this approach in Maven
  • the separation of test artifacts for a SpringDM bundle into both unit and integration tests; abstracting out the OSGi bits for unit testing while ensuring ${} placeholders work, and
  • shows a number of ways to exercise the code in your bundle via:
    • straight Camel route testing
    • using the OSGi BundleContext to verify that a bundle is installed and active
    • checking that a SpringDM OSGi service is deployed via the @Inject mechanism

The documentation in the README file goes into full detail as to how this is achieved.

Enjoy!

Bootstrap Projects for Getting Started with ServiceMix 4

Over the Christmas break I cleaned up and published a set of Maven projects for getting started with ServiceMix 4.4.1+ into GitHub. I found myself reusing the same code for a number of activities, and figured it may be of broader use to others. You can find it under FuseByExample/smx-bootstraps.

smx-bootstraps contains within it a set of OSGI Blueprints (DI, based on the ideas behind Spring) bundles that exercise some of the core things that you will want to do with ServiceMix:

  • defining services in bundles that can be reused in other bundles
  • use Camel for writing integration code
  • use ActiveMQ for sending persistent messages between bundles, regardless of whether they are in the same container or in others
  • request-response over messaging
  • Update 20/02/2012: RESTful web services!!!
  • externalising your environment configuration
  • group bundles into features

The README document at the project root explains how to get started with ServiceMix, deploy bundles, change code and play with config.

Why would you want to use them?

ServiceMix went through a massive generational change between versions 3.X and 4.X, moving from JBI to an OSGi based model. While development work on it is proceeding at a huge rate, the documentation hasn’t kept up – although it is being brought up to date in the background. smx-bootstraps contains small artifacts that are hopefully easy to understand and play with, along with instructions on how to use them in the container.

The project may also be use of use as a starting starting point to further development. A fairly clean project layout exists that you can use as a reference point, which acts as a supplement to the Maven archetypes that are publicly available, such as:

  • org.apache.camel.archetypes:camel-archetype-blueprint for generating Blueprint bundles to run Camel routes; similar to smx-pinger and smx-ponger bundles in smx-bootstraps
  • org.apache.karaf.archetypes:karaf-blueprint-archetype for generating simple Blueprint bundles; such as the smx-ponger-service bundle

I have found these bundles to be a really handy way of exercising ServiceMix features, and working with various configurations. Not having to code up something new each time is a huge time saver. Hopefully you should find this as well.

I expect to expand this little project as time goes on and I find myself recreating other use cases, such as exposing web services – next on my “todo list”. Please drop me a line at “jakub dot korab at gmail” if you find this useful or have any ideas that would fit in well. Of course being GitHub, feel free to fork it or contribute back changes.

Coincidence?

Jason Sankey just posted a list of OSGi tutorials: http://www.alittlemadness.com/?p=80

You have to love RSS…

OSGi vs. JSR-277

I checked out the OSGi presentation from the Parleys site http://www.bejug.org/confluenceBeJUG/display/PARLEYS/Spring+OSGi. Wow. I haven’t really spent any time thinking about this stuff in the past, but it seems that yet again the JSR process is trying to formalize something that doesn’t need it. OSGi seems to be a much lighter, more powerful model whereby you can switch bundles, the JSR-277 module equivalents, at runtime via an OSGi console or JMX. Check out http://www.osgi.org and http://en.wikipedia.org/wiki/OSGi for good synopses of the technology and what it can offer.

The huge advantage here is that the technology is stable, mature and available to use right now (there are a number of open-source implementations) rather than needing to wait for something that won’t be around until Java 7. All that and no new language structures.

There’s an interesting post by Peter Kriens from OSGi about how JSR-277 stacks up at
http://www.osgi.org/blog/2006/10/jsr-277-review.html

"The ambition level of JSR 277 is significantly lower than where the OSGi specifications are today, even way lower than we set out in 1998. In a way, for me the JSR 277 proposal feels rather toyish. "

It’s time to kick the tires on this one, and see where it might be useful.