Mar 29

Over the last couple of years, I developed quite a strong professional network in Dublin, which was working quite well in getting contract IT work. With the recent move to London, I have found that I may now have to fall back onto more traditional forms of hunting for contracts, i.e. through agents and job boards, even though I intend to keep pushing on other fronts.

It has quickly become apparent that the City is quite a different environment due to its scale, and that the traditional job board search methods are simply not up to the task. Consider the following job stats:

  • London has a population of 18 million, Dublin just over 1
  • Contract stats from listings on CWJobs/Jobserve in the last 2 days using the following search terms:
    • Java 44/110
    • .Net 57/120
    • Spring 30/27
    • Agile 24/33

Lots of contracts, contractors, agents and potential clients. Work out what you will about the environment from the numbers. Separating the wheat from the chaff takes time, and anything that makes the process too difficult for any of the parties involved is going to get skipped over.

First thing job boards.

With such a huge turnover of work, you need to keep track of stuff that you have seen before as roles get listed on an ongoing basis. You also need to make sure that any search automatically drops out stuff that’s not relevant. You probably don’t care about release management jobs or C++ if you are looking for Java work, and a lot of emphasis is placed here on industry experience within a particular niche (a simple filtering mechanism for clients).

RSS feeds are the way of the future here. I set up a couple of complex boolean searches on Jobserve and set up feeds to fire off to my Google Reader. This way I don’t have to keep a big spreadsheet of stuff that I have applied for, and don’t have to sift through things that I have already seen in search results. Hours of dead time freed up for more productive activities. Before I set these up, it was a battle just finding relevant work to apply for.

I don’t really like this form of contract hunting as it is inefficient at this scale, and certainly prefer the networking model, but any strategies that make this easier are worth working out.

Adage have a great article that I found my way onto from the LinkedIn blog, about how you can use social networking sites to supercharge the process. I think it’s probably a much better way to operate. Updates coming soon.

If you are in London and looking for someone smart who gets things done, knows how to communicate, has the ability to get people exited about stuff, with a Java background to technical architect level, feel free to contact me through LinkedIn.

Jan 25

Unit testing database code is a bit of a funny problem. Most developers can pretty easily get their heads around unit testing a piece of Java code using interfaces and mock objects. When it comes to database code or DAOs, it suddenly becomes particularly difficult. But why, what is so difficult about testing stuff against the database? Surprisingly enough, the answer is that it has nothing to do with coding or a particular framework, although these do play their parts. It comes down to a complex web of human interaction, version control and managing environments. Let me explain.

The standard unit test has three basic phases:

  • Setup (@Before)
  • Test (@Test)
  • Tear down (@After)

The first sets the test environment into an expected state, the second runs the test and checks that the outcome is as expected, while the final one clears up any test resources.

How does this relate to database testing? Let’s say that we have a DAO that performs a particular select statement. Our test should be to retrieve a particular number of records from a known set. Easy enough. The precondition of course, is that you have a known set to begin with.

It’s ALL about the environment.

Most large development projects go like this: The database guys update the schema. The developers write the code. The developers need a particular data set to exercise the various use cases so they add it to the schema. It all becomes a bit messy.

Eventually, very complex data sets are set up by everyone concerned in a primary schema that keeps getting updated. The database schema generally is not version controlled, as it is constantly being redefined using DDL statements run by the DBAs. Most of the time you will be lucky to get a backup of a schema, with all of the data truncated, as the schema and supporting code (i.e. the application) moves between environments.

Getting back to the test. You set up your data by hand in the master schema so that there were three items in the widgets table where some condition was true. You write your test, it runs against the schema, pulls out the expected three widgets and everything is great. You check in the tests. A week later your colleague, Bob, adds another widget to satisfy his test condition. Your test all of a sudden returns 4 items and the test breaks.

Of course, Bob didn’t actually run your test because he was too busy with his own and the test suite isn’t clean anyway because everyone is falling over each other.

Sound familiar?

What about inserts? The precondition: no sprockets were in the table, the test: insert a sprocket, the postcondition: a sprocket is in your table. Kind of hard to test under the above conditions isn’t it? For one thing, the exact data of the test sprocket may be in the table, so checking by value may give you false positives, while deleting it may get rid of more records than you wanted. What about concurrent tests? With a group of developers running the same tests, they start tripping over each other very quickly and the whole effort becomes an exercise in frustration. At this point the development manager throws up his hands, says that this automated testing thing is a load of bollocks and to get back to your work because they didn’t deal with all this when he was doing VB. Somewhere else, Kent Beck sheds a tear…

Let’s examine what goes on in Ruby on Rails. One of the best ideas that was popularized by this framework was its method for database unit testing. A developer’s workspace has multiple environments by default - development, test and production. You develop against the development schema, designing table structures, and playing with the user interface to your heart’s content. When you run unit tests, the following happens - the schema from development is copied into the test database with no data in it. The framework imports version controlled sets of test data (saved as YAML files) into this new schema. Whenever a test is run, it is guaranteed that the database will be in this state. Any changes a test makes are visible only within the scope of this one test. The tear down step cleans out your changes. This makes life so much simpler, especially if you have been working in the nightmare scenario above.

So how do we get the same sort of effect in a corporate development environment?

You need multiple database schemas in order to unit test your db code.

Pause and re-read that line. It’s not negotiable. Probably two per developer. One with sample data to use while you work on the user interface. The other, a temporary one for unit testing. A whole development team using the one schema does not work. Most projects do it, but that doesn’t mean that it’s a good idea.

Some suggestions for how to manage this. The DBAs have their own schemas. The full DDL for the database is kept in version control. After each change, the full database DDL is dumped and checked in. No UPDATE TABLE statements. Ever. This way you are guaranteed that if you ever want to get a baseline of your system, you can also rebuild the database as it existed at this time. I worked on a very large telecoms project with a huge development team, and this worked. Well.

The test data for your environments is stored in version control - at the very least, as dumps of insert statements. For unit testing purposes, a dedicated unit test framework is beneficial. DBUnit performs the same task in Java as described above for Rails - it loads test data from dumps (a number of formats are supported), and guarantees that the test database exists in the expected state when each test is run.

To test your database code, refresh your test schema with the one from version control - typically using your chosen build system. Ant tasks are generally pretty good for this. Now run your test cases. Gorgeous! No tripping over other people, and your tests are guaranteed to work the same each time. No excuses for a red bar.

So why is unit testing databases so difficult if it doesn’t have to be? Most of the time it involves process change and getting out of bad habits, not just a tool. And change means convincing people. Generally, managers do not understand what benefit there is in multiple database schemas, as it is seen to increase complexity and therefore risk, and DBAs like to have full control over what is going on on their servers. The topic of databases and processes is also a great one for religious zeal.

The process outlined above should explain the hows and whys to the individuals involved. The changes above mean a little bit more setup initially, but a saner development process.

A nice side effect is how easy upgrading databases through your environments can become. Run the latest DDL against a fresh schema, get the differences between it and your target environment using a database compare tool, and fire it off. Beautiful.

Aug 23

I have worked with a number of different employer-provided UML tools in the past and have often been left underwhelmed. Rational Rose is a complex memory-hogging beast, ArgoUML seems clunky (although I’m happy to work with it at home since it’s free), and older versions of Visio have needed Pavel Hruby’s stencil to provide good, if fairly basic UML support.

Which is why I have been so pleasantly surprised using Visio 2003’s native UML Model Diagram - it has the CASE like features of the others (Java code generation excluded - surprise, surprise) and it “Just Works”. Within half an hour I was happily churning out structure and collaboration diagrams. The defaults are pretty intuitive (4/5) and standard actions such as moving methods between classes/interfaces, and repackaging classes are as simple as drag and drop. Change the structural details of your classes in the Model Explorer and all your diagrams update just the way you expected them to.

Java support is non-existent out of the box (it supports VB, IDL, C# and C++), but the C# native types are close enough that I’m not that fussed.

Credit where credit is due - any tool that makes me this productive also makes me very happy.

Aug 13

Since the advent of dependency injection (DI) as a staple of enterprise development using tools such as Spring or Guice, your code has become a lot easier to test. You no longer need to code up voodoo such as plugging in dummy resource locators or the like based on some random environment variable to tell your code to switch into “test mode”. Everything is a POJO with interfaces as dependencies rather than classes, which means that it can be easily tested as a POJO. Nevertheless, it still surprises me as to how many people don’t know how to do this. Perhaps it is the perception that you need to have a running back end to test a service tier, or that you need the rest of the system to test a web front end. Not so.

Enter the mock object. According to Wikipedia - “mock objects are simulated objects that mimic the behavior of real objects in controlled ways”. And thanks to freely available Java toolkits such as EasyMock and jMock, making them available to your tests is as easy as plugin in a Jar file into your IDE. I’m a heavy user of EasyMock, so what follows is a simplified example of a jUnit test testing the middle tier of your typical DI-based application.

The following example deals with the following classes:

  • WorkItem - a simple value object
  • WorkItemException - an exception
  • WorkItemService - an interface of a middle tier class
  • WorkItemDao - an interface to the data layer of our application. This is what we are going to mock up.
  • WorkItemServiceImpl - an implementation of that class. This is what we are going to test.
  • WorkItemServiceImplTest - our test class

First, let’s get the uninteresting classes out of the way. Our WorkItem is just a bean that gets passed between the layers:

package jkorab.example.mock;

public class WorkItem { int workItemNo;

 public int getWorkItemNo() {    return workItemNo; }

 public void setWorkItemNo(int workItemNo) {   this.workItemNo = workItemNo; }}

The WorkItemException is an exception that gets thrown:

package jkorab.example.mock;

public class WorkItemException extends Exception {  private static final long serialVersionUID = 1L;}

Now we come to the class that normally sits in the business tier of our application:

package jkorab.example.mock;

public interface WorkItemService {

  /**   * Trivial method. Given a workItem number > 0, returns a workItem instance.   * Used to show how you might test the wrapping of an exception.   *    * @return WorkItem or null.   */  public abstract WorkItem generateWorkItem(int workItemNo);

}

Let’s take a look at the interface of the class it uses to do its work:

package jkorab.example.mock;

public interface WorkItemDao {

  /**   * Given a workItem number > 0, returns a workItem instance.   * @param workItemNo workItem number   * @return WorkItem if workItemNo > 0.   * @throws WorkItemException if workItemNo <= 0.   */  public WorkItem getWorkItem(int workItemNo) throws WorkItemException;

}

Our example is, as is the spirit of most tutorials, pretty contrived. The top tier of our application wants to load a WorkItem, so it passes a work item number to the service tier. The service tier gets the work item, and if it cannot find it, returns null. The DAO on the other hand, gets a work item number and it the number is less than or equal to 0, it throws an exception, otherwise it loads the WorkItem.

Here’s the implementation of the WorkItemService:

package jkorab.example.mock;

public class WorkItemServiceImpl implements WorkItemService {  private WorkItemDao workItemDao;

  public void setWorkItemDao(WorkItemDao workItemDao) {    this.workItemDao = workItemDao;  }

  /* (non-Javadoc)   * @see jkorab.example.mock.WorkItemService#generateWorkItem(int)   */  public WorkItem generateWorkItem(int workItemNo) {    WorkItem workItem = null;    try {      workItem = workItemDao.getWorkItem(workItemNo);    } catch (WorkItemException workItemException) {      // log and continue. we’ll return null.    }    return workItem;  }}

The class here is doing what the interface “said on the tin”. In addition it has a setter so that the DI container can supply it with a DAO implementation. That’s where our hook is that will enable us to test the service without an actual DAO implementation.

The first thing we need in order to write a test class is a mock DAO. This is the component that the service class will call. In addition we will also need a mock control. Think of this as a component that instructs your DAO as to how it should behave - kind of like a way of programming your Tivo/Set top box/VCR (for those who still have one ;)).

MockControl control = MockControl.createControl(WorkItemDao.class);WorkItemDao mockDao = (WorkItemDao) control.getMock();

WorkItemServiceImpl service = new WorkItemServiceImpl();service.setWorkItemDao(mockDao);

The MockControl class is used to get an instance appropriate to the interface we are mocking up. From this, calling getMock() will return to us an implementation of our DAO interface which we then set on the instance of the class we are testing. This would normally be done in the setUp() method of your jUnit test class, so that each of your tests is guaranteed to get a mock instance in a fresh state, which you can then play with.

In our test method, we can then get on with doing some work. First, we tell our DAO exactly how we would like it to behave:

// set the methods that you expect to be calledmockDao.getWorkItem(1);

// set the return valueWorkItem returnedWorkItem = new WorkItem();returnedWorkItem.setWorkItemNo(1);control.setReturnValue(returnedWorkItem);

// reset the control object to play back our scriptcontrol.replay();

The last line tells the mock control object that the mock DAO is now in a state where the test can be run on it.

WorkItem workItem = service.generateWorkItem(1);assertNotNull(workItem);assertEquals(1, workItem.getWorkItemNo());

// verify that the methods we expected to be called on the mock object// were actually calledcontrol.verify();

The last line tells the control object to check itself to ensure that all the methods that we told it would be called were in fact called. It will throw an Exception if they weren’t causing our test to fail.

The behaviour described above can be used to test the behaviours of your service class under any circumstances. We can mock up the throwing of an Exception, unexpected behaviours, whatever we want. Here’s the full test class:

package jkorab.example.mock;

import jkorab.example.mock.WorkItem;import jkorab.example.mock.WorkItemDao;import jkorab.example.mock.WorkItemException;import jkorab.example.mock.WorkItemServiceImpl;

import org.easymock.MockControl;

import junit.framework.TestCase;

public class WorkItemServiceImplTest extends TestCase {  private MockControl control;  private WorkItemDao mockDao;  private WorkItemServiceImpl service;

  public void setUp() {    control = MockControl.createControl(WorkItemDao.class);    mockDao = (WorkItemDao) control.getMock();

    service = new WorkItemServiceImpl();    service.setWorkItemDao(mockDao);  }

  /**   * Checks that a WorkItem will be returned if the class receives a workItem   * number greater than 0.   *    * @throws WorkItemException   */  public void testGenerateWorkItem() throws WorkItemException {    // set the methods that you expect to be called    mockDao.getWorkItem(1);    // set the return value    WorkItem returnedWorkItem = new WorkItem();    returnedWorkItem.setWorkItemNo(1);    control.setReturnValue(returnedWorkItem);    // reset the control object to play back our script    control.replay();

    WorkItem workItem = service.generateWorkItem(1);    assertNotNull(workItem);    assertEquals(1, workItem.getWorkItemNo());

    // verify that the methods we expected to be called on the mock object    // were actually called    control.verify();  }

  /**   * Checks that a null will be returned if the class receives a workItem   * number equal to 0 rather than throwing an exception.   * @throws WorkItemException    */  public void testGenerateWorkItemNullValue() throws WorkItemException {    // set the methods that you expect to be called    mockDao.getWorkItem(0);    // set the control to throw an exception    control.setThrowable(new WorkItemException());    control.replay();

    WorkItem workItem = service.generateWorkItem(0);    assertNull(workItem);  }}

This is a great way to test, not only because of its simplicity, but also it being future proof. Let’s examine another way of writing the same tests without using mock objects:

WorkItemDao mockDao = new MockDao() {  public WorkItem getWorkItem(int workItemNo) throws WorkItemException;    WorkItem returnedWorkItem = new WorkItem();    returnedWorkItem.setWorkItemNo(1);  }};

WorkItemServiceImpl service = new WorkItemServiceImpl();service.setWorkItemDao(mockDao);

WorkItem workItem = service.generateWorkItem(1);assertNotNull(workItem);assertEquals(1, workItem.getWorkItemNo());

At first glance, coding up an anonymous class looks like less work, but consider the following:

  • if your class is more complex than this (and to be honest, most are) then you will end up with potentially dozens of anonymous implementations.
  • if the DAO has more than one method, you will end up with a lot of code that does nothing.
  • if the interface on your DAO changes then you will end up with a lot of tests that won’t compile, because your classes no longer match the interface.

This just isn’t the case with mock objects. Mock objects are created at runtime, and you only have to define the behaviours that you are interested in. The technique can also be used to test existing code - refactor your dependencies to an interface, use DI or a constructor if this isn’t an option, and voila, a testable class. 100% code coverage, and future-proof. No more excuses.

If you are interested in more info on mock objects check out the MockObjects blog.

Jul 27

I tracked down a Javascript component today that allows you to format code snippets in your blog. Syntaxhighlighter by Alex Gorbatchev. Most popular languages supported.

Had to do a minor patch to run it off blogger, but it now works like a treat. Check out my last post on EJB.

Jul 27

It’s finally time to say goodbye to my trusty Venkman debugger for Firefox. My old friend has served me well for Javascript development, but I have found a new, better tool: Firebug. Javascript, DOM, XHR debugging, profiling, viewing, command line interface, profiling. Mmm…