Towards the end of last year, I developed a performance test tool which is available at GitHub for testing the behaviour of Apache Kafka (0.9 so far). The tool was inspired by, and informed by some of the limitations of, the performance tooling in ActiveMQ, which I’m currently retrofitting with the same mechanisms.
The kafka-perf-test project builds a Fat JAR that you can take with you into any environment running Java 8, and through the use of a single JSON or YAML config file configure up a range of consumers and producers with differing behaviours pointing at one or more Kafka installations. All of the standard Kafka properties for producers and consumers are configurable, which also makes it a useful learning tool if you want to work out what sort of behaviour to expect from a Kafka installation, or just to bulk load a bunch of messages in.
A sample YAML config can be as simple as:
---
config: {} # applies to all producers and consumers
producers:
config: # applies to all producers
bootstrap.servers: "tcp://localhost:9092"
request.timeout.ms: "10000"
key.serializer: "org.apache.kafka.common.serialization.ByteArraySerializer"
value.serializer: "org.apache.kafka.common.serialization.ByteArraySerializer"
batch.size: "0"
acks: "1"
max.block.ms: "10000"
instances:
- config: {} # applies to this producer
topic: "foo"
messagesToSend: 2000
messageSize: 1000
sendBlocking: false
There is still some work to go in getting decent metrics reporting, and general polish, but for headline numbers it’s good to go. Pull requests/issues/comments most welcome.