A Neate Blog

20 May 2020

Setup EmbeddedKafka for Unit Testing Kafka in Java

Tags: Java - Spring - Kafka

Unit testing Kafka can be a pain, it seems a lot of overkill to have an entire Kafka cluster ready just for unit testing purposes, luckily, spring-kafka-test helps with this by providing an Embedded Kafka instance you can use to produce or consume messages.

This article contains a short How-To guide on setting up Embedded Kafka to be used for unit testing an application.

The first thing needed is to update the pom.xml file to ensure you have the various Spring Kafka dependencies.

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.4.6.RELEASE</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <version>2.4.6.RELEASE</version>
    <scope>test</scope>
</dependency>

At the time of writing, there was an issue with the spring-kafka-test module not shutting down correctly after a mvn test execution hence the additional spring-kafka import.

After updating the POM file, the unit test class can be annotated with the Embedded Kafka details such as

@EmbeddedKafka(brokerProperties = {"log.dir=target/embedded-kafka/exampleService"}, controlledShutdown = true)
public class ExampleServiceTest

If any of the properties are omitted then the default ones are used, they aren’t mandatory.

After annotating the Java Class, the actual broker needs to be defined and initialised.

@ClassRule
public static EmbeddedKafkaRule embeddedKafkaRule = new EmbeddedKafkaRule(1, true, "example-topic-name");

private EmbeddedKafkaBroker embeddedKafkaBroker = embeddedKafkaRule.getEmbeddedKafka();

The constructor parameters passed to EmbeddedKafkaRule are:

  • int count - The number of desired brokers
  • boolean controlledShutdown
  • java.lang.String… topics - The topics to create

One final point to note, if your service uses properties to store the Kafka bootstrap servers address (Hostname for the Kafka server) then you can add the following in your application.properties file to extract the address of the EmbeddedKafka broker.

The property spring.embedded.kafka.brokers contains the address of the EmbeddedKafka instance meaning you can use this value in your application, for example if your service uses properties to store the Kafka bootstrap servers address (Hostname for the Kafka server) then you can add the following in your application.properties file to extract the address of the EmbeddedKafka broker.

custom.kafka.server_url=${spring.embedded.kafka.brokers}

Needless to say your property would replace custom.kafka.server_url

After following this short how-to, the Embedded Kafka Broker is ready to be used by your favourite Kafka library interaction whether that is Reactor, Spring or a custom one you’ve built.

TL;DR:- Using embedded kafka is a simple way to run unit tests that don’t have or need a local Kafka instance, such as your CI server

Useful Links: