This is a huge announcement Confluent made yesterday. Just a couple of weeks after releasing their new product called KsqlDB, we are now with Kafka 2.4 online.
The thing to know about confluent releases is that when they announce a new Kafka version, this includes not only Kafka itself (which they call kafka core). When Confluent announces a new release, this includes a new release for the whole stack, meaning Kafka core, Kafka streams and Kafka Connect. Thus, the release presentation video by Tim Berglund is 15 minutes long.
I’m not going to dress the list of all the new features that were put upfront, but a few of them clearly caught my attention, and I think a few words about them is well deserved !
KIP-392 : Kafka now allows to read data on follower replicas. Before this KIP, reads were allowed only on the leader replica, that could cause an overload, and a lot of cross-datacenter traffic when the topic was setup in datacenter. Now, consumers can read from replicas where client.rack and broker.rack match. Confluent had communicated about this KIP a couple of weeks ago. Thanks so much for this one !
KIP-455 : Kafka proposes now an admin interface where you can manage your partitions reassignment. In the former versions, you needed to do it with Zookeeper, extract the partitions topology and work it in JSON format before resubmitting it (hoping all is fine). Tough game!
KIP-496 : It is now possible to remove a consumer group offset within the kafka-consumer-groups.sh command. This is going to be much more convenient, and no more need to trick this by playing with offset retention delays anymore…
KIP-525 : Not mind blowing, but quite efficient. When creating a topic, the topic metadatas will be output back to the console. Much better than the Topic created aknowledgement.
KIP-307 : KStreams DSL will now allow custom names to the processors in a topology. This will make the topologies much more readable to debug!
KIP-471 : RocksDB (used by Kafka Streams as a state store) metrics are now exposed within Kafka Streams metrics. This should help users to investigate issues on RocksDB and the use of their state stores.