Kafka partition calculator for online news site data workloads

Best practices

News stories and content move from source to on-line news sites in real-time. Apache Kafka's durable, and fault-tolerant publish-subscribe messaging system is tailor-made to handle the demands of the news industry.

Streaming news articles are medium-sized, roughly around 500KB. The news article messages include text, metadata, and small references to multimedia elements. Message rate is variable, largely dependent on the size of the news articles. Peak activity typically occurs in the morning when new articles are published, with activity trailing off in the evening.

If you’re developing an online news or content streaming system, the inputs below are good default inputs to use for the partition calculator.

Read assumptions

The brokers are of similar capability.
The load on the brokers’ machines is similar.
The messages don't diverge too much in size.
The messages are evenly distributed across all partitions.
The number of brokers makes sense in this context.
Brokers have similar latencies between producers and consumers.
The throughput per producer is less than 10MB/s.
Individual brokers have less than 40k partitions.
The cluster has less than 200k partitions in total.

Kafka UI for your team

Get started free

Results

number of producers

number of consumer

Expected lag

number of partitions

Business size preset

The size of the company in terms of article production (not number of employees, readership numbers, market capitalization, etc…)

It affects calculations as it influences mostly data volume, and resource needs.

Medium

Number of brokers

The number of brokers in the cluster where the topic will be created.

This affects calculations as the load put in by producers and consumers should be equally distributed between all the brokers.

1 140

Producer processing time

The average time it takes to produce a message in ms.

This affects calculations as it puts a hard limit on how fast a single producer can produce messages.

0.1 ms1,000 ms

Consumer processing time

The average amount of time it takes to consume a message in ms.

This affects calculations as it puts a hard limit on how fast a single consumer can consume messages.

0.1 ms1,000 ms

Throughput

The amount of messages that the system should process per second.

This affects calculations as it will determine how much parallelism is required to achieve the desired throughput.

msg/s

1 msg/s10,000 msg/s

Results

number of producers

number of consumer

Expected lag

number of partitions

How to increase partitions

Learn how to to increase your topic partitions and what effects this will have on your cluster.

Read full article

Recommended configuration for a medium business size

Desired throughput (T) is quite low, but is an appropriate amount of articles of a newspaper company of this size.
Time to produce a single message (P) is 50 ms, given the medium-sized nature of the articles, on average.
Time to consume a single message (C) is 1 ms, due to the small effort needed to process the messages.
The number of brokers (B) is also very low and balances the need for redundancy and resource management within the given constraints.
The expected lag (L) is 0, which is ideal. In the realm of news, consumers demand up-to-the-minute updates, making it critical to keep latency as low as possible.
The recommended number of producers (RP) and consumers (RC) is 2, matching the number of brokers. This configuration maintains an optimal balance between the message production rate and consumption rate.
The number of partitions for topics (NP) should also be 2, fitting within the broker and cluster partition limits.

Each element of this configuration ensures that drivers and passengers experience minimal latency in location updates, contributing to smooth and responsive services.