Tuning Kafka for the Finance Sector: A Sharp Look at Fraud Detection

Best practices

There are a number of industries and use cases that feature streaming financial transactions. These include projects like securities trading, consumer banking, point of sales analysis, and fraud detection.

If we consider a system that streams financial data for fraud detection, it’s typical to see messages that are quite large, around 1MB in size. The messages might include transaction details, historical data, behavioral analytics, and more.

Financial data streams at a medium pace in fraud detection applications, usually around 10 messages per second (m/s). From time to time the streams might spike suddenly, based on global events, market conditions, or new emerging threats.

The partition calculator inputs below reflect these characteristics.

Read assumptions

The brokers are of similar capability.
The load on the brokers’ machines is similar.
The messages don't diverge too much in size.
The messages are evenly distributed across all partitions.
The number of brokers makes sense in this context.
Brokers have similar latencies between producers and consumers.
The throughput per producer is less than 10MB/s.
Individual brokers have less than 40k partitions.
The cluster has less than 200k partitions in total.

Kafka UI for your team

Get started free

Results

number of producers

1,600

number of consumer

800

Expected lag

500

number of partitions

1,600

Business size preset

The size of the company in terms of article production (not number of employees, readership numbers, market capitalization, etc…)

It affects calculations as it influences mostly data volume, and resource needs.

Medium

Number of brokers

The number of brokers in the cluster where the topic will be created.

This affects calculations as the load put in by producers and consumers should be equally distributed between all the brokers.

1 140

Producer processing time

The average time it takes to produce a message in ms.

This affects calculations as it puts a hard limit on how fast a single producer can produce messages.

0.1 ms1,000 ms

Consumer processing time

The average amount of time it takes to consume a message in ms.

This affects calculations as it puts a hard limit on how fast a single consumer can consume messages.

0.1 ms1,000 ms

Throughput

The amount of messages that the system should process per second.

This affects calculations as it will determine how much parallelism is required to achieve the desired throughput.

msg/s

1 msg/s10,000 msg/s

Results

number of producers

1,600

number of consumer

800

Expected lag

500

number of partitions

1,600

How to increase partitions

Learn how to to increase your topic partitions and what effects this will have on your cluster.

Read full article

Recommended configuration for a medium business size

A lag (L) of X1, a trade-off considering the size of the data chunks being processed.
An impressive X2 producer (RP) to maintain the required throughput.
The consumer count (RC) stands at X3, ensuring prompt data consumption.
Partitions (NP) skyrocket to X4, maximizing data distribution for optimal analysis.

This Kafka setup ensures that financial transaction data flows efficiently so that financial institutions can quickly detect and react to fraudulent activities, safeguard assets, and maintain trust.