The algorithm uses a sophisticated and efficient approach designed to determine the optimal number of partitions for Kafka topics, enhancing the performance and scalability of Kafka clusters. By considering essential factors such as message throughput, producer and consumer processing times, number of Kafka brokers, and failure tolerance, this algorithm offers accurate recommendations for partition size, number of producers, and consumers.

Leveraging mathematical formulas and Little's Law, the algorithm adjusts the lag, producer and consumer numbers to achieve the desired throughput while ensuring even data distribution across partitions. The result is a powerful tool that simplifies Kafka cluster optimization and empowers users to build robust and high-performing data streaming solutions tailored to their specific needs.

• MP → ceiling(T * P / L / F)
  • ex1: ceiling(1000 m/s * 0.001 s/m / 0.8 / 0.99) = 2 MP
  • ex2: ceiling(1000 m/s * 0.1 s/m / 0.8 / 0.99) = 127 MP
• MC → ceiling(T * C / L / F)
  • ex1: ceiling(1000 m/s * 0.001 s/m / 0.8 / 0.99) = 2 MC
  • ex2: ceiling(1000 m/s * 0.1 s/m / 0.8 / 0.99) = 127 MC
• RMP
  • RMP’ → round up (MP / L / F)
  • ex: 2 / 0.8 / 0.99 = 3
  • if RMP’ ≥ B: ((RMP’ + B - 1) // B) * B
    • // being the int division or the round down of the division result
    • ex: ((3 + 2 - 1) // 2) * 2 = 4
  • if RMP’ < B: start with RMP’ and enumerate integers towards B as RMP’’, take the first number where B mod RMP’’ is 0
    • ex: B = 15 and RMP’ = 4
      • step 1: RMP’’ = 4 → 15 mod 4 = 1 (no match)
      • step 2: RMP’’ = 5 → 15 mod 5 = 0 (match)
      • RMP is 5
• RMC
  • MC’ → round up (MP / L / F)
  • ex: 2 / 0.8 / 0.99 = 3
  • if MC’ ≥ B: ((MC’ + B - 1) // B) * B
    • // being the int division or the round down of the division result
    • ex: ((3 + 2 - 1) // 2) * 2 = 4
  • if MC’ < B: start with MC’ and enumerate integers towards B as MC’’, take the first number where B mod MC’’ is 0
    • ex: B = 15 and MC’ = 4
      • step 1: MC’’ = 4 → 15 mod 4 = 1 (no match)
      • step 2: MC’’ = 5 → 15 mod 5 = 0 (match)
      • MC is 5
• L → T * C (Little's Law)
• ex: 1000 m/s * 0.1 s/m = avg 100 messages waiting to be processed (lag)
• NP
  • NP’ → min(RMC, RMP)
  • NP'’ → multiply NP’ until it's bigger than max(RMC, RMP)
  • NP''’ → NP'’ - max(RMC, RMP)
  • NP''’’ → max(RMC, RMP) - min(RMC, RMP)
  • NP →
    • if NP''’’ < NP'’’ → max(RMC, RMP)
    • if NP''’’ > NP'’’ → NP'’
    • else min(RMP, RMC)
  • ex
    • B = 12, RMP = 4, RMC = 3 → 4
    • B = 3, RMP = 6, RMC = 9 → 9
    • B = 20, RMP = 100, RMC = 160 → 200
    • B = 11, RMP = 77, RMC = 33 → 99
• RP
  • if mod(max(RMP, NP), min(RMP, NP)) = 0 → RMP
  • else NP
• RC
  • if mod(max(RMC, NP), min(RMC, NP)) = 0 → RMC
  • else NP