Distributed Messaging System { Kafka }

Authors: Ricky Das, Vipin Kumar Dhiman

Abstract: Nowadays, log processing is very important for internet-based companies. In this paper, we talk about Kafka — a distributed messaging system we created to collect and deliver a large amount of log data with very low delay. Kafka uses some ideas from old log systems and messaging tools, and it works well for both offline and online data reading. While making Kafka, we took some different but useful steps to make it fast and scalable. Our tests showed that Kafka works better than two other famous messaging systems. We are already using Kafka in real-life systems, and it handles hundreds of gigabytes of new data every day.