Apache Kafka in a nutshell
A gentle introduction to Kafka
What is Apache Kafka?
Apache Kafka was initially designed and implemented by LinkedIn in order to serve as a message queue. Since 2012, Kafka has been open sourced under Apache Foundation and quickly evolved into a distributed streaming platform, which is used for the implementation of real-time data pipelines and streaming applications.
It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.
Kafka is a fault-tolerant and horizontally scalable platform, that can scale to hundreds of brokers being able to process millions of records per second. It was designed to simplify data pipelines, handle data streams and support batch and real-time analytics. It also supports at most once, at least once and exactly once semantic processing guarantees.
Why organisations use Kafka?
Modern organisations have various data pipelines that facilitate the communication between systems or services. Things get a bit more complicated when a reasonable number of services need to communicate with each other at real time.
