APACHE KAFKA

WHAT IS KAFKA?

Apache Kafka is an open-source platform for stream processing, designed for real-time data storage. It serves as an intermediary between senders and receivers, managing trillions of data events daily. Kafka is used for the production analytics pipeline & real time monitoring infrastructure. It is a distributed streaming platform that operates on the publish-subscribe messaging model, facilitating data exchange.

Apache Kafka was originally developed by LinkedIn, and later it was donated to the Apache Software Foundation.

CORE API s : Producer API , Consumer API , Streams API , Connector API

Key Features of Kafka

1. Scalability: Apache Kafka exhibits scalability across all four dimensions: event producers, event processors, event consumers, and event connectors. In essence, Kafka can effortlessly scale without any downtime.
2. High Volume: Kafka excels at processing vast volumes of data streams with ease.
3. Data Transformations: Kafka provides the capability to generate new data streams by leveraging data streams from producers.
4. Fault Tolerance: The Kafka cluster is resilient and capable of handling failures, including those involving masters and databases.
5. Reliability: Given Kafka's distributed, partitioned, replicated, and fault-tolerant nature, it offers exceptional reliability.

Real-Time Applications of Kafka

1)Twitter: Registered users can read and post tweets, while unregistered users can only read them. Twitter incorporates Storm-Kafka into its stream processing infrastructure.

2)LinkedIn: Apache Kafka plays a crucial role at LinkedIn, handling activity stream data and operational metrics. The Kafka messaging system supports various products, including LinkedIn Newsfeed and LinkedIn Today for online message consumption, as well as offline analytics systems like Hadoop.

3)Netflix: Netflix, a multinational provider of on-demand Internet streaming media, relies on Kafka for real-time monitoring and event processing.

4)Box: At Box, Kafka powers the production analytics pipeline and real-time monitoring infrastructure.