wisemonkeys logo
FeedNotificationProfileManage Forms
FeedNotificationSearchSign in
wisemonkeys logo

Blogs

Kafka - A Framework

profile
Shaikh Sufiyan
Oct 08, 2023
0 Likes
0 Discussions
147 Reads
  • What is Kafka?

👉 Kafka is an open source software which provides a framework for storing, reading and analysing streaming data.Being open source means that it is essentially free to use and has a large network of users and developers who contribute towards updates, new features and offering support for new users.💯

Kafka is designed to be run in a “distributed” environment, which means that rather than sitting on one user’s computer, it runs across several (or many) servers, leveraging the additional processing power and storage capacity that this brings.

 

  • How does Kafka work?
  1. Apache takes information – which can be read from a huge number of data sources – and organises it into “topics”. As a very simple example, one of these data sources could be a transactional log where a grocery store records every sale.
  2. Kafka would process this stream of information and make “topics” – which could be “number of apples sold”, or “number of sales between 1pm and 2pm” which could be analysed by anyone needing insights into the data.
  3. This may sound similar to how a conventional database lets you store or sort information, but in the case of Kafka it would be suitable for a national chain of grocery stores processing thousands of apple sales every minute.
  4. This is achieved using a function known as a Producer, which is an interface between applications (e.g. the software which is monitoring the grocery stores structured but unsorted transaction database) and the topics – Kafka’s own database of ordered, segmented data, known as the Kafka Topic Log.

 

👉Advantages of Kafka

  • High performance: Kafka helps the platform to process messages at a very high speed. The processing rates can exceed beyond 100k/seconds (low latency). It maintains stable performance under extreme data loads (Terabytes of messages are stored). The data is processed in a partitioned and ordered fashion.
  • Scalability: Kafka is a distributed system that can handle large volumes of data that can scale quickly without downtime. It provides scalability by allowing partitions to be distributed across different servers. 
  • Fault Tolerance: Kafka is a distributed system consisting of several nodes running together to serve the cluster. This distribution makes it resistant to a node or machine failure within the cluster. 
  • Durability: The Kafka system is highly durable. The message in Kafka can be persisted on disk as quickly as possible.
  • Easy accessibility: Data can be easily accessible to anyone as all our data gets stored in Kafka.
  • Eliminates multiple integrations: It eliminates multiple data source integrations as all a producer’s data goes to Kafka. This reduces complexity, time and cost. 

 

👉Disadvantages of Kafka

  • Not suitable for historical data: Kafka system doesn’t allow storing historical data for more than a few hours. 
  • Slow behavior: Kafka system becomes slow when the number of queues in a cluster increases.
  • Lack of monitoring tools: Kafka system doesn’t have a complete set of monitoring and managing tools. To overcome this, we can use third-party tools like Kafka Monitor (developed by Linkedin), Datadog and Prometheus help to monitor Kafka clusters. In addition, there are many other open-source and commercial options also available. 
  • No wildcard topic support: Kafka system only supports the exact topic name and won’t support wildcard topics. So, for example, if you have a topic metric_2022_01_01 & metric_2022_01_02, then it won’t support wildcard topic selection like metric_2022_*.
  • Reduces Performance: Brokers and consumers start compressing and decompressing the messages when their size increases. This will reduce the Kafka system’s performance and affect its throughput.

Comments ()


Sign in

Read Next

Fort Myers Day Trip from Arcadia: History, Riverfront Views & Coastal Adventures

Blog banner

Improving the Accuracy of GPS and GNSS

Blog banner

Why Nature-Based Learning Is Becoming Popular in Early Education?

Blog banner

Evolution of operating systems

Blog banner

Why Moms Are Choosing Customised School Accessories Over Generic Products

Blog banner

Embaded operating system

Blog banner

How Much Protein Do You Really Need Every Day?

Blog banner

INTRODUCTION

Blog banner

Subnet Masking

Blog banner

Computer Security

Blog banner

Operating system

Blog banner

This is my first blog.

Blog banner

Cyber Forensics in Healthcare: Protecting Patient Data and Preventing Breaches

Blog banner

Threads and concurrency

Blog banner

Direct Memory Access

Blog banner

Outlook.com

Blog banner

Sweet Mango Murabba

Blog banner

Emotional Suppression: The Hidden Costs Of Unfelt Feelings

Blog banner

Photorec - media recovery tool

Blog banner

Fun Christmas Activities For Toddlers & Kids

Blog banner

Photography

Blog banner

What is Email? Uses of Emails

Blog banner

Understanding Univariate, Bivariate, and Multivariate Analysis in Data Science

Blog banner

Why Your Child Cries at Drop-Off and How Preschools Handle It Gently

Blog banner

DATA BREACH CAUSES CHALLENGES PREVENTION AND FUTURE DIRECTIONS

Blog banner

Blog on Smartsheet.

Blog banner

Modern operating systems (OS)

Blog banner

Process, process creation and process termination

Blog banner

Operating Systems

Blog banner

Loneliness

Blog banner

Multiprocessor

Blog banner

NETWORK SECURITY RISKS

Blog banner

Points to consider if you're planning to visit Florida in 2026

Blog banner

TOGETHER WE CAN CONQUER #team

Blog banner

Review on Recovering Deleted Files

Blog banner

Admissions Open: Why This Is the Right Time to Choose the Best School for Your Child

Blog banner

Personalised Name Labels vs. Handwritten Stickers: Which Is Better for Your Child’s School Essentials?

Blog banner

Mumbai

Blog banner

Virtual Machine's

Blog banner

What is service level Agreement?

Blog banner

Virtual Memory

Blog banner

The Human Side of Healing: Why Technology should Support, Not Replace, Psychotherapy

Blog banner