kafka-basics

Kafka Basics
Kafka Basics

Contents
  1. About Kafka
  2. Architecture


     Kafka is a pub-sub(publisher-subscriber) messaging service.
    
     Topics are the key resources that stores messages in order and replicates them in a distributed way among different servers against data loss.
     This makes topics fault tolerant and scalable and Kafka as a reliable data provider to many consumers.
    
     Messages can be any type of information which represents state,log or event of a source system.
    
     Kafka cluster comprises of brokers(servers) and clients.
    
     Brokers are the services that run on kafka servers to store and manage messages in a distributed way and to provide the consumers with the requested topic data.
    
     Brokers responsbilities:
     1. Receive messages
     2. Store messages and replicate
     3. Serve requests from consumers
     4. Load balancing
    
     Kafka is a master less architecture. Any broker in the group can be act as leader. This leader selection is done by quorum.
     And the remaining brokers will act as followers.
    
     Clients are Producers(pubs) and Consumers(subs), which are applications/programs to send and receive messages via Kafka.
    
     Producers writes messages to the Kafka topics and can wait for acknowledgement.
    
     Consumers needs to subscribe to the required tooics and periodically checks for the new messages.
    
     Zookeeper is the centralized service to keep the configurations and important parameters in-sync among the distributed systems.
    
     ISR(In-Sync-Replicas) are stored in zookeeper

Kafka Architecture

High Level Architecture
Compiled on SATURDAY, 26-OCTOBER-2024, 11:31:55 PM IST

Comments

Post a Comment

Popular posts from this blog

hadoop-installation-ubuntu

Hadoop Setup Hadoop Setup Hadoop installation & configuration on Ubuntu Prerequisites: 1. Java open-jdk 8 2. SSH Server - open ssh server & client 3. Hadoop dist - 2.9.1 1. Installing JDK8 on Ubuntu $ sudo apt-get update $ sudo apt-get install openjdk-8-jdk $ whereis java or whereis jvm //To get the path of Java General installation directory - /usr/lib/jvm/java-8-openjdk-amd64 Adding Java JDK8 path to environment variables $ sudo gedit /etc/profile or $ gedit .bashrc export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin Verify the Java installation $ java -version $ javac -version 2. Extract hadoop-2.9.1.tar.gz Search and Download Apache Hadoop 2.9.1 from Apache Hadoop site And extract the file to a directory /home/mm/softwares/ 3. Configuring Hadoop directories on path $ sudo gedit /etc/profile or $ sudo gedit .bashrc export HADOOP_HOME=/h...
Read more

jenv-tool

Jenv Setup Jenv Setup Jenv tool is used to install, manage and switch between different Java versions easily. Automates the manual editing of changing paths and system properties. Homepage: Jenv Setup Jenv 1      1. Jenv project homepage 2      Homepage: https://github.com/jenv/jenv 3      4      2. Installation via scoop: easy way to install on Windows. For Other Operating systems Check here 5      cmd> scoop bucket add extras 6      cmd> scoop install extras/jenv 7      8      Prerequisite: Scoop should be installed and configured. See Config Scoop for more info. 9      10      3. Check the scoop ...
Read more

hive-installation-in-ubuntu

Hive Setup Hive Setup Download Hive Download and extract hive to a directory. Add below values to the .bashrc file export HIVE_HOME=/home/mkm/mm/softwares/hive-3.1.3 export PATH=$PATH:$HIVE_HOME/bin How to Setup Hive on Ubuntu in 3 different ways. Embedded Metastore(Derby) Pre-requisite -  Install Derby DB Local Metastore(Same Machine/JVM) Pre-requisite -  Install MySQL Server Remote Metastore(Remote Machine/JVM) Embedded Metastore(Derby) Will Update soon Local Metastore(Same Machine/JVM) 1. Check MySQL Server Instance is running or not $ ps -e | grep mysql or sudo service mysql status If not, Install MySQL Server by following page in Pre-requisite section 2. Login to the MySQL server $ mysql -u root -p Create Hive Metastore Database mysql> CREATE DATABASE metastore_db; Create new user login for Hive mysql> CREATE USER 'hiveadmin'@'localhost' IDENTIFIED BY ...
Read more