kafka-installation-ubuntu

Kafka Setup
Kafka Setup
 



Kafka setup

1. Download and extract kafka_2.12-3.8.0 to a directory

2. Configure .bahsrc
export KAFKA_HOME=/home/mkm/softwares/kafka_2.12-3.8.0
export PATH=$PATH:$KAFKA_HOME/bin
export CLASSPATH=$CLASSPATH:$KAFKA_HOME/libs/*:.

3. Configuring important properties in kafka.

KAFKA_HOME/config/zookeeper.properties
dataDir=/home/mkm/softwares/kafka_2.12-3.8.0/data_snapshots
clientPort=2181

KAFKA_HOME/config/server.properties
num.partitions=1
listeners=PLAINTEXT://:9092
zookeeper.connect=localhost:2181
logs.dir=/tmp/kafka-server1-logs

To start another server(broker), create server2.properties and change listener port
KAFKA_HOME/config/server2.properties
num.partitions=1
listeners=PLAINTEXT://:9093
zookeeper.connect=localhost:2181
logs.dir=/tmp/kafka-server2-logs

4. Start zookeeper server
$ zookeeper-server-start.sh /home/mkm/softwares/kafka_2.12-3.8.0/config/zookeeper.properties

5. Start kafka server
kafka-server-start.sh /home/mkm/softwares/kafka_2.12-3.8.0/config/server.properties
kafka-server-start.sh /home/mkm/softwares/kafka_2.12-3.8.0/config/server2.properties

6. Create a kafka topic
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic mysampletopic

7. Start kafka topic console producer
$ kafka-console-producer.sh --bootstrap-server localhost:9092 --topic mysampletopic

8. Start kafka topic console consumer
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mysampletopic

9. Read from kafka topic from beginning.
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mysampletopic --from-beginning
Compiled on THURSDAY, 29-AUGUST-2024, 12:53:18 PM IST

Comments

Post a Comment

Popular posts from this blog

hadoop-installation-ubuntu

Hadoop Setup Hadoop Setup Hadoop installation & configuration on Ubuntu Prerequisites: 1. Java open-jdk 8 2. SSH Server - open ssh server & client 3. Hadoop dist - 2.9.1 1. Installing JDK8 on Ubuntu $ sudo apt-get update $ sudo apt-get install openjdk-8-jdk $ whereis java or whereis jvm //To get the path of Java General installation directory - /usr/lib/jvm/java-8-openjdk-amd64 Adding Java JDK8 path to environment variables $ sudo gedit /etc/profile or $ gedit .bashrc export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin Verify the Java installation $ java -version $ javac -version 2. Extract hadoop-2.9.1.tar.gz Search and Download Apache Hadoop 2.9.1 from Apache Hadoop site And extract the file to a directory /home/mm/softwares/ 3. Configuring Hadoop directories on path $ sudo gedit /etc/profile or $ sudo gedit .bashrc export HADOOP_HOME=/h...
Read more

jenv-tool

Jenv Setup Jenv Setup Jenv tool is used to install, manage and switch between different Java versions easily. Automates the manual editing of changing paths and system properties. Homepage: Jenv Setup Jenv 1      1. Jenv project homepage 2      Homepage: https://github.com/jenv/jenv 3      4      2. Installation via scoop: easy way to install on Windows. For Other Operating systems Check here 5      cmd> scoop bucket add extras 6      cmd> scoop install extras/jenv 7      8      Prerequisite: Scoop should be installed and configured. See Config Scoop for more info. 9      10      3. Check the scoop ...
Read more

hive-installation-in-ubuntu

Hive Setup Hive Setup Download Hive Download and extract hive to a directory. Add below values to the .bashrc file export HIVE_HOME=/home/mkm/mm/softwares/hive-3.1.3 export PATH=$PATH:$HIVE_HOME/bin How to Setup Hive on Ubuntu in 3 different ways. Embedded Metastore(Derby) Pre-requisite -  Install Derby DB Local Metastore(Same Machine/JVM) Pre-requisite -  Install MySQL Server Remote Metastore(Remote Machine/JVM) Embedded Metastore(Derby) Will Update soon Local Metastore(Same Machine/JVM) 1. Check MySQL Server Instance is running or not $ ps -e | grep mysql or sudo service mysql status If not, Install MySQL Server by following page in Pre-requisite section 2. Login to the MySQL server $ mysql -u root -p Create Hive Metastore Database mysql> CREATE DATABASE metastore_db; Create new user login for Hive mysql> CREATE USER 'hiveadmin'@'localhost' IDENTIFIED BY ...
Read more