hive-architecture

Hive Architecture
Hive Architecture

Hive is a Data Warehousing tool based on Hadoop Framework

Highly scalable as it uses HDFS for storage and Map Reduce for processing

Useful for SQL scripting rather than programming approach

Provides great support for Analysts to run Adhoc queries, Data analysis & summarization

Hive Trift Service - Hive server offers Trift service to connect Hive from different applications like JDBC/ODBC & other Trift clients(Eg. Beeline) Hive Architecture

Hive clients - CLI, Trift client, JDBC/ODBC application, Web GUI(Ambari)

CLI(Command Line Interface) aka Shell interface is the default service to invoke Hive

Provides all supported operations through commands

Trift Client - Trift clients like Beeline comes default with Hive, interacts with Hive like from CLIj
Stages in execution

  1. User submits query from anyone of the Hive client
  2. Driver receives the query, creates session and invokes Compiler, Driver have Session management, Hive configuration file & JDBC APIs capabilities
  3. Compiler parses the query, does the semantic analysis and generates execution plan with the help of Metastore
  4. Metastore contains all the metadata info for every Table, Partitions, Column types & Serializers/Deserializers for read/Write data from HDFS
  5. Execution engine gets series of execution steps as Execution plan from Compiler. Plan is DAG of stages contains Map/Reduce jobs
  6. Engine submits appropriate Map or Reducer jobs along with Metastore data if serilizer/deserializer requires in that job
  7. Intermediate job results will be save to temp. file and forwards to next stage

Comments

Post a Comment

Popular posts from this blog

hadoop-installation-ubuntu

Hadoop Setup Hadoop Setup Hadoop installation & configuration on Ubuntu Prerequisites: 1. Java open-jdk 8 2. SSH Server - open ssh server & client 3. Hadoop dist - 2.9.1 1. Installing JDK8 on Ubuntu $ sudo apt-get update $ sudo apt-get install openjdk-8-jdk $ whereis java or whereis jvm //To get the path of Java General installation directory - /usr/lib/jvm/java-8-openjdk-amd64 Adding Java JDK8 path to environment variables $ sudo gedit /etc/profile or $ gedit .bashrc export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin Verify the Java installation $ java -version $ javac -version 2. Extract hadoop-2.9.1.tar.gz Search and Download Apache Hadoop 2.9.1 from Apache Hadoop site And extract the file to a directory /home/mm/softwares/ 3. Configuring Hadoop directories on path $ sudo gedit /etc/profile or $ sudo gedit .bashrc export HADOOP_HOME=/h...
Read more

jenv-tool

Jenv Setup Jenv Setup Jenv tool is used to install, manage and switch between different Java versions easily. Automates the manual editing of changing paths and system properties. Homepage: Jenv Setup Jenv 1      1. Jenv project homepage 2      Homepage: https://github.com/jenv/jenv 3      4      2. Installation via scoop: easy way to install on Windows. For Other Operating systems Check here 5      cmd> scoop bucket add extras 6      cmd> scoop install extras/jenv 7      8      Prerequisite: Scoop should be installed and configured. See Config Scoop for more info. 9      10      3. Check the scoop ...
Read more

hive-installation-in-ubuntu

Hive Setup Hive Setup Download Hive Download and extract hive to a directory. Add below values to the .bashrc file export HIVE_HOME=/home/mkm/mm/softwares/hive-3.1.3 export PATH=$PATH:$HIVE_HOME/bin How to Setup Hive on Ubuntu in 3 different ways. Embedded Metastore(Derby) Pre-requisite -  Install Derby DB Local Metastore(Same Machine/JVM) Pre-requisite -  Install MySQL Server Remote Metastore(Remote Machine/JVM) Embedded Metastore(Derby) Will Update soon Local Metastore(Same Machine/JVM) 1. Check MySQL Server Instance is running or not $ ps -e | grep mysql or sudo service mysql status If not, Install MySQL Server by following page in Pre-requisite section 2. Login to the MySQL server $ mysql -u root -p Create Hive Metastore Database mysql> CREATE DATABASE metastore_db; Create new user login for Hive mysql> CREATE USER 'hiveadmin'@'localhost' IDENTIFIED BY ...
Read more