hive-metastore

Hive Metastore
Hive Metastore

Metastore is a central repository of Hive Metadata

Metastore comprises two parts

Metadata service - Communicates with Hive driver and returns all Metadata it needed

Metadata Database - stores all metadata information inform of tables
What Metastore contains

  1. All Databases Metadata - Ids, Creation date, Owners
  2. All Tables Information - Custom properties, Schema Info, Indexes, Partitions with their Owners & creation date
  3. Rows input & output formats and Serializers/ Deserializers info
  4. All Users Information - Asigned Roles
Stages in execution
 


Metastore can be configured in either Embedded, Local or Remote location
Embedded Metastore - Service & Database(Derby by default) runs on same JVM where Hive runs and supports single session at a time.
Local Metastore service runs on same JVM like Hive but connects to external Database. Supports multiple sessions at a time.
In Remote Metastore mode, Metastore services runs on different processes, hosts and supports multiple sessions with added security
Metastore Configurations 

hive.metastore.warehouse.dir - /user/hive/warehouse (default) or external
hive.metastore.local - true(default) to use embedded metastore, if false configure hive.metastore.uris
hive.metastore.uris - CSV of Remote Metastore URLs
javax.jdo.option.ConnectionURL - Metastore connection URL Eg. jdbc:derby:;databaseName=metastore_db;create=true(default) or jdbc:mysql://localhost:3306/hive_mysql_db?createDatabaseIfNotExist=true
javax.jdo.option.ConnectionDriverName - Eg. org.apache.derby.jdbc.EmbeddedDriver(default) or com.mysql.jdbc.Driver
javax.jdo.option.ConnectionUserName - DB username
javax.jdo.option.ConnectionPassword - DB password

Comments

Post a Comment

Popular posts from this blog

hadoop-installation-ubuntu

Hadoop Setup Hadoop Setup Hadoop installation & configuration on Ubuntu Prerequisites: 1. Java open-jdk 8 2. SSH Server - open ssh server & client 3. Hadoop dist - 2.9.1 1. Installing JDK8 on Ubuntu $ sudo apt-get update $ sudo apt-get install openjdk-8-jdk $ whereis java or whereis jvm //To get the path of Java General installation directory - /usr/lib/jvm/java-8-openjdk-amd64 Adding Java JDK8 path to environment variables $ sudo gedit /etc/profile or $ gedit .bashrc export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin Verify the Java installation $ java -version $ javac -version 2. Extract hadoop-2.9.1.tar.gz Search and Download Apache Hadoop 2.9.1 from Apache Hadoop site And extract the file to a directory /home/mm/softwares/ 3. Configuring Hadoop directories on path $ sudo gedit /etc/profile or $ sudo gedit .bashrc export HADOOP_HOME=/h...
Read more

jenv-tool

Jenv Setup Jenv Setup Jenv tool is used to install, manage and switch between different Java versions easily. Automates the manual editing of changing paths and system properties. Homepage: Jenv Setup Jenv 1      1. Jenv project homepage 2      Homepage: https://github.com/jenv/jenv 3      4      2. Installation via scoop: easy way to install on Windows. For Other Operating systems Check here 5      cmd> scoop bucket add extras 6      cmd> scoop install extras/jenv 7      8      Prerequisite: Scoop should be installed and configured. See Config Scoop for more info. 9      10      3. Check the scoop ...
Read more

hive-installation-in-ubuntu

Hive Setup Hive Setup Download Hive Download and extract hive to a directory. Add below values to the .bashrc file export HIVE_HOME=/home/mkm/mm/softwares/hive-3.1.3 export PATH=$PATH:$HIVE_HOME/bin How to Setup Hive on Ubuntu in 3 different ways. Embedded Metastore(Derby) Pre-requisite -  Install Derby DB Local Metastore(Same Machine/JVM) Pre-requisite -  Install MySQL Server Remote Metastore(Remote Machine/JVM) Embedded Metastore(Derby) Will Update soon Local Metastore(Same Machine/JVM) 1. Check MySQL Server Instance is running or not $ ps -e | grep mysql or sudo service mysql status If not, Install MySQL Server by following page in Pre-requisite section 2. Login to the MySQL server $ mysql -u root -p Create Hive Metastore Database mysql> CREATE DATABASE metastore_db; Create new user login for Hive mysql> CREATE USER 'hiveadmin'@'localhost' IDENTIFIED BY ...
Read more