hive-file-formats

Hive File formats
Hive File formats

File formats/SerDe(Serialization/Deserialization) are used to sepcify the underlying storage type of a Table.
Importanat Factors to consider right file format
1. Fault tolerant
2. Consistency
3. Widely accepted format for sharing across applications
4. Support for performing quick analytical operations

(Row Object) ==> Serialization ==> (Output File Format) ==> Deserialization ==> (Row Object) 

Text - STORED AS TEXTFILE
RC(Row Columnar) - STORED AS RCFILE
ORC(Optimized Row Columnar) - STORED AS ORC/ORCFILE
Avro - STORED AS AVRO/AVROFILE
Parquet - STORED AS PARQUET/PARQUETFILE
Sequence - STORED AS SEQUENCEFILE




Text File format is the default for the Hive tables.
De facto format for CSV, TSV(Tab separated) and other special character separated values.
Input format  - org.apache.hadoop.mapred.TextInputFormat
Output format - org.apache.hadoop.mapred.TextOutputFormat




RC File format stores the data in binary key/value pairs.
The Source file is partitioned in to row splits(horizontally)
Each row split again partitioned in to column wise(vertically)
Metadata of each row split is Key and Value is row split data
Input format  - org.apache.hadoop.hive.ql.io.RCFileInputFormat
Output format - org.apache.hadoop.hive.ql.io.RCFileOutputFormat




The best file format in terms of file size compression.
Increases performance while reading, writing and processing data in Hive.
The input file is divided in to multiple row groups called as stripes.
Each stripe is of 250 MB of data having three parts.
The first part holds index, second one holds row data and last called footer stripe
The footer contains column level aggregations like count, max,min and sum of that stripe
orc.compress="ZLIB"(default) or "NONE"(no compression) or "SNAPPY"
Serde - org.apache.hadoop.hive.ql.io.orc.OrcSerde
Input format - org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
Output format - org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat




It is row based binary file format and language independent file sharing format
Best used format for streaming data pipelines created in Kafka, etc.
It supports schema evolution as it stores metadata in Json format, flexible for modifications
Serde - org.apache.hadoop.hive.serde2.avro.AvroSerDe
Input format  - org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat
Output format - org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat




Columnar based format and best chosen for data analytics code
Supports advanced data types/nested data structures
Data orginized in column groups rather than row groups, hence loading data of specific columns will be faster
Serde - org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
Input format  - org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
Output format - org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat




This format is developed to overcome small files problem in Big Data world
It combines many small files generated by reducers to one big file
Sequence format also stores data in binary key/value pairs
Input format  - org.apache.hadoop.mapred.SequenceFileInputFormat
Output format - org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat

Comments

Post a Comment

Popular posts from this blog

hadoop-installation-ubuntu

Hadoop Setup Hadoop Setup Hadoop installation & configuration on Ubuntu Prerequisites: 1. Java open-jdk 8 2. SSH Server - open ssh server & client 3. Hadoop dist - 2.9.1 1. Installing JDK8 on Ubuntu $ sudo apt-get update $ sudo apt-get install openjdk-8-jdk $ whereis java or whereis jvm //To get the path of Java General installation directory - /usr/lib/jvm/java-8-openjdk-amd64 Adding Java JDK8 path to environment variables $ sudo gedit /etc/profile or $ gedit .bashrc export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin Verify the Java installation $ java -version $ javac -version 2. Extract hadoop-2.9.1.tar.gz Search and Download Apache Hadoop 2.9.1 from Apache Hadoop site And extract the file to a directory /home/mm/softwares/ 3. Configuring Hadoop directories on path $ sudo gedit /etc/profile or $ sudo gedit .bashrc export HADOOP_HOME=/h...
Read more

jenv-tool

Jenv Setup Jenv Setup Jenv tool is used to install, manage and switch between different Java versions easily. Automates the manual editing of changing paths and system properties. Homepage: Jenv Setup Jenv 1      1. Jenv project homepage 2      Homepage: https://github.com/jenv/jenv 3      4      2. Installation via scoop: easy way to install on Windows. For Other Operating systems Check here 5      cmd> scoop bucket add extras 6      cmd> scoop install extras/jenv 7      8      Prerequisite: Scoop should be installed and configured. See Config Scoop for more info. 9      10      3. Check the scoop ...
Read more

hive-installation-in-ubuntu

Hive Setup Hive Setup Download Hive Download and extract hive to a directory. Add below values to the .bashrc file export HIVE_HOME=/home/mkm/mm/softwares/hive-3.1.3 export PATH=$PATH:$HIVE_HOME/bin How to Setup Hive on Ubuntu in 3 different ways. Embedded Metastore(Derby) Pre-requisite -  Install Derby DB Local Metastore(Same Machine/JVM) Pre-requisite -  Install MySQL Server Remote Metastore(Remote Machine/JVM) Embedded Metastore(Derby) Will Update soon Local Metastore(Same Machine/JVM) 1. Check MySQL Server Instance is running or not $ ps -e | grep mysql or sudo service mysql status If not, Install MySQL Server by following page in Pre-requisite section 2. Login to the MySQL server $ mysql -u root -p Create Hive Metastore Database mysql> CREATE DATABASE metastore_db; Create new user login for Hive mysql> CREATE USER 'hiveadmin'@'localhost' IDENTIFIED BY ...
Read more