hive-file-formats-compressions-usecases

Usecases to select Hive File formats & Compression codecs
Usecases to select Hive File formats & Compression codecs

Pros & Cons - Hive File formats

Format Name Pros Cons
Text Format Light weight and no dependencies
Popular format for data exchange
For datasets does not have any schema or CSV dependent ingestion 		Can not handle nested data structures
No/minimal compression
Slow to read & write
RC Format Better compression than text format
Stores info as blobs
Column oriented 		Fixed columns family
No/minimal compression
Not intended for supporting transactions
ORC Format Best compression of all formats
Stores info as blob stripes
Column oriented 		Fixed columns family
Costly write operations
Not intended for supporting transactions
Avro Format Supports schema evolution
Language independent & High performance
Best for heavy write operations
Supports Transactions
Row oriented 		Does not supports Enum, Null, Timestamp with zone types
Slow Serialization/Deserialization
Schema dependent
Parquet Format Best suits for analytical opertions
Supports complex and nested data types
Column oriented 		Costly write operations
Not intended for supporting transactions
Usecases of chosing best file formats and compression codecs

Scenario Ideal File format Ideal Compression
Small data with simple structure
No external dependencies
No nested & complex structure
Text No compression Required
More compression required
With high analytical operations
System reads columns rather than rows
 ORC Snappy(High Disk Usage/Less CPU)
GZIP(Low Disk Usage/High CPU)
Heavy write operations
Should support Schema evolution
System reads entire rows rather than columns
 Avro Snappy(High Disk Usage/Less CPU)
GZIP(Low Disk Usage/High CPU)
Heavy Anaytical Operations
System process set of columns instead of entire rows
Parquet Snappy(High Disk Usage/Less CPU)
GZIP(Low Disk Usage/High CPU)
Compression rate

File Format: TEXT offers 0% compression
Compression

File Format:  RC offers 15% compression
Compression

File Format:  ORC offers 60% compression
Compression

File Format:  AVRO offers 30% compression
Compression

File Format:  PARQUET offers 25% compression
Compression

File Format: sdsd offers 77% compression
Compression

Comments

Post a Comment

Popular posts from this blog

hadoop-installation-ubuntu

Hadoop Setup Hadoop Setup Hadoop installation & configuration on Ubuntu Prerequisites: 1. Java open-jdk 8 2. SSH Server - open ssh server & client 3. Hadoop dist - 2.9.1 1. Installing JDK8 on Ubuntu $ sudo apt-get update $ sudo apt-get install openjdk-8-jdk $ whereis java or whereis jvm //To get the path of Java General installation directory - /usr/lib/jvm/java-8-openjdk-amd64 Adding Java JDK8 path to environment variables $ sudo gedit /etc/profile or $ gedit .bashrc export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin Verify the Java installation $ java -version $ javac -version 2. Extract hadoop-2.9.1.tar.gz Search and Download Apache Hadoop 2.9.1 from Apache Hadoop site And extract the file to a directory /home/mm/softwares/ 3. Configuring Hadoop directories on path $ sudo gedit /etc/profile or $ sudo gedit .bashrc export HADOOP_HOME=/h...
Read more

jenv-tool

Jenv Setup Jenv Setup Jenv tool is used to install, manage and switch between different Java versions easily. Automates the manual editing of changing paths and system properties. Homepage: Jenv Setup Jenv 1      1. Jenv project homepage 2      Homepage: https://github.com/jenv/jenv 3      4      2. Installation via scoop: easy way to install on Windows. For Other Operating systems Check here 5      cmd> scoop bucket add extras 6      cmd> scoop install extras/jenv 7      8      Prerequisite: Scoop should be installed and configured. See Config Scoop for more info. 9      10      3. Check the scoop ...
Read more

hive-installation-in-ubuntu

Hive Setup Hive Setup Download Hive Download and extract hive to a directory. Add below values to the .bashrc file export HIVE_HOME=/home/mkm/mm/softwares/hive-3.1.3 export PATH=$PATH:$HIVE_HOME/bin How to Setup Hive on Ubuntu in 3 different ways. Embedded Metastore(Derby) Pre-requisite -  Install Derby DB Local Metastore(Same Machine/JVM) Pre-requisite -  Install MySQL Server Remote Metastore(Remote Machine/JVM) Embedded Metastore(Derby) Will Update soon Local Metastore(Same Machine/JVM) 1. Check MySQL Server Instance is running or not $ ps -e | grep mysql or sudo service mysql status If not, Install MySQL Server by following page in Pre-requisite section 2. Login to the MySQL server $ mysql -u root -p Create Hive Metastore Database mysql> CREATE DATABASE metastore_db; Create new user login for Hive mysql> CREATE USER 'hiveadmin'@'localhost' IDENTIFIED BY ...
Read more