hdfs-basic-commands

HDFS Basic commands
Basic operations on HDFS
Examples shell To format Namenode $ hdfs namenode -format To create a new folder in Hdfs $ hdfs dfs -mkdir /tmp To put file in HDFS $ hdfs dfs -put /emp.csv /tmp To print file contents $ hdfs dfs -cat /tmp/emp.csv To get file statistics - lines count $ hdfs dfs -cat /tmp1/emp.csv | wc -l To get file words count $ hdfs dfs -cat /tmp1/emp.csv | wc -w To get file characters count $ hdfs dfs -cat /tmp1/emp.csv | wc -c To move file $ hdfs dfs -mv /tmp1/emp.csv /tmp6 To delete file and directories from hdfs hdfs dfs -rm -r /tmp4 hdfs dfs -rm -r /tmp4/emp.csv To configure Trash interval in minutes in core-site.xml Default fs.trash.interval is zero <property> <name>fs.trash.interval</name> <value>30</value> </property> now delete the file $ hdfs dfs -rm /tmp4/emp4.csv deleted file will be in below directory $ hdfs dfs -ls /user/mm/.Trash/Current/mp4/ we can move from trash to user directory with mv $ hdfs dfs -mv /user/mm/.Trash/Current/tmp4/emp.csv /tmp4/ $ hdfs dfs -createSnapshot /tmp1 createSnapshot: Directory is not a snapshottable directory: /tmp1 $ hdfs dfsadmin -allowSnapshot /tmp1 To list all snapshot enabled directories $ hdfs lsSnapshottableDir To make directory snapshottable $ hdfs dfsadmin -allowSnapshot /tmp1 Allowing snaphot on /tmp1 succeeded To create snapshot $ hdfs dfs -createSnapshot /tmp1 To view snapshots of a directory $ hdfs dfs -ls /tmp1/.snapshot/ To view data of file snapshot $ hdfs dfs -tail /tmp1/.snapshot/s20220211-165939.252/userinfo1.csv To disable snapshot $ hdfs dfsadmin -disallowSnapshot /tmp1 To delete snapshot $ hdfs dfs -deleteSnapshot /tmp1 To get snapshot differences, . denotes current directory $ hdfs snapshotDiff <path> <fromSnapshot> . $ hdfs snapshotDiff <path> <fromSnapshot> <toSnapshot> Demo to test Snapshot differences $ hdfs dfs -put /home/mmk-prac/mm/data/emp.csv /emp $ hdfs dfsadmin -allowSnapshot /emp $ hdfs dfs -createSnapshot /emp $ hdfs dfs -ls /emp/.snapshot Found 1 items drwxr-xr-x - mmk-prac supergroup 0 2023-01-06 18:36 /emp/.snapshot/s20230106-183632.204 $ hdfs dfs -rm /emp/emp.csv $ hdfs dfs -put /home/mmk-prac/mm/data/emp2.csv /emp $ hdfs snapshotDiff /emp s20230106-183632.204 . Difference between snapshot s20230106-183632.204 and current directory under directory /emp: M . + ./emp2.csv - ./emp.csv