Most commonly used HDFS Commands
1. List all the files and directories under root hdfs directory
hdfs dfs -ls /
To list all the files and directories recursively, use lsr command as below.
hdfs dfs -lsr /
2. Copy a file or directory to another directory in hdfs
hdfs dfs -cp /hdfs/src/dir /hdfs/dest/dir/
hdfs dfs -cp /hdfs/src/dir/file1 /hdfs/dest/dir/
3. Move or rename a file or directory to another directory in hdfs
hdfs dfs -mv /hdfs/src/dir /hdfs/dest/dir
hdfs dfs -mv /hdfs/current/dir/file1 /hdfs/current/dir/file2
4. Create a directory in hdfs
hdfs dfs -mkdir /hdfs/new/dir/path
If parent directory not present, -p option can be user to create all the directories at one go.
hdfs dfs -mkdir -p /hdfs/new/dir/path
5. Read a file in hdfs
hdfs dfs -cat /hdfs/dir/file1.dat
If the file is snappy compressed, use text command instead to read the same.
hdfs dfs -text /hdfs/dir/file1.dat.snappy
6. Copy a file from local file system to hdfs file system
hdfs dfs -copyFromLocal /local/dir/path/file1.txt /hdfs/dir/path/file.txt
put command also does the same.
hdfs dfs -put /local/dir/path/file1.txt /hdfs/dir/path/file.txt
7. Copy a file from hdfs file system to local file system
hdfs dfs -copyToLocal /hdfs/dir/path/file.txt /local/dir/path/file1.txt
get command also does the same.
hdfs dfs -get /hdfs/dir/path/file.txt /local/dir/path/file1.txt
hdfs dfs -ls /
To list all the files and directories recursively, use lsr command as below.
hdfs dfs -lsr /
2. Copy a file or directory to another directory in hdfs
hdfs dfs -cp /hdfs/src/dir /hdfs/dest/dir/
hdfs dfs -cp /hdfs/src/dir/file1 /hdfs/dest/dir/
3. Move or rename a file or directory to another directory in hdfs
hdfs dfs -mv /hdfs/src/dir /hdfs/dest/dir
hdfs dfs -mv /hdfs/current/dir/file1 /hdfs/current/dir/file2
4. Create a directory in hdfs
hdfs dfs -mkdir /hdfs/new/dir/path
If parent directory not present, -p option can be user to create all the directories at one go.
hdfs dfs -mkdir -p /hdfs/new/dir/path
5. Read a file in hdfs
hdfs dfs -cat /hdfs/dir/file1.dat
If the file is snappy compressed, use text command instead to read the same.
hdfs dfs -text /hdfs/dir/file1.dat.snappy
6. Copy a file from local file system to hdfs file system
hdfs dfs -copyFromLocal /local/dir/path/file1.txt /hdfs/dir/path/file.txt
put command also does the same.
hdfs dfs -put /local/dir/path/file1.txt /hdfs/dir/path/file.txt
hdfs dfs -copyToLocal /hdfs/dir/path/file.txt /local/dir/path/file1.txt
get command also does the same.
hdfs dfs -get /hdfs/dir/path/file.txt /local/dir/path/file1.txt
8. Delete a file or directory from hdfs
Use rm command to delete a file
hdfs dfs -rm /hdfs/dir/path/file.txt
Use rmr command to delete a directory and it's contents
hdfs dfs -rmr /hdfs/dir/path/
9. Create a zero byte file in hdfs
hdfs dfs -touchz /hdfs/dir/path/file.txt
10. Verify a directory or file using test command in hdfs
hadoop fs -test -[defsz] URI
Options:
-d: f the path is a directory, return 0.
-e: if the path exists, return 0.
-f: if the path is a file, return 0.
-s: if the path is not empty, return 0.
-z: if the file is zero length, return 0.
Example:
hadoop fs -test -e /hdfs/dir/path
0 comments:
Post a Comment