Hadoop - ... or should we say "The Apache Hadoop software library"

mail

get / copyToLocal

Usage

copy one or more files from the HDFS storage to the local filesystem :

hdfs dfs -copyToLocal /path/on/HDFS/storage/file /path/to/local/directory
hdfs dfs -copyToLocal /path/on/HDFS/storage/file1 /path/on/HDFS/storage/file2 /path/to/local/directory

Flags

Flag	Usage
-f	overwrite the destination if it already exists

mail

put / copyFromLocal

Usage

copy one or more local files to the HDFS storage :

hdfs dfs -copyFromLocal localFile /path/on/HDFS/storage
hdfs dfs -copyFromLocal localFile1 localFile2 /path/on/HDFS/storage

Flags

Flag	Usage
-f	overwrite the destination if it already exists

mail

mkdir

Usage

create directories :

hdfs dfs -mkdir newDirectory
hdfs dfs -mkdir -p /path/to/newDirectory
hdfs dfs -mkdir newDirectory1 newDirectory2

Flags

Flag	Usage
-p	create parent directories along the path, just like Bash's mkdir -p does

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#mkdir

mail

distcp

Usage

distcp (distributed copy)

is a tool used for large inter/intra-cluster copying
can be invoked as :
- hadoop distcp [options] source destination
- hdfs dfs -distcp [options] source destination
  (source) This is not clear whether this syntax is right or not as I can't find many sources about it

mail

Hadoop glossary

location :
- default : hdfs:///user/userName/.Trash/Current
- within encryption zones : hdfs:///encryptionZone/.Trash/userName/Current/
deleted data
- stays in the trash for an amount of time configured by fs.trash.interval
- can be restored simply by being moved out of the .Trash/ directory (source)

mail

count

Usage

count the number of directories, files and bytes under the paths that match the specified file pattern
get the quota and the usage

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#count

mail

du

Usage

Display

sizes of files and directories contained in the given directory
or the length of a file in case its just a file

Flags

Flag	Usage
-h	use human-readable units : K, M, G,
-s	sum the files sizes rather than displaying each file size individually

Example

hdfs dfs -du -s -h '/path/to/myData/dataSource/dataSet/year=2022/month=10'
```
119.5 K	/path/to/myData/dataSource/dataSet/year=2022/month=10
```

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#du

mail

find

Usage

find all files matching the specified expression and apply actions to them
if no path is specified, default to the current working directory
if no action is specified, default to -print

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#find

mail

ls

Usage

list files. Default output format :

for a file :
1. permissions
2. number of replicas
3. userId
4. groupId
5. size
6. modification date
7. modification time
8. name
for a directory :
1. permissions
2. userId
3. groupId
4. modification date
5. modification time
6. name

Flags

Flag	Usage
-C	display the paths of files but not the other fields (i.e. don't display permissions, owner, group, dates, ) like ls -1 does)
-R	Recursively list subdirectories encountered

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#ls

mail

rm

Usage

delete files

Flags

Flag	Usage
-r -R	recursively delete a directory and its content The rmr command has been deprecated by rm -r
-safely	before deleting directory, prompt for confirmation if total number of files > hadoop.shell.delete.limit.num.files (set in core-site.xml, default: 100) it can be used with -skipTrash to prevent accidental deletion of large directories expect a delay when walking over large directory recursively to count the number of files to be deleted before the confirmation
-skipTrash	delete the specified files immediately instead of moving them to the trash This is a dangerous option which use is discouraged.

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#rm