Hadoop - ... or should we say "The Apache Hadoop software library"

mail

copyFromLocal

Usage

copy one or more local files to the destination filesystem :

hdfs dfs -copyFromLocal localFile /path/on/destination/filesystem
hdfs dfs -copyFromLocal localFile1 localFile2 /path/on/destination/filesystem

mail

mkdir

Usage

create directories :

hdfs dfs -mkdir newDirectory
hdfs dfs -mkdir -p /path/to/newDirectory
hdfs dfs -mkdir newDirectory1 newDirectory2

Flags

Flag	Usage
-p	create parent directories along the path, just like Bash's mkdir -p does

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#mkdir

mail

distcp

Usage

distcp (distributed copy)

is a tool used for large inter/intra-cluster copying
can be invoked as :
- hadoop distcp [options] source destination
- hdfs dfs -distcp [options] source destination
  (source) This is not clear whether this syntax is right or not as I can't find many sources about it

mail

Hadoop glossary

default location : /user/userName/.Trash/Current
deleted data
- stays in the trash for an amount of time configured by fs.trash.interval
- can be restored simply by being moved out of the .Trash/ directory (source)

mail

count

Usage

count the number of directories, files and bytes under the paths that match the specified file pattern
get the quota and the usage

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#count

mail

du

Usage

Display

sizes of files and directories contained in the given directory
or the length of a file in case its just a file

Flags

Flag	Usage
-h	use human-readable units : K, M, G,
-s	sum the files sizes rather than displaying each file size individually

Example

hdfs dfs -du -s -h '/path/to/myData/dataSource/dataSet/year=2022/month=10'
```
119.5 K	/path/to/myData/dataSource/dataSet/year=2022/month=10
```

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#du

mail

find

Usage

find all files matching the specified expression and apply actions to them
if no path is specified, default to the current working directory
if no action is specified, default to -print

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#find

mail

ls

Usage

list files. Default output format :

for a file :
1. permissions
2. number of replicas
3. userId
4. groupId
5. size
6. modification date
7. modification time
8. name
for a directory :
1. permissions
2. userId
3. groupId
4. modification date
5. modification time
6. name

Flags

Flag	Usage
-C	display the paths of files but not the other fields (i.e. don't display permissions, owner, group, dates, ) like ls -1 does)

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#ls

mail

rm

Usage

delete files

Flags

Flag	Usage
-r -R	recursively delete a directory and its content The rmr command has been deprecated by rm -r
-safely	before deleting directory, prompt for confirmation if total number of files > hadoop.shell.delete.limit.num.files (set in core-site.xml, default: 100) it can be used with -skipTrash to prevent accidental deletion of large directories expect a delay when walking over large directory recursively to count the number of files to be deleted before the confirmation
-skipTrash	delete the specified files immediately instead of moving them to the trash This is a dangerous option which use is discouraged.

Source : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html#rm