Bash Index : P - The 'P' Bash commands : description, flags and examples

pv

Usage :

pv (pipe viewer, pv) shows the progress of data through a pipeline by giving information such as :

Example :

dd if=/dev/zero | pv | dd of=./zero.txt bs=1 count=10000000

pkill

Usage :

pkill [kill options] [process selection options]

Literally : process kill. Actually sends a kill signal to the selected process(es).

pkill has a fake twin brother : pgrep : one selects, the other kills .
They share the process selection options, so the workflow is usually :

  1. tune pgrep options to match the right process(es)
  2. use the same selection options to "pkill" them
For this reason, options related to process selection are detailed in the pgrep article only.

Example :

example

passwd

Usage :

change user password

Flags :

Flag Usage
-d --delete delete a user's password (make it empty)
-l --lock lock the password of the named account by changing the encrypted password into an invalid string. This is done by prefixing the encrypted string with an !.
passwd --lock does not actually disable the account : details.
-u --unlock unlock the password of the named account. This option re-enables a password by removing the leading ! added by -l.

More about passwd --lock :

  • passwd --lock does not disable the account : the user may still be able to login using another authentication token (e.g. an SSH key).
  • To disable the account,
    usermod --expiredate 1
    will set the account's expire date to Jan 2, 1970. This will result in :
    • the account password will be in expired state
    • expired passwords have a grace time (a few days) to be changed before the account is actually locked
    • the account will get a warning at each login
  • To "unexpire" the password (source) :
    chage stuart
    Changing the aging information for stuart
    Enter the new value, or press ENTER for the default
    
    	Minimum Password Age [0]: 
    	Maximum Password Age [99999]: 
    	Last Password Change (YYYY-MM-DD) [2018-06-08]: 
    	Password Expiration Warning [7]: 
    	Password Inactive [-1]: 
    	Account Expiration Date (YYYY-MM-DD) [1970-01-02]: -1

Example :

Let's try this !

useradd bob; grep bob /etc/shadow
bob:!:17547:0:99999:7:::	new passwordless account is disabled
passwd bob; grep bob /etc/shadow
Enter new UNIX password: 123456
Retype new UNIX password: 123456
passwd: password updated successfully
bob:$6$OXeqopLJ$8xicqHptSgZ/JuUFFV1az4keocSKSa53bAC9BFdMRqn1d92ltyXI1Mc2Vx2a708Jhsvszeduuvm/49/uPp6FP1:17547:0:99999:7:::	the encrypted password
passwd -l bob; grep bob /etc/shadow
passwd: password expiry information changed.
bob:!$6$OXeqopLJ$8xicqHptSgZ/JuUFFV1az4keocSKSa53bAC9BFdMRqn1d92ltyXI1Mc2Vx2a708Jhsvszeduuvm/49/uPp6FP1:17547:0:99999:7:::	the extra ! shows it's locked
passwd -u bob; grep bob /etc/shadow
passwd: password expiry information changed.
bob:$6$OXeqopLJ$8xicqHptSgZ/JuUFFV1az4keocSKSa53bAC9BFdMRqn1d92ltyXI1Mc2Vx2a708Jhsvszeduuvm/49/uPp6FP1:17547:0:99999:7:::	unlocked
passwd -d bob; grep bob /etc/shadow
passwd: password expiry information changed.
bob::17547:0:99999:7:::	empty password
passwd bob; grep bob /etc/shadow
Enter new UNIX password: 123456
Retype new UNIX password: 123456
passwd: password updated successfully
bob:$6$nAmbe8KT$Dl3GXxWEjhRhuCPUA7dFyA7MZvAXtkAVJm14n0asq9OVioh1b1OK2mE0pYmTJqKtnY.kz5jGiaEJhs1JAEkTi.:17547:0:99999:7:::	password is back. Same clear text, different crypted text
passwd -dl bob; grep bob /etc/shadow
passwd: password expiry information changed.
bob:!:17547:0:99999:7:::	now empty AND locked
passwd -u bob; grep bob /etc/shadow
passwd: unlocking the password would result in a passwordless account.
You should set a password with usermod -p to unlock the password of this account.
bob:!:17547:0:99999:7:::	no change : will not make a passwordless account
passwd bob; grep bob /etc/shadow
Enter new UNIX password: 123456
Retype new UNIX password: 123456
passwd: password updated successfully
bob:$6$fyixHXv.$YT56pM8ExEtfsght7E8cxyP1EYmLwrBp1mW8sJ/ZVn8J.K0G9gQUC7mMycqXM0lAda1/.3KU.4Q56CtT8OtcH/:17547:0:99999:7:::	password added
usermod --expiredate 1 bob; grep bob /etc/shadow
bob:$6$fyixHXv.$YT56pM8ExEtfsght7E8cxyP1EYmLwrBp1mW8sJ/ZVn8J.K0G9gQUC7mMycqXM0lAda1/.3KU.4Q56CtT8OtcH/:17547:0:99999:7::1:
userdel -r bob; grep bob /etc/shadow
(nothing)

parted

Usage :

A partition manipulation program

Example :

Interactive parted session

  • parted
  • (parted) select /dev/sdb
    Utilisation de /dev/sdb
  • (parted) mkpart primary ext2 0 -0
    Avertissement: L'alignement de la partition ainsi définie n'est pas optimal au niveau performance.
    Ignorer/Ignore/Annuler/Cancel?
    Alignment can NOT be optimal since we're using the whole disk.
    Ignore
    (nothing)
  • (parted) print
    Modèle: VMware Virtual disk (scsi)
    Disque /dev/sdb : 21,5GB
    Taille des secteurs (logiques/physiques): 512B/512B
    Table de partitions : msdos
    Disk Flags:
    
    Numéro	Début	Fin	Taille	Type	Système de fichiers	Fanions
     1	1049kB	21,5GB	21,5GB	primary
  • (parted) rm 1
    (nothing)
  • (parted) print
    Modèle: VMware Virtual disk (scsi)
    Disque /dev/sdb : 21,5GB
    Taille des secteurs (logiques/physiques): 512B/512B
    Table de partitions : msdos
    Disk Flags:
    
    Numéro	Début	Fin	Taille	Type	Système de fichiers	Fanions
  • (parted) quit
  • Unambiguous abbreviations are allowed : p for print, or u for unit
  • Commands are case-insensitive
(source, index of commands)

Non-interactive parted session

Create a partition using the whole disk
  • parted -a optimal -s /dev/sdb -- mkpart primary ext2 0 -1s
  • parted -a optimal -s /dev/sdb -- mkpart primary ext2 0 -0
  • ext2 is there just to set the partition flag. No filesystem will be created.
  • Negative numbers count back from the end of the disk
  • If no unit is given, the default unit is assumed defaults to compact. Read about unit for details
  • -1s means "the last sector of the disk"
Check
  • parted -l displays everything
  • parted /dev/sdb print
Delete a partition
parted /dev/sdb rm 1

When parted complains Error: Can not open /dev/sdc - unrecognized disk label.

Well, it's all in the error message : /dev/sdc has a disk label parted is unable to handle.
In my (possibly specific) case, I was trying to manage virtual disks on virtual machines with VMware. And these disks simply had no label. So let's add one prior to any operation :

parted -a optimal -s /dev/sdc -- mklabel msdos mkpart primary ext2 0 -1s

parallel

Usage :

parallel (GNU Parallel) is not a standard shell command, but can be installed on Debian with the parallel package.

parallel is designed to build and execute shell command lines from standard input in parallel and should be straightforward for xargs and tee enthusiasts. parallel can often be used as a substitute for xargs or cat something | bash command.

synopsis :

  • parallel [options] [command [arguments]] < list_of_arguments
  • parallel [options] [command [arguments]] ( ::: arguments | :::+ arguments | :::: argfile(s) | ::::+ argfile(s) ) ...
The parallel command -- arguments syntax has been replaced by parallel command ::: arguments.

Flags :

Flag Usage
-j maxJobs run maxJobs jobs simultaneously
-i Enable argument placeholder (see examples below)
-n nbArgs number of arguments to pass to a command at a time. Defaults to 1.

Example :

Parallel downloads with wget.
parallel echo Hello world ::: 1 2 3
Hello world 1
Hello world 2
Hello world 3
parallel echo Hello world ::: {1..10}
Hello world 1
Hello world 2
...
Hello world 10
tmpFile=$(mktemp XXX.tmp); parallel "echo hi >> $tmpFile; sleep 1; echo bye >> $tmpFile" ::: 1 2 3; cat "$tmpFile"; rm "$tmpFile"
hi
hi
hi
bye 1
bye 2
bye 3
tmpFile=$(mktemp XXX.tmp); parallel "echo hi_{} >> $tmpFile; sleep 1; echo bye >> $tmpFile" ::: 1 2 3; cat "$tmpFile"; rm "$tmpFile"
hi_1
hi_2
hi_3
bye
bye
bye
tmpFile1=$(mktemp XXX.tmp); tmpFile2=$(mktemp XXX.tmp); seq 1 5 >>"$tmpFile1"; parallel -j 3 "echo \"Start {}\">>$tmpFile2; sleep 1; echo \"Stop {}\">>$tmpFile2" < "$tmpFile1"; cat "$tmpFile2"; rm "$tmpFile1" "$tmpFile2"
Start 1
Start 2
Start 3
Stop  1
Stop  2
Stop  3
Start 4
Start 5
Stop  4
Stop  5
listOfItemsToProcess=$(mktemp XXX.tmp); nbParallelProcesses=3; outputHistory=$(mktemp XXX.tmp); seq 1 5 >>"$listOfItemsToProcess"; parallel -j $nbParallelProcesses "echo \"START {}\">>$outputHistory; sleep 1; echo \"STOP {}\">>$outputHistory" < "$listOfItemsToProcess"; cat "$outputHistory"; rm "$listOfItemsToProcess" "$outputHistory"
START 1
START 2
START 3
STOP 1
STOP 2
STOP 3
START 4
START 5
STOP 4
STOP 5

pgrep

Usage :

process grep : look through the currently running processes and lists the process IDs which match the selection criteria to stdout. All the criteria have to match.

pgrep options pattern

pgrep foo will match all processes whose executable file name (i.e. process name) matches foo. If foo is part of the path but not of the executable file name, pgrep will find no match. To match against the executable file name and path, consider -f.

Flags :

Flag Usage
-f --full Match pattern against the full command line rather than just against the process name (binary)
-l --list-name Return the PID + process name
-n --newest Select only the newest (most recently started) of the matching processes
-u UID -u bob Match processes owned by UID / Bob (effective UID)
-U UID -U bob Match processes owned by UID / Bob (real UID)
-v Negates the matching

ping

Flags :

Flag Usage
(none) ping until stopped by CTRL-c
-b broadcast ping : ping -b x.x.x.255
-c n ping count : send n probe packets
-f ping flood : the local machine sends ping requests without waiting for an echo before sending the next request. For each sent request, a dot . is displayed, for each received echo, a backspace is displayed.
-i n ping with interval : wait n seconds between 2 packets. (By default, 1 second)
-I sourceInterface ping from Interface : sourceInterface can be either an interface name (eth0) or its IP address
-v verbose mode
This mode outputs the following line twice :
ping: socket: Permission denied, attempting raw socket...
which is more a debug message than the sign something is going wrong. You can safely ignore it (details).
-R Record Route
-s packetSize data-loaded ping : packetSize bytes (+ headers) are sent
-w deadline timeout, in seconds, before ping exits regardless of how many packets have been sent or received
-W timeout time to wait for a response, in seconds

Example :

Flooding from a FreeBSD host

ping -f -D -s 1472 x.x.x.x
sends a flood (-f) unfragmented (-D), data-loaded (-s) ping. The 1472 payload bytes are the maximum since we're sending an unfragmented packet on a LAN, where the MTU is 1500 bytes.
  • 1472 (payload) + 8 (IP headers) + 20 (ICMP ECHO REQUEST) = 1500 bytes
  • When receiving such input, an XP host has its network about 75% busy, and CPU about 30-40% busy.

By default, ICMP_ECHO_REPLY is disabled in XP SP1.

Destination host unreachable

You'll often get the Destination host unreachable error message with ping. This actually means you are 1 step ahead of "no response at all", which is good news but does not help much either . To further investigate, consider traceroute.
(see also)

ps

Usage :

ps reports a snapshot of the current processes. It accepts several kinds of options :
pgrep is sometimes more appropriate than ps [basic options] | grep

Flags :

Flag Usage
-a select all processes except both session leaders and processes not associated with a terminal
a lift the BSD-style "only yourself" restriction. Select all processes with a terminal. Select all processes when used with x
-A or -e select All processes
-C commandList select processes whose executable name is given in commandList
f forest : ASCII-art process hierarchy
-f full-format listing
-F extended full-format listing. Implies -f
l display BSD long format
-L Show Lthreads, possibly with LWP and NLWP columns
o formatSpecString
-o formatSpecString
--format formatSpecString
introduces the output formatting options
-O formatSpecString
(uppercase "o")
same as -o but preloaded with some default columns. Identical to -o pid,format,state,tname,time,command
--ppid n select processes which parent process ID is n
U userId
-u userId
--user userId
select processed owned by userId (user name orUID)

ps may display the user's uid instead of the user's name if the username is longer than 7 characters.

Output format flags (source) :

Flag Usage
args displays the command that is executing, with all its arguments
comm displays the name of the command that is executing (the binary ?)
etime elapsed time since the process was started : [[dd-]hh:]mm:ss
lstart Date + time the process was started : Fri Aug 21 23:32:21 2015.
lwp ID of the light weight process (thread)
nlwp thcount number of light weight processes (threads)
pcpu %cpu ratio : cputime of the process / realtime
pid PID of the selected process
ppid PID of the parent of the selected process
pmem %mem ratio : RSS / physical memory
rss rssize rsz resident set size : the non-swapped physical memory that a task has used, in kiloBytes (details : 1, 2)
sgi_p ID of the processor currently executing the process. Will display * if the process is not currently being executed
start If the process was started less than 24 hours ago, display its start time in 24h:mm:ss format. Otherwise, display the start date formatted like Aug 21.
start_time If the process was started today, display its start time in 24h:mm format. Otherwise, display the start date formatted like Aug21.
state State of the process : Running, Sleeping, Zombie, ...
user euser uname name of the user owning the process
vsize vsz virtual memory (=SWAP) size of the specified process, in KiB (needs details)
The output format can be specified in the form :
ps -o comma-separated,list,of,flags
You can change a column header by specifying a custom value for the selected flag :
ps -o pid='Process_ID',tt=TTY,time,comm=Command

If the custom value is empty, no header is displayed, which is convenient to output process names only : ps -o comm=

Example :

List all running processes :

  • ps -elf
  • ps aux / ps faux (Their output is not equivalent as both command don't return the same amount of results (check it with : watch -n 1 -d 'echo "-elf : "$(ps -elf | wc -l);echo " aux : "$(ps aux | wc -l)'))
  • pstree

List the 10 most CPU-hungry processes (source) :

ps -eo pcpu,pid,state,user,lstart,args | sort -k 1 -r | head -10

Find the amount of RAM (in KiB) used by a process :

normal command :
ps -C firefox-esr -o comm,rss
with the = hack to hide the header line :
ps -C firefox-esr -o rss=
in MiB :
  • ps -o comm,rss | awk '/firefox-esr/ {print $2 / 1024}'
  • echo "$(ps -C firefox-esr -o rss=) / 1024" | bc
  • let ramUsed=$(ps -C firefox-esr -o rss=)/1024; echo $ramUsed

Get extra information using the output formatting flags :

ps -p942 -o %cpu,%mem,lwp,nlwp,sgi_p,start,start_time,vsz,wchan,rss,user

Find zombies:

ps -e -o pid,ppid,args,state | grep -E 'Z$'
More about zombies (source) :

Zombie processes are already dead, so they can't be killed. They are the consequence of malfunction (or programming defect) in they parent processes. Here is how processes die :

  1. A process finishes, its status changes to EXIT_ZOMBIE.
  2. Its parent is notified of the termination of a child process by receiving the SIGCHLD signal.
  3. The parent process is expected to read the child process' exit status and stats with wait(). During that time, the child process is a zombie.
  4. Then the zombie process is completely removed from memory and from the process table.
This happens almost instantly, so in normal conditions, we may not see zombie processes.
Consequence of having zombies :
The footprint of a zombie is only the memory required to store its process descriptor, so zombies have no impact on system resources or performance : a few zombies are harmless. The only impact they _could_ have is that every zombie has its own PID, and PIDs are limited to 32768 (cat /proc/sys/kernel/pid_max). So an uncontrolled growth of the zombie population may exhaust the pool of PIDs.
Getting rid of zombies :
Zombies are dead processes. You cannot kill the dead. What you can try, is to send the SIGCHLD signal to their parent process (but that obviously already failed once, so...). To effectively definitely remove a zombie process :
  1. Kill its parent process
  2. The init process inherits the zombie process
  3. init periodically runs wait(), cleaning up its own zombie children.

How to find the full path of a command returned by ps ?

  • ps -eF | awk '/binary/ {print $2}' | xargs -I pattern ls -l /proc/pattern/exe
  • The command above will show a ls: cannot access /proc/whateverPid/exe: No such file or directory because awk matches its own process, which doesn't exist anymore once piped to ls via xargs. To workaround this :
    ps -eF | awk '$11 ~ "binary" {print $2}' | xargs -I pattern ls -l /proc/pattern/exe
  • If there are MANY processes (possibly) running the same binary (e.g. when running Oracle), you can check this by sorting the displayed binaries :
    ps -eF | awk '$11 ~ "binary" {print $2}' | xargs -I pattern ls -l /proc/pattern/exe | awk '{print $NF}' | sort -u

paste

Usage :

Concatenate the corresponding lines of the input files.

Example :

Considering 2 files :
  • file1
    file1, line1
    file1, line2
    file1, line3
  • file2
    file2, line1
    file2, line2
    file2, line3
paste file1 file2
file1, line1	file2,line1
file1, line2	file2, line2
file1, line3	file2, line3

pwgen

Usage :

pwgen generates passwords which are designed to be easily memorized by humans, while being as secure as possible. Human-memorable passwords are never going to be as secure as completely completely random passwords. In particular, passwords generated by pwgen without the -s option should not be used in places where the password could be attacked via an off-line brute-force attack.

Flags :

Flag Usage
-0 --no-numerals No number in the generated password
-1 Prints the generated passwords one per line
-A --no-capitalize No capital letter in the generated password
-c --capitalize Include at least one capital letter in the password (enabled by default).
-n --numerals Include at least one number in the password (enabled by default).
-s --secure Generate completely random, hard-to-memorize passwords.
-y --symbols Include at least one special character in the password.

Example :

Generate a screenful of n-characters long password :

pwgen n

Generate a truly secure password :

Considering that :
  • ALL 8-characters passwords have been cracked and are probably already in rainbow tables
  • passwords up to 13 characters can be cracked in less than 24 hours
  • the size of the character sets matter more than the password length

pwgen -ys 15

The most important (and forgotten) thing regarding password security is changing it regularly : you're still safe if you've changed your password before it's cracked.

Generate some funny words :

This is for fun only : the words generated with the methods below —albeit created with a password generator— must NOT be used as passwords. Indeed, the syntax rules (lowercase characters only, patterns of alternating consonants / vowels) makes them extremely poor passwords.

These methods can be used to make up characters / places / cities / ... names in fictional writing work, for instance (don't forget to adapt the "selection RegExp" to the length of words you wish to generate).

single word :
wordLength=10; generateOneWord() { pwgen -0 -1 -A "$wordLength" | grep -Ei '^[b-df-hj-np-tv-xz][aeiouy]{1,2}[b-df-hj-np-tv-xz][aeiouy]{1,2}[b-df-hj-np-tv-xz]{1,2}'; } ; until generateOneWord ; do true; done
multiple words :
wordLength=10; nbWordsToGenerate=12; nbGeneratedWords=0; generateOneWord() { consonant='[b-df-hj-np-tv-xz]'; vowel='[aeiouy]'; pwgen -0 -1 -A "$wordLength" | grep -Ei "^($consonant$vowel{1,2}$consonant$vowel{1,2}$consonant{1,2}$vowel{2}|$vowel$consonant{1,2}$vowel{1,2}$consonant{1,2}$vowel{1,2}$consonant)"; } ; until [ "$nbGeneratedWords" == "$nbWordsToGenerate" ] ; do generateOneWord && nbGeneratedWords=$((nbGeneratedWords+1)); done