Bash Index - My favorite Bash commands : description, flags and examples

bzip2

Compress data need a little more details...
You definitely should have a look at pbzip2 :
  1. apt-get install pbzip2
  2. Create aliases :
    alias bzip2='pbzip2'
    alias bunzip2='pbzip2 -d'

journalctl

Usage :

Query the systemd journal
Wrapping long lines is not very intuitive.

Flags :

Flag Usage
-f
--follow
Show only the most recent journal entries, and continuously print new entries as they are appended to the journal (i.e. like tail -f)
--no-pager Do not pipe output into a pager
a pager is a utility used to display text, scroll up/down, scroll 1line/1page, search, ..., such as less
-o outputOption
--output=outputOption
Controls the formatting of the journal entries that are shown. Among available values : verbose
-r
--reverse
reverse output so that the newest entries are displayed first
-S 'YYYY-MM-DD hh:mm:ss'
--since='YYYY-MM-DD hh:mm:ss'
Show entries on/newer than the specified timestamp. Missing time will be interpreted as 00:00:00, missing seconds will be interpreted as :00
-u unit
--unit=unit
Show messages for the specified systemd unit unit
-U 'YYYY-MM-DD hh:mm:ss'
--until='YYYY-MM-DD hh:mm:ss'
Show entries on/older than the specified timestamp. Missing time will be interpreted as 00:00:00, missing seconds will be interpreted as :00

Example :

Some commands :

How to wrap long lines ?

Lines output by journalctl are often longer than the display width, and the interesting part of messages is generally at the end of the line (i.e. not visible )

Just use the key to view long lines

man journalctl is not very helpful when it comes to wrapping long lines (search wrap and pager for details). To view long lines, you can also :
journalctl -u networking --no-pager
But this will throw everything to the screen, making it impossible to read. So you'll end up with :
journalctl -u networking --no-pager | less
i.e. asking no pager, please in the first place, then piping everything into ... a pager .

This construct could prove useful in scripts, however it is not necessary when trying to grep something which is hidden by the default pager :

journalctl -u networking --output=verbose | grep foo
will actually search and find foo, even though foo is beyond the right screen border when running :
journalctl -u networking --output=verbose

vipw

Usage :

vipw is designed to allow safe edition of the /etc/passwd and /etc/shadow files by putting a lock on the edited file to prevent file corruption due to concurrent modifications (like editing while is changing his password). vipw starts vi to edit the files.

Flags :

Flag Usage
-s Edit the shadow file

Example :

edit /etc/password :

vipw

edit /etc/shadow :

vipw -s

break

Usage :

break n exits from the smallest enclosing for, while, or until loop, or from the nth enclosing loop, if n is specified (defaults to 1).

Example :

Basic example : interrupting a single for loop :

#!/usr/bin/env bash
for i in {1..2}; do
	echo $i
	break
	echo 'this will never be executed'
done
displays :
1

2 nested for loops :

#!/usr/bin/env bash
for i in {1..2}; do
	for j in {a..b}; do
		echo $i$j
		break
		echo 'this will never be executed'
	done
done
displays :
1a
2a

2 nested for loops and break 2 :

#!/usr/bin/env bash
for i in {1..2}; do
	for j in {a..b}; do
		echo $i$j
		break 2
		echo 'this will never be executed'
	done
done
displays :
1a

vmstat

Usage :

Report virtual memory statistics : vmstat [options] delay|count

Flags :

Output fields :

  • Swap :
    • si : Amount of memory swapped in from disk per second
    • so : Amount of memory swapped out to disk per second

Example :

Am I swapping ?

Run : vmstat -n 1
If si and so fields are always 0, then the system is currently not swapping. If free shows swap usage, it must be because some application has used the swap but somehow its not cleaned yet. At such situation, a swapoff + swapon command would be handy.

join

Usage :

Join 2 files on their matching records, as does the SQL operator.

Example :

The Daltons :

Considering 2 files :

  • names.txt
    Joe Dalton
    Jack Dalton
    William Dalton
    Averell Dalton
  • ages.txt
    Joe 23
    Jack 22
    William 21
    Averell 20

join names.txt ages.txt returns :

Joe Dalton 23
Jack Dalton 22
William Dalton 21
Averell Dalton 20

xargs

Usage :

Construct argument lists and invoke utility : xargs options utility

xargs may return exit codes such as 123, 124, 125, 126 or 127 if something went wrong when calling 'utility' (for details : ).

By default, xargs reads newline-separated input values.

Flags :

Flag Usage
-I pattern Replace pattern with values piped into xargs (example)
-n n Pop n arguments at a time from the list to feed utility
-P n Execute at most n parallel instances of utility. Defaults to 1. (source).
-n should be specified when using -P or there is a risk of firing only 1 instance of utility.

Example :

tar all files of the current directory :

find . -maxdepth 1 -type f | xargs tar cf archive.tar

Download all URLs listed in a file (source) :

cat urlList.txt | xargs wget -c
This example, albeit quite functional, is actually deprecated by wget's -i.

Archive the n latest files matching pattern :

ls -t *pattern* | head -n n | xargs tar cfz /path/to/archive.tgz

Cannot umount a filesystem because of open files ? Kill, kill, KILL !!! :

Killing so many processes this way is VERY! BAD!. Do this only if :

  • these processes are started at boot time
  • AND you can not switch to a lower runlevel
  • AND you can not stop them all
  • AND you plan to reboot shortly
  • AND (many other reasons to convince you this is a bad thing )

lsof /mnt/myFilesystem | awk '{print $2}' | sed '1d' | xargs kill -15

How to use the values piped to xargs into its sub-command ?

for i in {1..5}; do echo $i; done | xargs -I pattern echo 'this is line pattern.'

Looks like xargs substitutions have precedence over Bash.
To use values more than once :

for i in {a..e}; do echo $i; done | xargs -I whatIWant sh -c 'echo -n "I can say \"whatIWant\"."; echo " And I can say it even louder : \"$(echo whatIWant | tr '[:lower:]' '[:upper:]') !\""'

sh -c 'someCommand' above means "run someCommand within the /bin/sh shell". Some "advanced" commands / flags / options may not be available in this shell. Consider bash -c 'someCommand' then.

bc

Usage :

bc is a calculator that can be used either interactively or within scripts

Example :

Some basic examples :

echo '1+1' | bc
2
echo '331507551852/1024/1024/1024' | bc
308
echo 1/2 | bc; echo 1/2 | bc -l
0
.50000000000000000000
Looks like the -l flag is only necessary for divisions. Without it, bc makes "integer divisions" : for i in {8..16}; do echo -n "$i / 3 = "; echo $i/3 | bc; done :
8 / 3 = 2
9 / 3 = 3
10 / 3 = 3
11 / 3 = 3
12 / 3 = 4
13 / 3 = 4
14 / 3 = 4
15 / 3 = 5
16 / 3 = 5

How to round float numbers into integers (source, see also) ?

echo '(3.141592654+0.5)/1' | bc
3
echo '(2.71828+0.5)/1' | bc
3
printf %0.f 3.141592654
3
number=3.333; multiplier=3; echo $(printf %0.f $(echo "$number*$multiplier" | bc))
10

Whether or not you're using the -l flag does not return the same result :
number=4; divider=5; echo "$number/$divider" | bc; echo "$number/$divider" | bc -l

0
.80000000000000000000
So the rounded values will be VERY different too.

How to convert numbers between HEX, BIN and DEC (source) ?

To do so, use the ibase (input base) and obase (output base) bc options :

echo 'ibase=16; FF' | bc Hexadecimal letters must be written uppercase.
255
echo 'obase=16; 255 | bc
FF
echo 'ibase=10; obase=2; 73' | bc
1001001

base64

Usage :

Encode and decode strings (or files) into/from base64 format.

Flags :

Flag Usage
(none) encode parameter into base64 format
-d base64EncodedString
--decode base64EncodedString
decode base64EncodedString
-w n
--wrap=n
wrap encoded lines after n characters (defaults to 76).
Use 0 to disable line wrapping (i.e. output a giant single line)

Example :

Encode (same with -d to decode) :

  • echo -n HelloWorld | base64
  • Or :
    1. base64 -
    2. HelloWorld If you press now, it'll be appended to the string to encode.
    3. CTRL-d-d
    4. The encoded string will be displayed right after the input string : SGVsbG9Xb3JsZA==

kill / killall / pkill

Usage :

from the softest to the toughest kill :
  1. kill -1
  2. kill -15
  3. kill -9
The CTRL'ed keys and the shell (Source 1, 2) :

When a CTRL-... key sequence is pressed in a terminal, the terminal sends a corresponding signal to the shell, which forwards it to the stdin of the currently running program.
Mapping of key sequences / signals : stty -a
Bash shortcut keys

Flags :

Flag Usage
-l list available signals
-signalName
-s signalName
send the signal signalName.
Examples :
kill -s stop 12345
kill -s STOP 12345
kill -s sigstop 12345
kill -s SIGSTOP 12345
sleep 10 & kill -1 $!
sleep 10 & kill -hup $!
sleep 10 & kill -sighup $!
sleep 10 & kill -s hup $!
sleep 10 & kill -s sighup $!
Signal Description # Usage
SIGHUP Hangup 1 This argument makes kill send the Hang Up signal to processes. This probably originates from the modem/dial-in era. Processes have to be programmed to actually listen to this signal and do something with it. Most daemons are programmed to re-read their configuration when they receive such a signal. Anyway, this is very likely the safest kill signal there is, it should not obstruct anything.
SIGINT Interrupt 2 This signal is sent to the program being executed when CTRL-c is pressed in a terminal. Most programs will stop, you may lose data.
SIGQUIT Quit and dump core 3
SIGILL Illegal instruction 4
SIGTRAP Trace/breakpoint trap 5
SIGABRT Process aborted 6
SIGBUS Bus error: "access to undefined portion of memory object" 7
SIGFPE Floating point exception: "erroneous arithmetic operation" 8
SIGKILL Kill (terminate immediately) 9 The kernel will let go of the process without informing the process of it :
  1. the process will vanish immediately, without completing its current task
  2. the process won't have the opportunity to free its resources (RAM, files, sockets, ...)
  3. children processes won't be notified of their father dying. As orphans, they'll be adopted by init

kill -9 could result in data loss. This is the "hardest", "roughest" and most unsafe kill signal available, and should only be used to stop something that seems unstoppable.

This signal is NOT catchable.
SIGUSR1 User-defined 1 10
SIGSEGV Segmentation violation 11
SIGUSR2 User-defined 2 12
SIGPIPE Write to pipe with no one reading 13
SIGALRM Signal raised by alarm 14
SIGTERM Termination (request to terminate)
This is the default signal.
15 Tell the process to :
  1. stop whatever it's doing
  2. inform its children processes that it's being stopped
  3. wait for children processes to stop
  4. free resources
  5. end itself
This is a normal process shutdown, which is safe to perform. Not a bad idea anyway trying to send a -1 / -HUP signal 1st.
There _may_ be situations where the process is on an uninterruptible task and ignores this signal.
SIGCHLD Child process terminated, stopped (or continued*) 17
SIGCONT Continue if stopped 18
SIGSTOP Stop executing temporarily 19 This signal is NOT catchable.
SIGTSTP Terminal stop signal 20 This signal is sent after a CTRL-z in a terminal.
SIGTTIN Background process attempting to read from tty ("in") 21
SIGTTOU Background process attempting to write to tty ("out") 22
SIGURG Urgent data available on socket 23
SIGXCPU CPU time limit exceeded 24
SIGXFSZ File size limit exceeded 25
SIGVTALRM Signal raised by timer counting virtual time: "virtual timer expired" 26
SIGPROF Profiling timer expired 27
SIGPOLL Pollable event 29
SIGSYS Bad syscall 31

More about signals :

  • for details : man 7 signal
  • catchable signals : when such a signal is sent to a process, it is able to catch it with trap (or ignore it) and perform various "shutdown" operations it needs to do. Non-catchable signals affect the process immediately.

basename

basename file : returns the file name without its path.

basename file string : same as above + removes string from the end of the file name. This can be used to remove a known file extension.

yum

Usage :

Install software packages while maintaining dependencies on Red Hat.

Configuration files (details) :

  • /etc/yum.conf : yum main configuration file
  • /etc/yum.repos.d/* : repositories

Flags :

Flag Usage
makecache
makecache fast
download and make usable all the metadata for the currently enabled yum repositories
with fast : just try to make sure the repositories are current
repolist list configured repositories. Filter with all (default) / enabled / disabled
-y --assumeyes assume that the answer to any question which would be asked is yes

Example :

Deal with repositories :

List repositories (source) :
  • yum repolist all
  • verbose mode : yum repolist -v
  • repo IDs only : yum repolist | tail -n +4 | head -n -1 | cut -d' ' -f1

An exclamation mark ! leading a repository name means this repository has expired metadata.

Enable / disable a repository (source) :
  • yum-config-manager --enable repositoryId
  • yum-config-manager --disable repositoryId
repositoryId is displayed when listing repositories.

Deal with packages :

List installed packages (source) :
yum list installed
list all installed packages
yum list installed docker-ce
search a specific package
You'll have to provide the exact package name. Otherwise :
yum list installed | grep packageSearchPattern
yum list output format (source) :
name.arch		[epoch:]version-release		repo or @installed-from-repo
Search a package :

yum search packageSearchPattern
This displays a list of matching patterns, their version, and the repository they belong to or the string installed when applicable.

Get information on a package :
yum info packageSearchPattern
List files provided by a package (installed or not) :
Install package using a given repository :
yum -c /etc/yum.repos.d/myCompany.repo install fetchmail

Install security updates (source) :

List all updates that are security relevant, and get a return code on whether there are security updates :
yum check-update --security
Upgrade packages that have security errata up to the latest available package :
yum update --security
Upgrade packages that have security errata up to the last security errata package :
yum update-minimal --security

Don't forget -y when running non-interactively (like in an Ansible playbook)

Get a summary of advisories not installed yet :
yum updateinfo summary
Modules complémentaires chargés : product-id, search-disabled-repos, subscription-manager
Updates Information Summary: updates
	28 Bugfix notice(s)
	 4 Enhancement notice(s)
Security: kernel-3.10.0-514.26.1.el7.x86_64 is an installed security update
Security: kernel-3.10.0-514.10.2.el7.x86_64 is the currently running version		reboot !
updateinfo summary done
update vs update-minimal :
yum update
update every currently installed package (and ensure that all dependencies are satisfied)
yum update-minimal updateType
like update, but if you have the package foo-1 installed and have foo-2 (bugfix) and foo-3 (enhancement) available, then update-minimal --bugfix will update you to foo-2.