Bash Index : S - The 'S' Bash commands : description, flags and examples

stdbuf

Usage :

stdbuf options command
Run command with modified buffering operations for its standard streams.
Some man pages show this example :
tail -f access.log | stdbuf -oL cut -d aq aq -f1 | uniq
WTF does this aq aq mean ? Is it an advanced cut option ? Let's find out on cut manpage.
No reference to aq aq in the documentation, but we can still see in the "See also" section :
info coreutils aqcut invocationaq

Looks like someone had a hard time importing man pages having single quotes '.
This is confirmed by reading the documentation in a terminal :

This typo has been copy-pasted as-is many times .

Flags :

Flag Usage
-omode --output=mode adjust standard output stream buffering to mode

Buffering modes :

  • L : the corresponding stream will be line buffered. This option is invalid with standard input.
  • 0 : the corresponding stream will be unbuffered

Example :

Immediately display unique entries from access.log :

tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq

journalctl -u ssh | cut -d ' ' -f6- | uniq
journalctl -u ssh | stdbuf -oL cut -d ' ' -f6- | uniq

	==> both _seem_ to behave the same. Maybe "journalctl" is not a good candidate to experiment on this ;-)

script

Usage :

Make typescript of terminal session, which could be handy when writing tutorials or documenting an installation process, for example.

script options /path/to/typescript/file

Flags :

Flag Usage
-a append to /path/to/typescript/file rather than overwriting it / creating a new file
-c command run command instead of an interactive shell
-f flush output after each write

Example :

How to use it :

  1. script /path/to/typescript/file
  2. the commands you type and their output is recorded into /path/to/typescript/file but not recorded in real time, unless -f is used
  3. stop recording :
    • exit
    • CTRL-d
  4. You can now view the typescript :
    • cat /path/to/typescript/file
    • less -R /path/to/typescript/file

seq

This command is listed here for the sake of completeness, but it has the pitfall of starting unnecessary new processes. This is why, in most cases —especially in scripts / loops— shell brace expansion should be used instead (details).
Print a sequence of numbers :
ascending
seq 4 8
4
5
6
7
8
ascending with interval
seq 5 3 17
5
8
11
14
17
descending, with mandatory interval
seq 17 -3 5
17
14
11
8
5

shred

Usage :

Overwrite the specified file repeatedly, in order to make it harder for even very expensive hardware probing to recover the data, and optionally delete it.
shred relies on a very important assumption: that the file system overwrites data in place. This is the traditional way to do things, but many modern file system designs do not satisfy this assumption. The following are examples of file systems on which shred is not effective, or is not guaranteed to be effective in all file system modes:
  • log-structured or journaled file systems, such as those supplied with AIX and Solaris (and JFS, ReiserFS, XFS, Ext3, ...)
  • file systems that write redundant data and carry on even if some writes fail, such as RAID-based file systems
  • file systems that make snapshots, such as Network Appliance's NFS server
  • file systems that cache in temporary locations, such as NFS version 3 clients
  • compressed file systems
Regarding Ext3
  • In data=journal mode (which journals file data in addition to just metadata), the above disclaimer applies and shred is of limited effectiveness.
  • In both the data=ordered (default) and data=writeback modes, shred works as usual.
  • journaling modes can be changed by adding data=something to the mount options for a particular file system in /etc/fstab.
There is no consensus on how to REALLY, securely, erase a disk. Things get even worse when it comes to SSDs. The only good advice around seems to be to encrypt the drive itself. This way, whatever is still readable on the drive (i.e. everything) is still protected by the encryption key.

Flags :

Flag Usage
-n iterations --iterations=iterations overwrite iterations times instead of the default (3)
-u --remove truncate and remove file after overwriting
  • The default is not to remove the files because it is common to operate on device files like /dev/hda, which should not be removed.
  • Using --remove is safe (recommended!) when operating on regular files.
-v --verbose see shred working : successively overwriting the target file, then renaming it, then deleting it :
myTempFile=$(mktemp); echo "$myTempFile contains secret data" > "$myTempFile"; cat "$myTempFile"; shred -uv "$myTempFile"
-z --zero add a final overwrite with zeros to hide shredding

Example :

shred -n 35 -z -u filename

Parameters :
-n 35
overwrite 35 times the file with random bytes
-z
then overwrite the file with zeros
-u
truncate then delete the file

systemctl

Usage :

Utility to control "stuff managed by systemd" a.k.a units

Flags :

Flag Usage
-l --full Do not ellipsize unit names, process tree entries, journal output, or truncate unit descriptions in the output of status, list-units, list-jobs, and list-timers
daemon-reload Reload systemd's configuration. This will rerun all generators (see systemd.generator(7)), reload all unit files, and recreate the entire dependency tree. While the daemon is being reloaded, all sockets systemd listens on behalf of user configuration will stay accessible.
not be confused with reload
reload pattern Asks all units listed on the command line to reload their configuration (i.e. ask daemons managed by systemd to reload their own configuration)
This will reload the service-specific configuration, not the unit configuration file of systemd. If you want systemd to reload the configuration file of a unit, use daemon-reload. In other words: for the example case of Apache, this will reload Apache's httpd.conf in the web server, not the apache.service systemd unit file.
not be confused with daemon-reload

Example :

Start / stop / status of a daemon :

systemctl start|stop|status daemon.service

While debugging, it may be useful to prefix systemctl invokation with date :

date; systemctl start docker.service
so that it's easier to identify journalctl entries if the operation failed.

View the unit configuration file of a service :

systemctl cat docker.service

su

Usage :

change user ID or become superuser

Flags :

Flag Usage
- -l --login Make the shell a login shell. The environment will be changed to what would be expected if the user actually logged in as the specified user. Otherwise, the environment is passed along, with the exception of $PATH.
-c command --command=command Pass the command command to the shell.
-s shell --shell=shell Invoke the shell shell

Example :

If "sudo su -" returns This account is currently not available.

Trying to become root ends either on su: Authentication failure or on su: Permission denied

In the scenarios below, running either su or su - makes no difference.
What I've tested :
as a member of the root group + su + right password
logged as root, as expected
as a member of the root group + su + wrong password
su: Authentication failure
not as a member of the root group + su + any password (right or wrong makes no difference)
su: Permission denied

This behaviour is caused by this uncommented line of /etc/pam.d/su which forces users to be a member of group root before they can use su :

auth	required	pam_wheel.so	group=root

random notes

man su on Debian :
	-, -l, --login
	Fournir à l'utilisateur un environnement similaire à celui qu'il aurait obtenu s'il s'était connecté directement.
	Lorsque - est utilisé, il doit être indiqué comme dernier paramètre de su. Les autres formes (-l et --login) ne présentent pas cette restriction.


http://ss64.com/bash/su.html
	-, -l, --login
	Make the shell a login shell. This means the following:
		unset all environment variables except `TERM', `HOME', and `SHELL' (which are set as described above), and `USER' and `LOGNAME' (which are set, even for the super-user, as described above)
		set `PATH' to a compiled-in default value.
		Change to USER's home directory.
		Prepend `-' to the shell's name, intended to make it read its login startup file(s).


some tests as root :

su bob -c "whoami"
	bob

su bob -c "whoami; echo $HOME"
	bob	==> Expected, as we didn't change the environment.
	/root

su - bob -c "whoami; echo $HOME"
	bob	==> I expected /home/bob here. What happened ?
	/root

/!\ this _could_ be caused by the fact that logging as root was ALREADY made via "su" (?)

set

Usage :

Toggle options within a script. At the point in the script where you want the options to take effect, use set -o optionName or, in short form : set -shortOption. These two forms are equivalent.
To disable an option : set +o optionName, or set +shortOption.

Flags :

optionName shortOption Usage
noexec n Read commands in script, but do not execute them (syntax check)
errexit e Abort script at first error, when a command exits with non-zero status (except in until or while loops, if-tests, list constructs)
nounset u Attempt to use undefined variable outputs error message, and forces an exit
verbose v Print each command to stdout before executing it
xtrace x Similar to verbose, but expands commands

ss

Usage :

ss is used to dump socket statistics. It allows showing information similar to netstat. It can display more TCP and state informations than other tools. (Some even say that "ss directly queries the kernel and can respond much faster that netstat")

ss -pan is a good equivalent to netstat -laputen (which will be deprecated soon)

Flags :

Flag Usage
-a --all Display all sockets
-l --listening Display listening sockets only
-n --numeric Show service names in numeric format
-r --resolve resolve numeric address/ports
-p --processes Show process using socket
-t --tcp Display only TCP sockets
-u --udp Display only UDP sockets
-x --unix Display only Unix domain sockets
(filters)

Filters :

State filters :
state anyTcpState, to be chosen from (source) :
  • All standard TCP states: established, syn-sent, syn-recv, fin-wait-1, fin-wait-2, time-wait, closed, close-wait, last-ack, listen and closing.
  • all : for all the states
  • connected : all the states except for listen and closed
  • synchronized : all the connected states except for syn-sent
  • bucket : states, which are maintained as minisockets, i.e. time-wait and syn-recv
  • big : opposite to bucket
Address filters :
  • dst addressPattern : matches remote address and port
  • src addressPattern : matches local address and port
  • dport operator anyPort : compares remote port to a number
  • sport operator anyPort : compares local port to a number
  • autobound : checks that socket is bound to an ephemeral port
With :
  • addressPattern is at the a.b.c.d:port format. If either the IP address or the port part is absent or replaced with *, this means wildcard match.
  • operator is one of <=, >=, ==, ... To make this more convenient for use in unix shell, alphabetic FORTRAN-like notations le, gt, etc. are accepted as well.
  • Expressions can be combined with and, or and not, which can be abbreviated in C-style as &, &&, ...

sudo

Flags :

Flag Usage
-H --set-home change the value of the $HOME environment variable into the home directory of the target user (i.e. mostly root, so /root). Normally, using sudo does not alter $HOME (details)
bash -c 'echo $USER $HOME'; sudo bash -c 'echo $USER $HOME'; sudo -H bash -c 'echo $USER $HOME'
This can be the default behaviour, so the command above may not highlight anything.
-i simulate initial login. This runs the shell specified in /etc/passwd for the target user as a login shell. This means that login-specific resource files such as .profile or .login will be read by the shell.
If a command is specified, it is passed to the shell for execution. Otherwise, an interactive shell is executed. (Full details in man sudo)
-k ask sudo to forget the user's elevated permissions now rather than waiting for 15 minutes (details)
-l list the allowed and forbidden commands for the invoking user (or user specified with -U) on the current host.
-u kevin Run the specified command as the user kevin.

Example :

Chain several commands within a single sudo :

sudo bash -c 'whoami; date; echo Hello World'

strings

Usage :

Find printable strings in files.

Example :

strings /usr/bin/iplimit | grep Stop will output Stop that, lame flooder.

strace

Usage :

Trace system calls and signals
Each output line is made of 3 fields :
  1. the first entry on the left is the system call being performed
  2. the bit in the parentheses are the arguments to the system call
  3. the right side of the equals sign is the return value of the system call

Flags :

Flag Usage
-e raw=select Print raw, undecoded arguments for the specified set of system calls (here : select)
-o file Write the trace output to the file file rather than stderr
-p PID Attach to the process with the process ID PID

System calls :

Flag Usage
accept() When a request on a listening socket is refused / incomplete, accept() returns -1. Otherwise, it creates a new connected socket, and returns a new file descriptor referring to that socket.
fstat() Return metadata about a file (in the form of a "stat struct"). This file can be specified either by path for stat(), or by file descriptor for fstat().
recvfrom() Receive a message from a socket
select() Programs use select() to monitor file descriptors (specified as 2nd parameter) on a system for activity. select() will block for a configurable period of time waiting for activity on the supplied file descriptors, after which it returns with the number of descriptors on which activity took place.
This can be remembered as "wait for activity or timeout, and report where activity occured"

stat

Usage :

display file or file system status

Flags :

Flag Usage
-c --format use the specified format :
  • %x last access
  • %z last change
  • %n file name
  • %y last
  • %Y last

Example :

Get a file's last access, last change

stat -c 'LAST ACCESS:%x LAST CHANGE:%z FILE:%n' file

split

Usage :

Split files into pieces : split [options] fileToSplit prefixOfSplittedFiles

Flags :

Flag Usage
-a suffixLength Use suffixLength letters to form the suffix portion of the filenames of the split file.Make this suffix long enough so that there can be at least as many suffices as splits.
-l nbLines Specify the number of lines in each resulting file piece.

Example :

Split myBigFile into 1000 lines chunks :

split -l 1000 -a 3 myBigFile myBigFile_part_
This will create subfiles : myBigFile_part_aaa, myBigFile_part_aab, myBigFile_part_aac, ..., myBigFile_part_...

source (or .)

Usage :

source someFile (or . someFile) reads and executes commands from someFile in the current shell context.

What's specific with source ?

  • someFile is executed even though its execution bit is not set
  • someFile is executed within the current shell context, which allows :
    • loading variables into the current interactive shell session :
      1. source myConfigFile
      2. other commands using variables
    • using someFile as a configuration file for a script, without leaving variables in the shell environment once the script is over :
      sourcedFile=$(mktemp sourcedFile.XXXXXXXX); echo 'value=42' > "$sourcedFile"; chmod -x "$sourcedFile"; scriptFile=$(mktemp scriptFile.XXXXXXXX); echo "source $sourcedFile; echo 'sourced'; echo \"during the script : \\\$value='\$value'\"" > "$scriptFile"; chmod +x "$scriptFile"; ls -l "$sourcedFile" "$scriptFile"; cat "$sourcedFile" "$scriptFile"; ./"$scriptFile"; echo "after the script : \$value='$value'"; rm "$sourcedFile" "$scriptFile"
      value=42	cat "$sourcedFile"
      source sourcedFile.akr6paIl; echo 'sourced'; echo "during the script : \$value='$value'"	cat "$scriptFile"
      sourced		started running the script
      during the script : $value='42'	the variable exists in the script context
      after the script : $value=''	no value anymore
      Since a dedicated subshell (having its own context) is spawned when executing a script, if you want variables to survive for future scripts or commands after the script ends, you'll have to :
      • export variables within the script (details)
      • or source the script
  • The return status is the exit status of the last command executed from someFile, or zero if no commands are executed.

Trying to source a file having DOS line endings led the shell to complain for syntax errors on EVERY line. Consider converting line endings into DOS format.

About exported variables :

configFile='./script.conf'; scriptFile='./script.sh'; echo -e '#!/usr/bin/env bash\nvar1=value1\nexport var2=value2' > "$configFile"; echo -e '#!/usr/bin/env bash\necho "\tvar1 = $var1"\necho "\tvar2 = $var2"' > "$scriptFile"; echo -e "\n'source'd config file :"; cat "$configFile"; source "$configFile"; echo -e "\nCommand line (current shell context) :\n\tvar1 = $var1\n\tvar2 = $var2\n\nScript (subshell context) :"; bash "./$scriptFile"; rm "$configFile" "$scriptFile"
'source'd config file :
#!/usr/bin/env bash
var1=value1			not exported
export var2=value2

Command line (current shell context) :
	var1 = value1		exists 
	var2 = value2

Script (subshell context) :
	var1 =			unset 
	var2 = value2

sort

Usage :

sort assumes :

Flags :

Flag Usage Example
-h sort human numeric values echo -e "1M\n1G\n10K\n2K" | sort -h
2K
10K
1M
1G
-kn sort data based on the nth column

This assumes columns are whitespace or TAB-separated. Otherwise, consider -t

df -h | sort -r -k5
grep -c 'article id="' *ml | sort -t ':' -k 2 -nr | head -20
-n sort numerically. Default is alphabetically.
-o outputFile
--output=outputFile
Write result to outputFile instead of standard output
-r --reverse sort in reverse order (i.e. : descending).
Default sorting order is ascending.
echo -e "a\nb\nc" | sort -r
c
b
a
-R --random-sort shuffle, but group identical keys.
See shuf
echo -e "a\na\nb\nc" | sort -R
b
c
a
a
-t 'x'
--field-separator 'x'
specify the field separator when using -k. Default is whitespace or TAB sort -nr -t ':' -k3 /etc/passwd | head -10
-T directory
--temporary-directory=directory
use directory for temporaries, not $TMPDIR or /tmp. Multiple options specify multiple directories
-u sort unique : don't display duplicated lines. sort -u is equivalent to sort | uniq

Example :

Sort occurences of a URL from an Apache error log by decreasing order :

grep ' 500 ' 2012-07-03-apache-access.log | cut -d ' ' -f 11 | sort | uniq -c | sort -nr | less

All the magic is in the uniq -c prior to sorting.

Delete duplicate lines from myFile :

fileWithDuplicates='myFile'; tmpFile=$(mktemp); mv "$fileWithDuplicates" "$tmpFile"; sort -u "$tmpFile" -o "$fileWithDuplicates"; rm "$tmpFile"

snmpwalk

Usage :

snmpwalk -On -c snmpCommunity -v snmpVersion host OID

Flags :

Flag Usage
-O Output formatting options :
-On : displays the OID numerically

Example :

snmpwalk -On -c foo -v 2c 10.44.36.253 1.3.6.1.4.1.15497.1.1.1.11

shuf

Usage :

Shuffle the input rows

Flags :

Flag Usage
-e Consider every command line parameter as an input row
-i min-max
--input-range=min-max
Take numbers between min and max as input options to chose from
-n numLines Display at most numLines lines

Example :

Shuffle string parameters :

shuf -e A Z E R T Y may output Z R E A T Y or T R Y E Z A

Generate numberOfRandomNumbers random numbers within a specified range (source) :

shuf -i rangeMin-rangeMax -n numberOfRandomNumbers

shopt

Usage :

Toggle shell options

Flags :

Flag Usage
-s
-s optionName
List options that are set
set the option optionName
-u
-u optionName
List options that are uset
unset the option optionName

Option Usage
autocd a command name that is the name of a directory is executed as if it were the argument to cd
cdspell Autocorrect minor typos while using cd
dirspell Autocorrect minor typos during word completion on a directory name (provided the directory name has a trailing /)

setfacl

Usage :

Set/modify file ACL. This allows granting specific rights to specific users/groups on specific files, without setting global permissions. For instance, if Bob's home directory is :
drwx------ 64 bob developers 4,0K jan. 15 20:38 bob/
and we would like Alice to have read access to /home/bob/, we can :

Preliminary

Before using ACL, we must make sure that :
  1. the kernel supports them : zgrep POSIX_ACL /boot/config* (used to be /proc/config.gz), which may output :
    CONFIG_EXT2_FS_POSIX_ACL=y
    CONFIG_EXT3_FS_POSIX_ACL=y
    CONFIG_EXT4_FS_POSIX_ACL=y
    CONFIG_REISERFS_FS_POSIX_ACL=y
    CONFIG_JFS_POSIX_ACL=y
    CONFIG_XFS_POSIX_ACL=y
    CONFIG_BTRFS_FS_POSIX_ACL=y
    CONFIG_FS_POSIX_ACL=y
    CONFIG_TMPFS_POSIX_ACL=y
    CONFIG_JFFS2_FS_POSIX_ACL=y
    CONFIG_9P_FS_POSIX_ACL=y
  2. the filesystem we're about to play with is mounted with the acl flag (which enables support of the ACL. Just in case : noacl does the opposite).
    Have a look at /etc/fstab. (details : mount man page)

Command syntax :

setfacl -m permissions fileOrFolder
with permissions : prefix:userOrGroup:rights
  • prefix :
    • u: to change user rights
    • g: to change group rights
    • o: to change other rights. No need to specify userOrGroup
    • d:u: to declare default user rights
    • d:g: to declare default group rights
    • d:o: to declare default other rights
  • rights : specified with r, w, x or - (=not granted), exactly like in chmod.

Flags :

Flag Usage
-m modify an existing ACL entry
-R Recursive : apply rights to all files and directories. -R must be supplied before -m : -Rm
-x remove an ACL entry

Example :

Grant rw rights to a user on a single file :

setfacl -m u:alice:rw- file

Grant rw rights to a user on all files of a directory :

setfacl -Rm u:alice:rw- directory

Set default rights so that new files will inherit them :

setfacl -m d:u:alice:rw directory/

service

Usage :

Manage daemons

Example :

List status of all daemons :

service --status-all | less
This executes service serviceName status on all services, and returns :
  • [ + ] : service is running
  • [ - ] : service is stopped
  • [ ? ] : unknown / doesn't reply to the "service status" command

Manage a daemon :

service serviceName start|stop|restart|status