Bash Index : G - The 'G' Bash commands : description, flags and examples

getopts

Usage :

getopts is a shell built-in used to parse positional parameters (example)
getopts optstring name [args]

How it works :

  1. Each time it is invoked, getopts places the next option in the shell variable name, initializing name if it does not exist, and the index of the next argument to be processed into the variable OPTIND. OPTIND is initialized to 1 each time the shell or a shell script is invoked.
  2. When an option requires an argument, getopts places that argument into the variable OPTARG. The shell does not reset OPTIND automatically; it must be manually reset between multiple calls to getopts within the same shell invocation if a new set of parameters is to be used.
  3. When the end of options is encountered, getopts exits with a return value greater than zero.
In other words :
  1. getopts unstacks positional parameters one at a time (which is why it's usually within a while loop).
  2. once processed, each parameter value ends in the name variable, which explains the following case ... esac construct.
  3. when there are no more parameters to unstack, getopts returns a UNIX_FAILURE status code, causing the end of the while loop.

The option-string aka optstring (source) :

  • a leading colon (:) disables the verbose error handling
  • to specify a flag expects a value, just append it a colon (:). This value is mandatory and will cause an error message, unless silenced with a leading colon in the optstring (see preceding point)

Example :

Read flags only (no value) :

#!/usr/bin/env bash

while getopts ':a' opt; do
	case "$opt" in
		a)
			echo '-a was triggered!' >&2
			;;
		\?)
			echo "Invalid option: -$OPTARG" >&2
			;;
	esac
done
./test.sh -z
Invalid option: -z
./test.sh -a
-a was triggered!

Read flags and values :

#!/usr/bin/env bash

while getopts ':a:' opt; do
	case "$opt" in
		a)
			echo "Value of parameter '$opt' : '$OPTARG'"
			;;
		\?)
			echo "Invalid option: -$OPTARG" >&2
			;;
	esac
done
./test.sh -z
Invalid option: -z
./test.sh -a
(nothing)
./test.sh -a 12
Value of parameter 'a' : '12'
./test.sh -a 12 34
Value of parameter 'a' : '12'

gzip

Usage :

Compress data
You definitely should have a look at pigz :
  1. apt-get install pigz
  2. Create aliases :
    alias gzip='pigz'
    alias gunzip='unpigz'

Flags :

Flag Usage
-c --stdout --to-stdout Write output on standard output and keep original files unchanged.
gzip someFile
replaces someFile with its compressed counterpart someFile.gz. With -c, you may run :
gzip -c someFile > whatever
to have both the compressed and uncompressed versions of someFile.
gzip stores the original file name into the compressed file so that gunzip can recover it (check it with : file whatever). However, mixing gzip -c ... and gunzip ... (without -c) produces unexpected results.
regularFile=$(mktemp --tmpdir regular.XXXXXXXX); compressedFile=$(mktemp --tmpdir compressed.XXXXXXXX); for i in {1..100}; do echo "$i : Hello world" >> "$regularFile"; done; gzip -c "$regularFile" > "$compressedFile"; ls -lhU "$regularFile" "$compressedFile"; file "$compressedFile"; gunzip "$compressedFile"; rm "$regularFile" "$compressedFile"
-rw------- 1 bob users 1.7K Oct 29 11:18 /tmp/regular.JS4MvZYv
-rw------- 1 bob users	255 Oct 29 11:18 /tmp/compressed.af0tDooE
/tmp/compressed.af0tDooE: gzip compressed data, was "regular.JS4MvZYv", last modified: Sat Oct 29 11:18:41 2016, from Unix
gzip: /tmp/compressed.af0tDooE: unknown suffix -- ignored
--rsyncable Make the compressed file rsync-friendly.
gzip uses adaptative compression methods in which changing a single byte of the source data may cascade many changes in the compressed file. As a result, this would defeat Rsync's "send-only-changed-bytes" strategy. The --rsyncable disables the adaptative compression. (details)
"rsyncable" compressed files are less than 1% larger than non-rsyncable files
gunzip makes no difference between rsyncable / not rsyncable files.

groupadd

Usage :

Create a new group

Flags :

Flag Usage
-g gid Assigns the group ID gid for the new group.

Example :

groupadd -g 1234 newGroupName

grep

Usage :

Return lines matching a pattern
The word grep comes from the ed command g/re/p : "global regular expression print".
If grep returns :
Binary file someFile matches
  • It's pretty likely that someFile has "exotic" characters (i.e. non-ASCII characters such as ISO-8859 or Unicode)
  • The matching lines won't be returned. Consider -a to get them.
(Almost) only grep may return something like : Binary file ... matches. Check this with :
for binaryFile in /bin/* /usr/bin/*; do strings $binaryFile | grep -E 'Binary file.*matches' && echo -e "$binaryFile\n"; done
On my workstation, /bin/grep and usr/bin/emacs (and variants / symlinks) are listed.

Exit Status :

Code Condition Details
0 some match was found
1 no match found
2 an error occurred. Except if a match is found while errors are silenced by the quiet mode.

Flags :

Flag Usage
-An --after-context=n display the n lines after the one matching the pattern
-a --text process a binary file as if it were text (see note above)
-Bn --before-context=n display the n lines before the one matching the pattern
-b --byte-offset print the 0-based byte offset within the input file before each line of output
With -o : print the offset of the matching part itself.
-Cn --context-context=n equivalent to -Bn -An
-c --count suppress normal output; instead print a count of matching lines for each input file
--color=value highlight matched string. value can be :
  • auto : when output is redirected, the color information won't be passed
  • always : when output is redirected, the color information will also be passed
  • never : no highlighting
-d action --directories=action if the input is a directory, use action to process it. action can be :
  • skip : (explicit)
  • read : (default) read just as if they were ordinary files (and usually generate an error : grep: myDir: Is a directory)
  • recurse : go down into the directory, similar to -r
-e foo -e bar search multiple patterns : foo and bar
-E --extended-regexp consider the given pattern as an extended regular expression. grep -E is equivalent to egrep
  • When the regular expression contains a shell variable, it's generally easier to use an intermediate variable :
    • for searchedCharacter in {a..z}; do echo -e "foo\nbar\nbaz" | grep -E "$searchedCharacter"; done
    • for searchedCharacter in {a..z}; do regex="$searchedCharacter{2}\$"; echo -e "foo\nbar\nbaz" | grep -E "$regex"; done
  • grep -E is greedy (i.e. * will match the longest possible string). If you need a lazy star, consider -P :
    string='Hello world'; echo "$string" | grep -E 'e.*o'; echo "$string" | grep -P 'e.*?o'
    Hello world
    Hello world
--exclude-dir=pattern
  • without -r (directories to search are given on the command line) : skip directories with a name suffix matching pattern
  • with -r : skip directories with a base name matching pattern
name suffix
either the whole name, or any string between 2 /
base name
everything after the last /
pattern
can use *, ?, and [...] as wildcards, and \ to quote a wildcard or backslash character literally

 * with -r : skip directories with a base name matching pattern
AND
 * base name : everything after the last /

But, when doing :
	grep -r whatever /path/to/foo/bar/baz/* --exclude-dir=bar

there will be a point while recursing into directories (once in '/path/to/foo/bar') when 'bar' will be the base name, match the exclude clause, then ignore this directory and sub-directories.

AS A RESULT :
	grep -r whatever /foo/bar/baz/* --exclude-dir=bar
will actually exclude 'bar' anyway
-G --basic-regexp Interpret the pattern as a basic regular expression. This is the default.
-h --no-filename do not show the name of the file where a match is found. This is the default when grepping a single file
-H --with-filename print the name of the file where a match is found. This is the default when grepping several files
-i --ignore-case ignore case
-l --files-with-matches show only the name of files having a match. By default, grep outputs the file name and every matched line.
"list matching files"
-L --files-without-match show only the name of files having no match
-m n --max-count=n Stop reading input after n matching lines
grep -m n pattern is a good alternative to grep pattern | head -n
-n --line-number prefix each line of output with the line number within its input file
-o --only-matching show only the part of a matching line that matches the pattern
-P pattern
--perl-regexp pattern
consider pattern as a Perl regEx. This option is marked as experimental in the manual page.
Collides with egrep/-E (?)
-q --quiet --silent do not write anything to standard output (Read more).
-r --recursive
-R
read all files under each directory, recursively
only -R follows symlinks
-v --invert-match invert the sense of matching to select non-matching lines
-w word --word-regexp word return lines containing matches that form whole words. word matches either at the beginning/end of line, and between characters that are not letters, digits or the underscore.

Example :

Elegant hack for grep -v grep (source) :

ps aux | grep '[t]erminal'
This will match lines containing terminal, which grep '[t]erminal' does not.

Search a word in a file, then display 1 line before/after the matching line + line number

grep -A1 -B1 -n -i "expense" report.txt
ignoring case, extract every line from the file reports.txt containing the word "expense", display the lines before and after each match, and print the relevant line numbers.

Count occurrences of word within file (OK with 1 occurrence / line)

grep -c "word" file
this actually counts the number of lines containing "word" at least once

Count occurrences of word within file (OK with n occurrences / line)

grep -o "word" file | wc -l

Search and highlight non-ASCII characters within file (source) :

grep --color='auto' -P -n "[\x80-\xFF]" file

Search lines and environment matching a RegEx :

grep --color=auto -B 10 -P "Return code was : [^0][0-9]*" error.log

How to grep a TAB (source) :

echo -e "line 1\nline 2\n\tline 3 has a [TAB]\nline 4" | grep "CTRL-v TAB"
Ok in console, but do not use in scripts since CTRL-v TAB may be a shortcut of the text editor and produce unexpected results.
echo -e "line 1\nline 2\n\tline 3 has a [TAB]\nline 4" | grep -P "\t"
for GNU grep only
echo -e "line 1\nline 2\n\tline 3 has a [TAB]\nline 4" | grep $'\t'
Works only with single quotes (more about the $-escape syntax).
echo -e "line 1\nline 2\n\tline 3 has a [TAB]\nline 4" | awk '/\t/'
The is the most portable solution, works on all shells and platforms.

quiet mode and return codes :

Let's try some grep's :
echo -e 'a\nb\nc' | grep 'b'; echo $?
b
0
echo -e 'a\nb\nc' | grep -q 'b'; echo $?
0
echo -e 'a\nb\nc' | grep 'z'; echo $?
1
echo -e 'a\nb\nc' | grep -q 'z'; echo $?
1
echo -e 'a\nb\nc' | grep ; echo $?
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
2
echo -e 'a\nb\nc' | grep -q; echo $?
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
2
As a summary, both in normal and quiet modes :
  • result found + no error = 0
  • no result found + no error = 1
  • if error : 2

getfacl

Usage :

Read the ACL of a file :
getfacl file

Example :

getfacl can also be used in the context of a backup/restore of file permissions (source) :

  1. Backup : getfacl -R /path/to/dirContainingFilesWithPermissionsToBackup > /path/to/permissions.txt
  2. Play with file permissions
  3. Restore : setfacl --restore=/path/to/permissions.txt

groups

Usage :

Display a user's group memberships

Example :

groups bob
bob : bob adm dialout cdrom plugdev lpadmin admin sambashare

getent

Usage :

getent lets admins get entries in a number of important text files called databases. This includes the passwd and group databases which store user information.

Since getent uses the same name service as the system, it will show all information, including that gained from network information sources such as LDAP.

The databases it searches in are :

man getent :
NAME
	getent - get entries from Name Service Switch libraries

SYNOPSIS
	getent [option]... database key...

DESCRIPTION
	The getent command displays entries from databases supported by the Name Service Switch libraries, which are con-
	figured in /etc/nsswitch.conf. If one or more key arguments are provided, then only the entries that match the
	supplied keys will be displayed. Otherwise, if no key is provided, all entries will be displayed (unless the data-
	base does not support enumeration).



cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:	compat sss
group:	compat sss
shadow:	compat sss

hosts:		files dns
networks:	files

protocols:	db files
services:	db files
ethers:		db files
rpc:		db files

netgroup:	nis



man 5 nsswitch.conf
DESCRIPTION
	The Name Service Switch (NSS) configuration file, /etc/nsswitch.conf, is used by the GNU C Library to determine the
	sources from which to obtain name-service information in a range of categories, and in what order. Each category
	of information is identified by a database name.

format :
	databaseName:	source1	source2	...	sourceN

sources are queries in the order there are declared :
passwd:		compat sss
	1. 'compat'
	2. then 'sss'


Compatibility mode (compat)
	The NSS "compat" service is similar to "files" except that it additionally permits special entries in corresponding
	files for granting users or members of netgroups access to the system.


https://wiki.ubuntu.com/Enterprise/Authentication/sssd

Exit Status :

Code Condition Details
0 Command completed successfully
2 No result found

Example :

Get information about a user account :

  • grep ^root: /etc/passwd
  • getent passwd root
Both return :
root:x:0:0:root:/root:/bin/bash
  • The ^ and : in the grep command are there to make sure we hit the username field of /etc/passwd . getent is much less error-prone.
  • In the getent ... command line above, passwd refers to the passwd database.

Test whether a group exists :

  • grep ^myGroup: /etc/group
  • getent group myGroup
If myGroup exists, getent returns a Unix success and displays :
myGroup:x:myGroupId:joe,jack,william,averell
Otherwise, it displays nothing and returns a Unix failure.