I like Bash scripting - Scripting's my favorite (HowTo's)

Files can be converted with utilities :

• dos2unix / unix2dos : these utilities are usually installed by default on Red Hat systems
• tofrodos : this is available as a Debian package (man page)

Convert files from DOS to Unix :

pinCodeLength=10; pinCode=''; while [ "${#pinCode}" -le "$pinCodeLength" ]; do pinCode+=$RANDOM; done; echo "$pinCode" | head -c "$pinCodeLength"

My favorite fruit is :
apples (love the red ones ;-)

On the back of my phone, you can see an...
apple-shaped logo.

New-York is nicknamed :
the "big apple" since the 1970s.

My favorite fruit is :
apples (love the red ones ;-)

On the back of my phone, you can see an...
android-shaped logo.

New-York is nicknamed :
the "big apple" since the 1970s.

BEGIN				{ matchingLineNumber=-1 };		just initializing a variable
/phone/				{ matchingLineNumber=NR };		upon finding our needle, just set this variable with the current line number
NR == matchingLineNumber+1	{ gsub(/apple/, "android", $0) };	on the next line, do the substitution
				{ print }				and print the current line

It is sadly frequent that we have to dive into old + cluttered + hardly readable code, that is not even properly indented (or not indented at all). One of the first things to do, then, is to improve readability with proper indentation. This won't magically turn lead into gold, but it helps anyway. Especially if it can be done effortlessly .

a ( b
c d [ e ] f [
g h { j (
k ) } l m
n ] o ) p
q r

a ( b
	c d [ e ] f [
		g h { j (
				k ) } l m
		n ] o ) p
q r

drwxr-xr-x	root	root		/
drwxr-xr-x	root	root		/this/
drwxr-xr-x	bob	developers	/this/is/
drwxr-xr-x	bob	developers	/this/is/a/
drwxr-xr-x	bob	developers	/this/is/a/very/
drwx------	bob	developers	/this/is/a/very/long/
drwx------	bob	developers	/this/is/a/very/long/pathTo/
-rw-------	bob	developers	/this/is/a/very/long/pathTo/myFile

Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Etiam mollis viverra ligula,
Lorem ipsum dolor sit amet,
ut luctus magna imperdiet eget.
Ut consectetur laoreet venenatis.
Nulla euismod sapien nec sodales tempor.
Lorem ipsum dolor sit amet,
Suspendisse sagittis odio eu urna imperdiet,
vitae sollicitudin ante mattis.

Same as above, with contents stored in a file :

not commented
/*comment part 1
comment part 2
comment part 3*/ but I want to keep the end of this line
not commented either

tr | sed | tr method :

Other (better !) method (inspired by) :

3

• DON'T :
  • 'foo\'bar'
  • ... (there are plenty of other creative ways of failing at this )
• DO : 'foo'"'"'bar'

Explanation of how '"'"' is interpreted as just ' :

 aaa  b  ccc
'foo'"'"'bar'
^   ^^^^^   ^
1   23456   7

The good ol'method :

hello world

The += method :

• stat -c %y someFile
  2018-11-21 10:37:05.683199311 +0100
• date -r someFile
  Wed Nov 21 10:37:05 CET 2018
• date -r someFile +%F_%H-%M-%S
  2018-11-21_10-37-05

Solution 1 (badness=1) :

command1 > "/path/to/logFile" 2>&1
command2 >> "/path/to/logFile" 2>>&1
command3 >> "/path/to/logFile" 2>>&1

Solution 2 (badness=100) :

(
command1
command2
command3
) > "/path/to/logFile" 2>&1

jump to the solution

• more about compound commands
• more about output redirection
• more about exec

exec can be used to redirect all the outputs. If, at some point of the script, we want to "stop redirecting the outputs", we have to (source) :

Named file descriptors (aka "automatic file descriptor allocation")

#!/usr/bin/env bash

outFile=$(mktemp)

echo 'foo'

exec {fileDescriptor}>&1 1>"$outFile"
echo 'bar'

exec 1>&${fileDescriptor} {fileDescriptor}>&-
echo 'baz'

cat "$outFile"
rm "$outFile"
echo "The chosen file descriptor was : '$fileDescriptor'"

foo
baz
bar
The chosen file descriptor was : '10'

Snippet to copy-paste :

exec {previousStdout}>&1 {previousStderr}>&2 1>"/path/to/logFile" 2>&1

(script content goes here)

exec 2>&${previousStderr} {previousStderr}>&- 1>&${previousStdout} {previousStdout}>&-

• myScript.sh is doing its job fine
• I'd like it to have a slightly different behavior depending on whether it's been fired
  • interactively : i.e. from a terminal
  • non-interactively : by at or cron

• with $PS1
• with $-
• with [ -t fd ]

#!/usr/bin/env bash
#	To fire this script via 'at' :
#	at $(date --date "now +1 minutes" '+%H%M') -f myScript.sh

exec 1> output.txt

########################################## ##########################################################

# source
[ -z "$PS1" ] && interactive='no' || interactive='yes'; echo "interactive 1 : '$interactive'"

# when run :
#	manually			==>	yes
# 	in a script			==>	no
# 	in a script fired by 'at'	==>	yes
# 	in a script fired by 'cron'	==>	no


########################################## ##########################################################

# source
case $- in *i*) interactive='yes' ;; *) interactive='no' ;; esac; echo "interactive 2 : '$interactive'"

# when run :
#	manually			==>	yes
# 	in a script			==>	no
# 	in a script fired by 'at'	==>	no
# 	in a script fired by 'cron'	==>	no


########################################## ##########################################################

echo "dollarDash : '$-'"
# when run :
#	manually			==>	himBHs
# 	in a script			==>	hB
# 	in a script fired by 'at'	==>	s
# 	in a script fired by 'cron'	==>	hB

• set a variable in the crontab and check it in your script (source)
• investigate the process tree until you find (or not) cron's PID (source)

1. Append these functions to ~/.bash_aliases :

   getFileSnippet() {
   	local fileToInspect=$1
   	local startLine=$2
   	local stopLine=$3
   	tmpFile=$(mktemp --tmpdir='/run/shm' tmp.XXXXXXXX)
   	sed -n "$startLine,${stopLine}p" "$fileToInspect" > "$tmpFile"
   	echo "$tmpFile"
   	}
   
   compareSnippets() {
   	[ $# -ne 6 ] && { echo 'Wrong number of arguments, 6 expected.'; return $UNIX_FAILURE; }
   	local fileToInspect1=$1
   	local startLine1=$2
   	local stopLine1=$3
   	local fileToInspect2=$4
   	local startLine2=$5
   	local stopLine2=$6
   
   	for argumentToCheck in fileToInspect1 fileToInspect2; do
   		[ -f "${!argumentToCheck}" ] || { echo "Argument '${!argumentToCheck}' is not a file."; return $UNIX_FAILURE; }
   	done
   
   	snippet1=$(getFileSnippet "$fileToInspect1" "$startLine1" "$stopLine1")
   	snippet2=$(getFileSnippet "$fileToInspect2" "$startLine2" "$stopLine2")
   	diff "$snippet1" "$snippet2"
   	rm "$snippet1" "$snippet2"
   	}

2. compare code snippets with :
   compareSnippets fileA 117 124 fileB 159 166

foo1
bar1
foo2 REMOVE THIS LINE AND THE FOLLOWING ONE
bar2
foo3
bar3

foo1
bar1
foo3
bar3

echo -e 'foo1\nbar1\nfoo2 REMOVE THIS LINE AND THE FOLLOWING ONE\nbar2\nfoo3\nbar3' | awk 'BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}'

• it's made of ;-separated rules. Each rule is evaluated separately against each input line
• rules are formatted like : condition {action}. condition can be a regular expression match or a variable check

BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}

BEGIN				{ matchingLineNumber = -1 }
/REMOVE/			{ matchingLineNumber = NR; next }
NR == matchingLineNumber + 1	{ next }
				{ print }

• line 1 : just initializing a variable. Setting it to 0 would make our 3^rd awk line affect the line 1 of the input. Hence the -1
• line 2 : if the current input line matches the 'REMOVE' regexp, keep the line number, stop processing rules and continue to the next input line
• line 3 : if the current input line directly follows a line matching the regexp, stop processing rules and continue to the next input line
• line 4 : no condition, just print the current input line

It works whatever the position of the matching line within the input :

echo -e 'foo2 REMOVE THIS LINE AND THE FOLLOWING ONE\nbar2\nfoo1\nbar1\nfoo3\nbar3' | awk 'BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}'

echo -e 'foo1\nbar1\nfoo3\nbar3\nfoo2 REMOVE THIS LINE AND THE FOLLOWING ONE\nbar2' | awk 'BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}'

• echo 'a b' | awk '{print $2" "$1}'
• echo 'a b' | sed -r 's/(.+) +(.+)/\2 \1/'

myVariable=$(echo {a..c} | tr ' ' '\n'); echo "myVariable : '$myVariable'"

myVariable : 'a
b			definitely a multiline variable
c'

echo "$myVariable" | while read aSingleLineOfMyVariable; do
	echo "a single line : '$aSingleLineOfMyVariable'"
done; echo "last value : '$aSingleLineOfMyVariable'"

a single line : 'a'	works fine inside the loop
a single line : 'b'
a single line : 'c'
last value : ''		undefined variable since it only exists in the subshell created while piping

tmpFile=$(mktemp); echo "$myVariable" > "$tmpFile"; while read aSingleLineOfMyVariable; do
	echo "a single line : '$aSingleLineOfMyVariable'"
done < "$tmpFile"; echo "last value : '$aSingleLineOfMyVariable'"; rm "$tmpFile"

a single line : 'a'
a single line : 'b'
a single line : 'c'
last value : ''		$aSingleLineOfMyVariable is lost when leaving the loop

This does the job but :

• creating a temporary file is not very elegant
• creating a temporary file may have a performance cost if repeated numerous times
• may require write permissions to /tmp
• variables created within the loop are local to the loop

while IFS= read aSingleLineOfMyVariable; do
	echo "a single line : '$aSingleLineOfMyVariable'"
done < <(printf '%s\n' "$myVariable"); echo "last value : '$aSingleLineOfMyVariable'"

a single line : 'a'
a single line : 'b'
a single line : 'c'
last value : ''		$aSingleLineOfMyVariable is lost again, continue reading

Whatever command is (including something like echo "$someVariable"), the <(command) construct (aka process substitution) expands to a file and, as such, can be fed into anything with < or >.

previousIfs="$IFS"; IFS=$'\n'; for aSingleLineOfMyVariable in $myVariable; do
	echo "a single line : '$aSingleLineOfMyVariable'"
done; echo "last value : '$aSingleLineOfMyVariable'"; IFS="$previousIfs"

a single line : 'a'
a single line : 'b'
a single line : 'c'
last value : 'c'	$aSingleLineOfMyVariable is not lost this time 

This comment says this is because while loops create a subshell, whereas for loops don't. I'm afraid this is wrong... (it is !)

Let's check this :

. ./functions.sh

# Include an external file even though the current script is not launched from its own directory
directoryOfThisScript="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
. "$directoryOfThisScript/functions.sh"

• directoryOfThisScript=$(dirname "$0")
• directoryOfThisScript="${0%/*}" (roughly same as above, with a "manual" approach, source)

• for a script named script.sh
• several files to "source"
• a configuration file named script.sh.conf
• and error management

######################################### includes ##################################################
nameOfThisScript=$(basename "${BASH_SOURCE[0]}")
directoryOfThisScript="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

configFile="$directoryOfThisScript/$nameOfThisScript.conf"
functionsFile="$directoryOfThisScript/functions.sh"

for fileToSource in "$configFile" "$functionsFile"; do
	source "$fileToSource" 2>/dev/null || {
		echo "File '$fileToSource' not found"
		exit 1
		}
done
######################################### /includes #################################################

return + $?

For integers only (and "not too big" ones, preferably). Actually, the returned number will be n modulo 256 :

foo() { return $i; }; for i in {254..258}; do foo $i; echo "$i $?"; done

254 254
255 255
256 0
257 1
258 2

echo + $()

For anything you like : numbers, strings, ...

Let's imagine a command returned 10 lines of text, and we want to display only the 1^st, 3^rd and 8^th lines.

With grep : let's ask grep to add line numbers, then select the lines to display (again with grep), then hide the line numbers :

for i in {a..j}; do echo "Line $i"; done | grep -n '' | grep -E '^(1|3|8):' | sed -r 's/^[0-9]+://'

Even simpler, with sed :

for i in {a..j}; do echo "Line $i"; done | sed -n '1p; 3p; 8p'

• myString='0123456789'
• n=3. This does not always come easy in the command lines : sometimes it has to change to "length of string - 3", or to be incremented / decremented because the "start" character is sometimes included / sometimes excluded. Or both . These commands are not the most convenient ones and should be avoided in real-life situations, but here they are so that I'll stop wondering "Could I do that with ... ?"

UNIX_SUCCESS=0
UNIX_FAILURE=1

isInteger() {
	[[ $1 =~ ^[-+]?[0-9]+$ ]] && echo $UNIX_SUCCESS || echo $UNIX_FAILURE
	}

[ $(isInteger "$myVariable") -eq "$UNIX_FAILURE" ] && {
	(shout how unhappy you feel)
	}

myString='hello'
length=$((${#myString}-1))
for i in $(eval echo "{0..$length}"); do
	echo $i : ${myString:i:1}
done

myString='hello'
for((i=0; i<${#myString}; i++)); do
	echo $i : ${myString:i:1}
done

Use eval.

If you want to store the result of a command (but not the command itself) into a variable, just use the $(command) construct :

• With a for loop
• With a case construct
• With a regular expression

With a for loop :

#!/usr/bin/env bash

valueToCheck="$1"
listOfAcceptedValues='foo bar baz'

UNIX_SUCCESS=0
UNIX_FAILURE=1
valueToCheckIsInTheList=$UNIX_FAILURE

for value in $listOfAcceptedValues; do
	echo "Testing '$valueToCheck' against '$value'"
	[ "$valueToCheck" == "$value" ] && { echo "'$valueToCheck' is a valid value."; valueToCheckIsInTheList=$UNIX_SUCCESS; break; }
done

[ "$valueToCheckIsInTheList" -eq "$UNIX_FAILURE" ] && { echo "'$valueToCheck' is not a valid value."; exit $UNIX_FAILURE; }

With a case construct :

#!/usr/bin/env bash

valueToCheck="$1"

case "$valueToCheck" in
	'foo')
		echo "'$valueToCheck' is a valid value."
		# do something with 'foo'
		;;
	'bar')
		echo "'$valueToCheck' is a valid value."
		# do something different with 'bar'
		;;
	'baz')
		echo "'$valueToCheck' is a valid value."
		# do something different again with 'baz'
		;;
	*)
		echo "'$valueToCheck' is not a valid value."
		# deal with it !!!
		;;
esac

With a regular expression :

#!/usr/bin/env bash

listOfAcceptedValues='foo bar baz'
listOfValuesToCheck="foo bar baz poo 123 bam ofo fo f ''"

for valueToCheck in $listOfValuesToCheck; do
	[[ "$listOfAcceptedValues" =~ "$valueToCheck" ]] && result1='' || result1=' not'
	[[ "$listOfAcceptedValues" =~ (^|[[:space:]])"$valueToCheck"($|[[:space:]]) ]] && result2='' || result2=' not'
	echo -e "'$valueToCheck'\tis : (1)$result1 a valid value, \t(2)$result2 a valid value."
done

'foo'	is : (1) a valid value,		(2) a valid value.
'bar'	is : (1) a valid value,		(2) a valid value.
'baz'	is : (1) a valid value,		(2) a valid value.
'poo'	is : (1) not a valid value,	(2) not a valid value.
'123'	is : (1) not a valid value,	(2) not a valid value.
'bam'	is : (1) not a valid value,	(2) not a valid value.
'ofo'	is : (1) not a valid value,	(2) not a valid value.
'fo'	is : (1) a valid value,		(2) not a valid value.
'f'	is : (1) a valid value,		(2) not a valid value.
''''	is : (1) not a valid value,	(2) not a valid value.

'alice': NOPE
'bob' is valid
'bobby': NOPE

Other methods :

Repeat a single character :

Repeat a string :

ABCD ABCD ABCD ABCD[SPACE]

ABCD ABCD ABCD ABCD[SPACE]

Generic case :

#!/usr/bin/env bash

answerIsValid=''

while [ -z "$answerIsValid" ]; do
	echo "Continue ? [yn]"
	read answer
	[[ "$answer" == [yYnN] ]] && answerIsValid=1 || echo -e "Invalid answer\n"
done

The list of characters can also be provided as a variable :

#!/usr/bin/env bash

answerIsValid=''
validCharacters='yYnN\[\]'

while [ -z "$answerIsValid" ]; do
	echo "Continue ? [yn]"
	read answer
	[[ "$answer" == [$validCharacters] ]] && answerIsValid=1 || echo -e "Invalid answer\n"
done

The basics :

Ready-to-use one-liners :

Lorem
ipsum
dolor
sit
amet,
...

0 Lorem
1 ipsum
2 dolor
3 sit
4 amet,
5 ...

A word of warning :

1
2
3
4

3	number of fruits
5	length of string apple

0 Lorem
1 ipsum
2 dolor
3 sit
4 amet
Length : 5

0 Lorem
1 ipsum
2 dolor
4 amet	no more myArray[3]
Length : 4

Numbers listed in a file :

Numbers returned to STDOUT :

nbExpectedArgs=1
if [ $# -ne $nbExpectedArgs ]; then
	echo "Usage: $(basename $0) <argument>"
	exit 1
fi

#!/usr/bin/env bash

nbExpectedArgs=3

usage() {
	cat <<-EOF
	Usage: $(basename $0) <argument1> <argument2> <argument3>
	blah
	blah
	blah
	EOF
	}

if [ $# -ne $nbExpectedArgs ]; then
	usage
	exit 1
fi

The long error message is displayed with cat rather than echo.