I like Bash scripting - Scripting's my favorite (HowTo's)

mail

How to convert a string to lowercase / uppercase ?

mail

How to loop over a long list of values ?

Situation

Just consider this snippet :
for word in Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut gravida dolor ornare, bibendum nunc eu, ornare nisl. Integer ut diam est.; do
	echo "$word"
done
It works fine but lacks readability, especially as the list grows longer.

Details

This _may_ help :
while read word; do
	echo "$word"
done < <(cat <<-EOLIST
Lorem
ipsum
dolor
sit
amet,
consectetur
adipiscing
elit.
Ut
gravida
dolor
ornare,
bibendum
nunc
eu,
ornare
nisl.
Integer
ut
diam
est.
EOLIST
)
but has the same readability issue when the list gets loooong .

Solution

  1. store values in a dedicated file : values.txt
  2. while read word; do
    	echo "$word"
    done < values.txt
mail

How to declare a multi-line string variable ?

The basic method

  • Let's try basic things first, it _could_ work :
    myVariable='line 1
    line 2
    line 3'; echo $myVariable
    line 1 line 2 line 3
  • Same with a slight change :
    myVariable='line 1
    line 2
    line 3'; echo "$myVariable"
    line 1
    line 2
    line 3
    The magic is not in the variable itself, but in what you instruct echo to do with it.

An advanced method

This method is said to "better" support characters like quotes / special characters, but looks like it doesn't support hardcore things like that :
That long string was made with :
	pwgen -ys 64 --num-passwords=4

myVariable='dtIqDWwwA{[o>,%M:p;zW>ri$wdYM&9`:U%juyA1kB&lk"Hu<*05]|c=I?6~@0`h
Y\]DzR2{S=oO:m}@3_G}[6bc+g&{N;L1)MLKr1U9$+HIh}J{=bs%WM60i_Vq'QK"
wiy~Pg^?,A7ISC]t[UX`'*B%Nt~F3Qo,Hon1JPD9hdJ{|DOYQ%2s`nUJ;w*Ra\el
r1'CMx%*{X}:rG@C94V)0uczM#8Vh08`L@mR:Yv5|Od/[<\+$6M_J*5Gi$3)C2yf'; echo "$myVariable"

read -d '' myVariable << EOF; echo $myVariable
dtIqDWwwA{[o>,%M:p;zW>ri$wdYM&9`:U%juyA1kB&lk"Hu<*05]|c=I?6~@0`h
Y\]DzR2{S=oO:m}@3_G}[6bc+g&{N;L1)MLKr1U9$+HIh}J{=bs%WM60i_Vq'QK"
wiy~Pg^?,A7ISC]t[UX`'*B%Nt~F3Qo,Hon1JPD9hdJ{|DOYQ%2s`nUJ;w*Ra\el
r1'CMx%*{X}:rG@C94V)0uczM#8Vh08`L@mR:Yv5|Od/[<\+$6M_J*5Gi$3)C2yf
EOF
But it works fine with "normal text" :
read -d '' myVariable << EOF; echo "$myVariable"
A SQL query goes into a bar,
walks up to two tables and asks,
"Can I join you?"
EOF
A SQL query goes into a bar,
walks up to two tables and asks,
"Can I join you?"
This construct comes with a limitation : read returns 1 as its exit code, which will interrupt code running within set -e
read -d '' myVariable << EOF; echo $?; echo "$myVariable"
Hello world
EOF
1
Hello world
With set -e :
set -e; read -d '' myVariable << EOF; echo $?; echo "$myVariable"
Hello world
EOF
The script / current shell exits.
Workaround :
set -e; read -d '' myVariable << EOF || true; echo $?; echo "$myVariable"
Hello world
EOF
0
Hello world

A special case

Use this whenever you'd like a variable :
  • to be declared on several lines
    • for readability
    • to avoid the 80-column limit
  • but to be a single long line anyway
myVariable="A SQL query goes into a bar, \
walks up to two tables and asks, \
\"Can I join you?\""; echo "$myVariable"; echo $myVariable
A SQL query goes into a bar, walks up to two tables and asks, "Can I join you?"			echo works fine with 
A SQL query goes into a bar, walks up to two tables and asks, "Can I join you?"			 or without quotes
mail

How to increment a variable ?

All these commands do the job :
So far, no idea which is "better" or should be preferred —if any.
mail

How to write script logs to stdout and to stderr and to myScript.log ?

Situation

Solution

solution details pro con
use tee at launch launch the script like this :
./myScript.sh | tee myScript.log
  • short & simple
  • if you forget the tee part :
    • no logs
    • or you have to kill + relaunch the script
use redirections
and tee in the script
replace all script lines :
someCommand [arguments]
with :
someCommand [arguments] 2>&1 | tee myScript.log
  • does the job
  • but only for very short scripts
  • ugly
  • unreadable on anything longer than a one-liner
  • does not meet my minimal changes requirement
add a redirection at the
beginning of the script
Add to the beginning of the script :
#!/usr/bin/env bash

exec 1>myScript.log 2>&1

[script code]
  • meets my minimal changes requirement
  • effectively writes everything to myScript.log
  • but nothing appears in the console, albeit requested
redirection + tee hack
(source)
Add to the beginning of the script :
#!/usr/bin/env bash

exec > >(tee myScript.log) 2>&1	see breakdown below

[script code]
  • does the job
  • meets all my requirements
  • will look like magic to many (including myself in a few weeks)
  • I found a limitation

About this exec > >(tee ) command :

Breakdown

What does this command do ?

exec > >(tee myScript.log) 2>&1

exec
  • exec alters file descriptors
  • the exec > someFile 2>&1 construct redirects all outputs of a script (stdout + stderr) into someFile :
tee myScript.log
  1. tee reads data on its standard input
  2. sends that data to :
    • its standard output
    • and to the myScript.log file
>()
this is a process substitution
the whole command :
STDOUT + STDERR          =====>          tee          ==+==>          write everything
of all commands                                         |             to logfile
of my script                                            |
                                                        |
                                                        +==>          send everything
                                                                      to console

Limitation

  • I made a script like this MWE :
    #!/usr/bin/env bash
    
    doThings() {
    	echo "wait $1 second"
    	echo 'ERROR' >&2
    	sleep $1
    	echo "done (wait $1)"
    	}
    
    main() {
    	doThings 0.25s &
    	doThings 0.5s &
    	wait
    	echo 'the end'
    	}
    
    main
    wait 0.25s second
    wait 0.5s second
    ERROR
    done (wait 0.25s)
    ERROR
    done (wait 0.5s)
    the end
    This logs nothing so far, but apart from this, everything is going extremely well.
  • If I add this exec > >(tee ) hack to the script :
    #!/usr/bin/env bash
    
    exec > >(tee test.log) 2>&1		just added this line
    
    doThings() {
    	echo "wait $1 second"
    	echo 'ERROR' >&2
    	sleep $1
    	echo "done (wait $1)"
    	}
    
    main() {
    	doThings 0.25s &
    	doThings 0.5s &
    	wait
    	echo 'the end'
    	}
    
    main
    wait 0.25s second
    ERROR
    wait 0.5s second
    ERROR
    done (wait 0.25s)
    done (wait 0.5s)		execution is stuck here forever
    
    
    • this looks related to the wait : once commented, the script runs until the end, but the output becomes inconsistent :
      the end
      wait 0.25s second
      wait 0.5s second
      done (wait 0.25s)
      done (wait 0.5s)
    • I think tee itself is seen as a background command, and wait is waiting for it to end, which won't happen while the script is still executing. Hence waiting forever.
    • tee staying alive looks like a known question (but no solution in these links )
  • Workaround :
    #!/usr/bin/env bash
    
    doThings() {
    	echo "wait $1 second"
    	echo 'ERROR' >&2
    	sleep $1
    	echo "done (wait $1)"
    	}
    
    main() {
    	doThings 0.25s &
    	doThings 0.5s &
    	wait
    	echo 'the end'
    	}
    
    main 2>&1 | tee test.log		use tee directly on main()
    
    echo '-------------log--------------'	this is only to
    cat test.log				show that output
    echo '------------/log--------------'	and log are identical
    wait 0.25s second
    ERROR
    wait 0.5s second
    ERROR
    done (wait 0.25s)
    done (wait 0.5s)
    the end
    -------------log--------------
    wait 0.25s second
    ERROR
    wait 0.5s second
    ERROR
    done (wait 0.25s)
    done (wait 0.5s)
    the end
    ------------/log--------------
mail

How to stop processing the input after reading n lines ?

Let's write 10 million numbers and play with the 10 first ones only !

With grep :

number=100000000; time (seq $number | grep -E '^.$'); time (seq $number | grep -E -m 9 '^.$')
1

9

real    0m1.035s
user    0m1.489s
sys     0m0.212s
1

9

real    0m0.001s
user    0m0.002s
sys     0m0.000s
Both commands :
  1. write numbers from 1 to 100000000
  2. output single-digit numbers
  3. but while the 2nd command stops after finding the 9, the 1st command keeps reading the input until the end
While sed (read below) actually stops working after reading n lines of input, grep stops after finding n matching lines. If we specify -m 10 instead of -m 9 in the grep command above, grep will read the input until the end, searching for a 10th match that doesn't exist.

With sed (inspired by) :

number=10000000; time (seq $number | sed -rn 's/^.$/X/'); time (seq $number | sed -rn 's/^.$/X/;10q')
real    0m0.556s
user    0m0.599s
sys     0m0.049s

real    0m0.001s
user    0m0.002s
sys     0m0.000s
Both commands :
  1. write numbers from 1 to 10000000
  2. replace single-digit numbers with X
  3. but while the 2nd command stops replacements after reading 10, the 1st command keeps reading the input until the end

With awk :

number=10000000; time (seq $number | awk '$0 < 10'); time (seq $number | awk '$0 < 10; $0 > 10 {exit}')
1

9

real    0m1.396s
user    0m1.455s
sys     0m0.020s
1

9

real    0m0.002s
user    0m0.002s
sys     0m0.000s
Like previous examples :
  • the 1st command checks ALL lines of input
  • while the 2nd doesn't care once numbers are > 10

alternate solution :

number=10000000; time (seq $number | awk '$0 < 10'); time (seq $number | awk 'NR >= 10 {exit}; {print}')
mail

How to keep / remove lines of text from start / until end of file, between tokens, starting at token / line x and for the next n lines ?

If the input is made of a giant single line (concatenated/minified HTML/CSS/...) such as :
blablabla<startToken><stuffIWantToKeepOrRemove><endToken>blablabla
A workaround would be to :
  1. replace <startToken> with \n<startToken>\n
  2. do the same with <endToken>, so that the input becomes :
    blablabla
    <startToken>
    <stuffIWantToKeepOrRemove>
    <endToken>
    blablabla
  3. once tokens are shun on distinct lines, you may apply recipes below
from ... until ... keep remove
start of file token
token end of file
token token
startToken='Line 3'; endToken='Line 7'
for i in {0..9}; do echo "Line $i"; done | \
awk -v ST="$startToken" -v ET="$endToken" '
	BEGIN			{ betweenTokens=0 }
	$0 ~ ST		{ betweenTokens=1 }
	$0 ~ ET		{ betweenTokens=0 }
	betweenTokens == 1	{ print }'
to include the "end token" line in the results, swap the $0 ~ ET and betweenTokens == 1 lines
startToken='Line 3'; endToken='Line 7'
for i in {0..9}; do echo "Line $i"; done | \
awk -v ST="$startToken" -v ET="$endToken" '
	BEGIN			{ betweenTokens=0 }
	$0 ~ ST		{ betweenTokens=1 }
	$0 ~ ET		{ betweenTokens=0 }
	betweenTokens == 1	{ next }
				{ print }'
token token + n lines
startToken='Line 3'; nbLinesToDelete=4
for i in {0..9}; do echo "Line $i"; done | \
awk -v ST="$startToken" -v N="$nbLinesToDelete" '
	BEGIN			{ matchingLineNumber = -9999 }
	$0 ~ ST		{ matchingLineNumber = NR; next }
	NR > matchingLineNumber && NR < (matchingLineNumber + N) \
				{ next }
				{ print }'
  • removes n lines including the line of the token
  • to remove the line of the token AND the next n lines, change the highlighted < into <=

About the -9999 initialization value :

  • let's call y this initialization value
  • the NR > && NR < line becomes : if (NR > y) and (NR < y+N) then "remove line"
  • NR starts at 1 (NR > 0), so NR > y for all values of y such as y < 1
  • as for NR < y+N :
    • with y=0 :  true if NR < N
    • with y=-1 : true if NR < N-1
    • with y=-2 : true if NR < N-2
  • meaning : lines from the start of the input until the (N+y)th are also removed (by mistake )
  • any value of y is safe until NR < y+N is true again
    • considering NR > 0, this becomes : y+N > NR > 0
    • then y+N > 1, and so N > 1-y
    • -9999 works until we want to delete a block of 10000 lines
mail

How to match / grep a TAB ?

mail

How to handle unset variables when the -u / nounset flag is raised ?

Situation

Solution

Define a default value :
set -u; unset a; b=${a:-}; [ "$b" ] || echo 'a is unset'; [ -z "$b" ] && echo 'a is unset'; [ -n "$b" ] || echo 'a is unset'
a is unset
a is unset
a is unset
Now you can catch the error and handle it yourself the way you like.
mail

How to trim a line to a specific length ?

mail

How to feed several variables at once ?

Let's consider :

copy-paste-edit solution

output=$(echo 'value1 value2 value3')		intermediate variable because the command is supposed to be run only once
myVar1=$(echo "$output" | cut -d' ' -f1)
myVar2=$(echo "$output" | cut -d' ' -f2)
myVar3=$(echo "$output" | cut -d' ' -f3)

echo "$myVar1 $myVar2 $myVar3"
value1 value2 value3

alternate solution

read myVar1 myVar2 myVar3 < <(echo 'value1 value2 value3')

echo "$myVar1 $myVar2 $myVar3"
value1 value2 value3

What if values are not SPACE-separated and the output has many lines and , i.e. not-so-basic situation

  • The not-so-basic command that generates a multiline output with values having a leading text :
    echo -e 'not interesting output\nboring\nboring again\nthe values are : value1, value2, value3.\nBORING !!!\nwill this end ?'
    not interesting output
    boring
    boring again
    the values are : value1, value2, value3.
    BORING !!!
    will this end ?
  • The same read hack can work here. All we have to do is mangle this output so that we get values (and values only) separated by SPACE. To do so, one of the weapons of choice is awk :
    echo -e 'not interesting output\nboring\nboring again\nthe values are : value1, value2, value3.\nBORING !!!\nwill this end ?' | awk -F ':' '/the values are/ { $1=""; gsub(/[,\.]/, " ", $2); print; }'
    Depending on the situation, different awk hacks will be necessary.
  • Putting it all together :
    read myVar1 myVar2 myVar3 < <(echo -e 'not interesting output\nboring\nboring again\nthe values are : value1, value2, value3.\nBORING !!!\nwill this end ?' | awk -F ':' '/the values are/ { $1=""; gsub(/[,\.]/, " ", $2); print; }')
    
    echo "$myVar1 $myVar2 $myVar3"
mail

Convert files between DOS and Unix formats

Operating systems differ on the way they handle the newline character :
OS family newline character displayed character Character code
Unix (Linux, *BSD, Mac OS X) LF \n 0x0a
Apple (before Mac OS X) CR \r 0x0d
MS-DOS, Windows CR+LF \r\n 0x0d0a

Files can be converted with utilities :

Convert files from DOS to Unix :

  • convert and replace file :
    • dos2unix file
    • fromdos file
  • convert and create new file :
    • dos2unix -o originalFile newFile
    • fromdos -b file (file will be renamed as file.bak)
mail

How to generate a pin code with a specific length ?

Here, a pin code is a digits-only string (some people call this a "number" ) designed to be used as a password. It must be non-obvious (hence not your birth date nor your car plate / phone number / health insurance ID), and since we (humans) really suck with randomness, let's get help from a computer :

pinCodeLength=10; pinCode=''; while [ "${#pinCode}" -le "$pinCodeLength" ]; do pinCode+=$RANDOM; done; echo "$pinCode" | head -c "$pinCodeLength"

mail

How to alter a line when the previous line matches ?

Situation

I have :
My favorite fruit is :
apples (love the red ones ;-)

On the back of my phone, you can see an...
apple-shaped logo.

New-York is nicknamed :
the "big apple" since the 1970s.
I want :
My favorite fruit is :
apples (love the red ones ;-)

On the back of my phone, you can see an...
android-shaped logo.

New-York is nicknamed :
the "big apple" since the 1970s.

Solution

echo -e "My favorite fruit is :\napples (love the red ones ;-)\n\nOn the back of my phone, you can see an...\napple-shaped logo.\n\nNew-York is nicknamed :\nthe \"big apple\" since the 1970s." | awk 'BEGIN { matchingLineNumber=-1 }; /phone/ { matchingLineNumber=NR }; NR == matchingLineNumber+1 { gsub(/apple/, "android", $0) }; { print }'

Details

The overall concept is pretty similar to this example. Consider the unfolded Awk code :
BEGIN				{ matchingLineNumber=-1 };		just initializing a variable
/phone/				{ matchingLineNumber=NR };		upon finding our needle, just set this variable with the current line number
NR == matchingLineNumber+1	{ gsub(/apple/, "android", $0) };	on the next line, do the substitution
				{ print }				and print the current line
mail

How to (re-)indent code automatically ?

It is sadly frequent that we have to dive into old + cluttered + hardly readable code, that is not even properly indented (or not indented at all). One of the first things to do, then, is to improve readability with proper indentation. This won't magically turn lead into gold, but it helps anyway. Especially if it can be done effortlessly .

This is what reIndentCode.sh is about. Turning this :
a ( b
c d [ e ] f [
g h { j (
k ) } l m
n ] o ) p
q r
into this :
a ( b
	c d [ e ] f [
		g h { j (
				k ) } l m
		n ] o ) p
q r
This shell script is a quick-n-dirty solution I implemented when dealing with a 50KiB Perl script going awry. I bet it has many limitations, but it can be a starting point for the next time or anybody in a similar situation.
mail

How to get a cascading list of owner + group + permissions for a specified file ?

Situation

Sometimes you end in a situation where you can not write into /this/is/a/very/long/pathTo/myFile, even though myFile has the proper bits set. So it would be interesting to have a clear picture of owner + group + permissions, from the top, down to myFile.

Solution

The solution below works fine, so I'll leave it here —maybe it can become an inspiration for future needs— but there's actually a single command to do the very same thing : namei .
fullPathToStudy='/this/is/a/very/long/pathTo/myFile'; nbFields=$(echo "$fullPathToStudy" | grep -o '/' | wc -l); output=''; for ((i=1; i<=nbFields+1; i++)); do currentPath="$(echo "$fullPathToStudy" | cut -d '/' -f 1-$i)/"; [ -d "$currentPath" ] && output+="\n$(ls -ld "$currentPath")"; done; [ -f "$fullPathToStudy" ] && output+="\n$(ls -l "$fullPathToStudy")"; echo -e "$output" | awk '{print $1" "$3" "$4" "$NF}' | column -s ' ' -t
drwxr-xr-x	root	root		/
drwxr-xr-x	root	root		/this/
drwxr-xr-x	bob	developers	/this/is/
drwxr-xr-x	bob	developers	/this/is/a/
drwxr-xr-x	bob	developers	/this/is/a/very/
drwx------	bob	developers	/this/is/a/very/long/
drwx------	bob	developers	/this/is/a/very/long/pathTo/
-rw-------	bob	developers	/this/is/a/very/long/pathTo/myFile
mail

How to find duplicate lines in a file ?

Situation

Let's consider a file such as :
Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Etiam mollis viverra ligula,
Lorem ipsum dolor sit amet,
ut luctus magna imperdiet eget.
Ut consectetur laoreet venenatis.
Nulla euismod sapien nec sodales tempor.
Lorem ipsum dolor sit amet,
Suspendisse sagittis odio eu urna imperdiet,
vitae sollicitudin ante mattis.
How can I spot the duplicated lines ?

Solution

echo -e "Lorem ipsum dolor sit amet,\nconsectetur adipiscing elit.\nEtiam mollis viverra ligula,\nLorem ipsum dolor sit amet,\nut luctus magna imperdiet eget.\nUt consectetur laoreet venenatis.\nNulla euismod sapien nec sodales tempor.\nLorem ipsum dolor sit amet,\nSuspendisse sagittis odio eu urna imperdiet,\nvitae sollicitudin ante mattis." | sort | uniq -c | sort -n | awk '$1 > 1 && $2 !~ "^(#.*)?$" {print}'

Same as above, with contents stored in a file :

sort fileWithDuplicateLines | uniq -c | sort -n | awk '$1 > 1 && $2 !~ "^(#.*)?$" {print}'

mail

How to remove multi-line comments ?

Situation

Considering this piece of code below, how can I remove the comments ?
not commented
/*comment part 1
comment part 2
comment part 3*/ but I want to keep the end of this line
not commented either
This snippet will be fed into lines hereafter :
echo -e 'not commented\n/*comment part 1\ncomment part 2\ncomment part 3*/ but I want to keep the end of this line\nnot commented either'

Solution

There are plenty of different / context-specific / incompatible / incomplete ways to do this

tr | sed | tr method :

The idea is to :
  1. turn the whole input into a single giant line
  2. remove the comments
  3. turn the remainings back into distinct lines
echo -e 'not commented\n/*comment part 1\ncomment part 2\ncomment part 3*/ but I want to keep the end of this line\nnot commented either' | tr '\n' 'X' | sed -r 's|(.*)/\*.*?\*/(.*)|\1\2|g' | tr 'X' '\n'

Doesn't work if there are several distinct multi-line comments :

echo -e 'not commented\n/*comment 1 part 1\ncomment 1 part 2\ncomment 1 part 3*/ but I want to keep the end of this line\nnot commented either\n/* comment 2 part 1\ncomment 2 part 2\ncomment 2 part 3*/, keep this\nnot commented either.' | tr '\n' 'X' | sed -r 's|(.*)/\*.*?\*/(.*)|\1\2|g' | tr 'X' '\n'

Alternate solution

Other (better !) method (inspired by) :

cat << EOF | sed 's|/\*|\n&|g; s|*/|&\n|g' | sed '/\/\*/,/*\//d' | sed '/^$/d'
not commented
/*comment 1 part 1
comment 1 part 2
comment 1 part 3*/ but I want to keep the end of this line
not commented either
/* comment 2 part 1
comment 2 part 2
comment 2 part 3*/, keep this
not commented either.
EOF
Details :
's|/\*|\n&|g
change whatever_before/*comment into
whatever_before
/*comment
making the /* the first characters of a new line
s|*/|&\n|g'
change comment*/whatever_after into
comment*/
whatever_after
sed '/\/\*/,/*\//d'
delete block of lines starting with /* and ending with */ (i.e. the comments we just isolated on distinct lines)
sed '/^$/d'
delete empty lines
mail

How to count occurrences of a character in a string ?

echo 'Hello world' | grep -o 'l' | wc -l
3
mail

How to escape single quotes within single-quoted strings ?

Solution

Details

Explanation of how '"'"' is interpreted as just ' :

There are 3 successive quoted strings : aaa, b and ccc :
 aaa  b  ccc
'foo'"'"'bar'
^   ^^^^^   ^
1   23456   7
  • 1 (') : start 1st quotation (aaa) using single quotes
  • 2 (') : end 1st quotation
  • 3 (") : start 2nd quotation (b) using double-quotes
  • 4 (') : quoted character, the one we wanted to escape
  • 5 (") : end 2nd quotation
  • 6 (') : start 3rd quotation (ccc) using single quotes
  • 7 (') : end 3rd quotation
mail

How to concatenate strings ?

The good ol'method :

myString='hello'; myString="$myString world"; echo "$myString"
hello world

The += method :

  • This is a Bash-specific construct (i.e. /bin/bash only. It will fail in scripts starting with #!/bin/sh & al.)
  • Not supported in 3.x Bash versions (exact version not found, feel free to investigate the Bash source code)
  • myString='hello'; myString+=" world"; echo "$myString"
    hello world
  • Whatever the variable type (even though they look like integers), += concatenates :
    value=42; value+=31; echo "$value"
    4231
  • To workaround this and actually sum, use let :
    value=42; let value+=31; echo "$value"
    73
    You can also subtract, multiply and divide :
    value=42; let value-=40; echo "$value"; value=42; let value*=4; echo "$value"; value=42; let value/=2; echo "$value"
    2
    168
    21
mail

How to get a file modification date ?

Please, please, PLEASE : don't parse the output of ls !!!

There are dedicated + convenient + reliable + simple commands to get a file modification date :

mail

How to write all the outputs of a script into a file ?

Situation

Solution 1 (badness=1) :

command1 > "/path/to/logFile" 2>&1
command2 >> "/path/to/logFile" 2>>&1
command3 >> "/path/to/logFile" 2>>&1
Does the job but will clutter your script making it barely readable. Ok for very short scripts.

Solution 2 (badness=100) :

(
command1
command2
command3
) > "/path/to/logFile" 2>&1
This hack looks so awkward I honestly couldn't have imagined that myself. I saw that in a 1000+ lines non-indented shell script full of UUoC, backticks value=`command`, parsing the output of ls, ...

jump to the solution

Details

Solution

#!/usr/bin/env bash

showResult() {
	local blockType=$1
	echo "======== cat '$logFile' after '$blockType' block"
	cat "$logFile"
	echo '======== /cat'
	rm "$logFile"
	echo
	}


logFile="$0.log"
myVariable='initial value'

########## with '()' ##########
#	==> creates a subshell
echo "before () : $myVariable"
(
	myVariable='changed inside ()'
	echo 'hello'
	echo 'world'
	echo "in () : $myVariable"
) > "$logFile"
echo "after () : $myVariable"

showResult '()'


########## with '{}' ##########
#	==> no subshell
echo "before {} : $myVariable"
{
	myVariable='changed inside {}'
	echo 'hello'
	echo 'world'
	echo "in () : $myVariable"
} > "$logFile"
echo "after {} : $myVariable"

showResult '{}'


########## with 'exec' ##########
#	==> uses file descriptors
echo "before exec : $myVariable"
exec 10>&1 20>&2 1>"$logFile" 2>&1

myVariable='changed inside exec'
echo 'hello'
echo 'world'
echo "in exec : $myVariable"

exec 2>&20 20>&- 1>&10 10>&-
echo "after exec : $myVariable"

showResult 'exec'
before () : initial value
after () : initial value
======== cat './test.sh.log' after '()' block
hello
world
in () : changed inside ()
======== /cat

before {} : initial value
after {} : changed inside {}
======== cat './test.sh.log' after '{}' block
hello
world
in () : changed inside {}
======== /cat

before exec : changed inside {}
after exec : changed inside exec
======== cat './test.sh.log' after 'exec' block
hello
world
in exec : changed inside exec
======== /cat

Details

exec can be used to redirect all the outputs. If, at some point of the script, we want to "stop redirecting the outputs", we have to (source) :

step description command file descriptors
0 before any redirection (n/a)
  • 1 : /dev/stdout
1 prepare for the recovery, then redirect exec 3>&1 1>logFile
  • 3 : /dev/stdout
  • 1 : logFile
2 use the output redirection : anything that should normally be written to screen goes to logFile any command you like
3 recover the original standard output (i.e. "stop redirecting") exec 1>&3 3>&-
  • 1 : /dev/stdout
  • 3 : closed. nowhere ?

Named file descriptors (aka "automatic file descriptor allocation")

Based on the example above :
  • I don't know whether there are restrictions (or risk of collisions) on file descriptor numbers...
  • file descriptors 3 to 9 are available (source)
  • starting from Bash 4.1 (May 2010) (source), it is possible (recommended?) to use "automatic file descriptor allocation" instead of picking numbers manually (source) :
#!/usr/bin/env bash

outFile=$(mktemp)

echo 'foo'

exec {fileDescriptor}>&1 1>"$outFile"
echo 'bar'

exec 1>&${fileDescriptor} {fileDescriptor}>&-
echo 'baz'

cat "$outFile"
rm "$outFile"
echo "The chosen file descriptor was : '$fileDescriptor'"
foo
baz
bar
The chosen file descriptor was : '10'
  • x&<y and x&>y both mean "make x a copy of y". The only difference is that they respectively refer to an input and output file descriptor.
  • if y is -, x will be closed.
source

Snippet to copy-paste :

exec {previousStdout}>&1 {previousStderr}>&2 1>"/path/to/logFile" 2>&1

(script content goes here)

exec 2>&${previousStderr} {previousStderr}>&- 1>&${previousStdout} {previousStdout}>&-

mail

How to detect whether a script is run interactively or not ?

Situation

As most (if not all) questions found here, this question arose from a real-life situation. That time, I had to fix an old script written by someone who's not even part of the company anymore...
My readings and the tests I made lead me to the conclusion that asking this question is the sign of poor script design. Functionalities like : should be enough to design a script that works fine —exactly the same way— whatever method is used to start it : manually / at / cron.
For best practices, have a look at :

Details

Let's try some methods and check whether they answer our question :
#!/usr/bin/env bash
#	To fire this script via 'at' :
#	at $(date --date "now +1 minutes" '+%H%M') -f myScript.sh

exec 1> output.txt

########################################## ##########################################################

# source
[ -z "$PS1" ] && interactive='no' || interactive='yes'; echo "interactive 1 : '$interactive'"

# when run :
#	manually			==>	yes
# 	in a script			==>	no
# 	in a script fired by 'at'	==>	yes
# 	in a script fired by 'cron'	==>	no


########################################## ##########################################################

# source
case $- in *i*) interactive='yes' ;; *) interactive='no' ;; esac; echo "interactive 2 : '$interactive'"

# when run :
#	manually			==>	yes
# 	in a script			==>	no
# 	in a script fired by 'at'	==>	no
# 	in a script fired by 'cron'	==>	no


########################################## ##########################################################

echo "dollarDash : '$-'"
# when run :
#	manually			==>	himBHs
# 	in a script			==>	hB
# 	in a script fired by 'at'	==>	s
# 	in a script fired by 'cron'	==>	hB

Solution

There is no easy / standard / reliable way to distinguish use cases, but there are some workarounds :
mail

How to transpose line to column / column to line ?

Line to column Column to line
"Single" input
  • fieldSeparator=';'; echo "foo${fieldSeparator}bar${fieldSeparator}baz" | tr "$fieldSeparator" '\n'
  • echo -e 'foo\nbar\nbaz' | xargs
  • fieldSeparator=';'; echo -e 'foo\nbar\nbaz' | xargs | tr ' ' "$fieldSeparator"
"Multiple" input See https://unix.stackexchange.com/questions/520031/pivot-file-values#answer-520047
mail

How to compare code snippets ?

  1. Append these functions to ~/.bash_aliases :
    getFileSnippet() {
    	local fileToInspect=$1
    	local startLine=$2
    	local stopLine=$3
    	tmpFile=$(mktemp --tmpdir='/run/shm' tmp.XXXXXXXX)
    	sed -n "$startLine,${stopLine}p" "$fileToInspect" > "$tmpFile"
    	echo "$tmpFile"
    	}
    
    compareSnippets() {
    	[ $# -ne 6 ] && { echo 'Wrong number of arguments, 6 expected.'; return $UNIX_FAILURE; }
    	local fileToInspect1=$1
    	local startLine1=$2
    	local stopLine1=$3
    	local fileToInspect2=$4
    	local startLine2=$5
    	local stopLine2=$6
    
    	for argumentToCheck in fileToInspect1 fileToInspect2; do
    		[ -f "${!argumentToCheck}" ] || { echo "Argument '${!argumentToCheck}' is not a file."; return $UNIX_FAILURE; }
    	done
    
    	snippet1=$(getFileSnippet "$fileToInspect1" "$startLine1" "$stopLine1")
    	snippet2=$(getFileSnippet "$fileToInspect2" "$startLine2" "$stopLine2")
    	diff "$snippet1" "$snippet2"
    	rm "$snippet1" "$snippet2"
    	}
  2. compare code snippets with :
    compareSnippets fileA 117 124 fileB 159 166
mail

How to remove the line matching a regexp and the following one ?

Situation

I have :
foo1
bar1
foo2 REMOVE THIS LINE AND THE FOLLOWING ONE
bar2
foo3
bar3
I want :
foo1
bar1
foo3
bar3

Solution

echo -e 'foo1\nbar1\nfoo2 REMOVE THIS LINE AND THE FOLLOWING ONE\nbar2\nfoo3\nbar3' | awk 'BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}'

Details

Let's consider the awk '' part remembering :
BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}
becomes :
BEGIN				{ matchingLineNumber = -1 }
/REMOVE/			{ matchingLineNumber = NR; next }
NR == matchingLineNumber + 1	{ next }
				{ print }
It works whatever the position of the matching line within the input :

echo -e 'foo2 REMOVE THIS LINE AND THE FOLLOWING ONE\nbar2\nfoo1\nbar1\nfoo3\nbar3' | awk 'BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}'

echo -e 'foo1\nbar1\nfoo3\nbar3\nfoo2 REMOVE THIS LINE AND THE FOLLOWING ONE\nbar2' | awk 'BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}'
mail

How to swap command-line arguments ?

Situation

While building a one-liner, one of the commands outputs :
a b
whereas the next command expects :
b a
How may I swap them ?

Solution

mail

How to read a multiline variable line by line ?

Situation

myVariable=$(echo {a..c} | tr ' ' '\n'); echo "myVariable : '$myVariable'"
myVariable : 'a
b			definitely a multiline variable
c'
echo "$myVariable" | while read aSingleLineOfMyVariable; do
	echo "a single line : '$aSingleLineOfMyVariable'"
done; echo "last value : '$aSingleLineOfMyVariable'"
a single line : 'a'	works fine inside the loop
a single line : 'b'
a single line : 'c'
last value : ''		undefined variable since it only exists in the subshell created while piping

Details

tmpFile=$(mktemp); echo "$myVariable" > "$tmpFile"; while read aSingleLineOfMyVariable; do
	echo "a single line : '$aSingleLineOfMyVariable'"
done < "$tmpFile"; echo "last value : '$aSingleLineOfMyVariable'"; rm "$tmpFile"
a single line : 'a'
a single line : 'b'
a single line : 'c'
last value : ''		$aSingleLineOfMyVariable is lost when leaving the loop
Nice try but :
  • no improvement since the previous test
  • creating a temporary file is not very elegant and can have a performance cost if repeated numerous times
  • still can't catch the last value and use it outside of the loop
while IFS= read aSingleLineOfMyVariable; do
	echo "a single line : '$aSingleLineOfMyVariable'"
done < <(printf '%s\n' "$myVariable"); echo "last value : '$aSingleLineOfMyVariable'"
a single line : 'a'
a single line : 'b'
a single line : 'c'
last value : ''		$aSingleLineOfMyVariable is lost again, continue reading
Whatever command is (including something like echo "$someVariable"), the <(command) construct (aka process substitution) expands to a file and, as such, can be fed into anything with < or >.

Solution

previousIfs="$IFS"; IFS=$'\n'; for aSingleLineOfMyVariable in $myVariable; do
	echo "a single line : '$aSingleLineOfMyVariable'"
done; echo "last value : '$aSingleLineOfMyVariable'"; IFS="$previousIfs"
a single line : 'a'
a single line : 'b'
a single line : 'c'
last value : 'c'	$aSingleLineOfMyVariable is not lost this time 

This comment says this is because while loops create a subshell, whereas for loops don't. I'm afraid this is wrong... (it is !)

Let's check this :

Test #1 :

tmpFile=$(mktemp); echo -e 'a\nb\nc' > "$tmpFile"; while read item; do echo "item (during loop) : $item"; done < "$tmpFile"; echo "item (after loop) : $item"; rm "$tmpFile"
item (during loop) : a
item (during loop) : b
item (during loop) : c
item (after loop) :		empty variable
for item in {a..c}; do echo "item (during loop) : $item"; done; echo "item (after loop) : $item"
item (during loop) : a
item (during loop) : b
item (during loop) : c
item (after loop) : c		still exist outside of the loop 

Looks like it was right, after all ?

Test #2 :

for i in 'in for loop'; do myVariable='foo'; echo "$i"; done; echo "myVariable : $myVariable"
in for loop
myVariable : foo

A variable set within a for loop still exists after the loop.

while true; do myVariable='bar'; echo 'in while loop (1)'; break; done; echo "myVariable : $myVariable";
while [ -z "$i" ]; do myVariable='baz'; echo 'in while loop (2)'; i=1; done; echo "myVariable : $myVariable"
in while loop (1)
myVariable : bar
in while loop (2)
myVariable : baz

A variable set within a while loop still exists after the loop, whichever way the loop ends : break or regular exit. This proves the comment linked above WRONG !

while read item; do myVariable=meu; echo "item : '$item', myVariable : '$myVariable'"; done < <(echo -e 'ga\nbu\nzo'); echo "item : '$item', myVariable : '$myVariable'"
item : 'ga', myVariable : 'meu'
item : 'bu', myVariable : 'meu'
item : 'zo', myVariable : 'meu'
item : '', myVariable : 'meu'

Trying to workaround with a process substitution makes no difference

unset myVariable; for i in whatever; do read myVariable; echo "myVariable : '$myVariable'"; done; echo "myVariable : '$myVariable'"
or :
unset myVariable i; while [ -z "$i" ]; do read myVariable; echo "myVariable : '$myVariable'"; i=foo; done; echo "myVariable : '$myVariable'"
+ any +
myVariable : 'any'
myVariable : 'any'

A variable set with read, both in for and while loops, survives the end of the loop.

tmpFile=$(mktemp); echo -e 'ga\nbu\nzo' > "$tmpFile"; while read item; do myVariable=meu; echo "item : '$item', myVariable : '$myVariable'"; done < "$tmpFile"; echo "item : '$item', myVariable : '$myVariable'"; rm "$tmpFile"
item : 'ga', myVariable : 'meu'
item : 'bu', myVariable : 'meu'
item : 'zo', myVariable : 'meu'
item : '', myVariable : 'meu'

A variable set with a while read construct doesn't survive the end of the loop. Would that be the reason ?

The revelation (source) :

Indeed, the while read construct is the explanation of this behavior. While consuming lines of input, while read myVariable...
  1. catches a line of input and stores it in myVariable
  2. if it succeeds (i.e. not the end of the input), the while loop continues normally
  3. otherwise (i.e. no more input data) :
    1. myVariable gets an empty value
    2. this makes read return a UNIX_FAILURE
    3. the loop condition is false : the while loop ends
  4. once outside of the while loop, myVariable looks empty. It's actually been overwritten with an empty value by the last read
mail

How to convert a relative path into an absolute path to source from anywhere ?

Situation

Consider script.sh that sources functions.sh like this :
. ./functions.sh

Details

This implies script.sh and functions.sh are in the same directory, and it works only if script.sh is launched from its own directory. Otherwise, the relative path ./functions.sh can not be resolved because ./ is interpreted as "the directory from which the command is launched".

Solution

To workaround this, you can automatically translate the relative path into an absolute path before source-ing :
# Include an external file even though the current script is not launched from its own directory
directoryOfThisScript="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
. "$directoryOfThisScript/functions.sh"

Alternate solution

Full snippet for a script that has several files to source + error management :
######################################### includes ##################################################
nameOfThisScript=$(basename "${BASH_SOURCE[0]}")
directoryOfThisScript="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

configFile="$directoryOfThisScript/$nameOfThisScript.conf"
functionsFile="$directoryOfThisScript/functions.sh"

for fileToSource in "$configFile" "$functionsFile"; do
	source "$fileToSource" 2>/dev/null || {
		echo "File '$fileToSource' not found"
		exit 1
		}
done
######################################### /includes #################################################

Alternate solution

This works fine until the script is symlinked and launched via a link.

jump to the solution

#!/usr/bin/env bash

tmpDir=$(mktemp -d --tmpdir zzzMyTmpDir.XXXXXXXX)

subDir="$tmpDir/subDir"
theScript="$subDir/script.sh"
theConfigFile_basename='config.sh'; theConfigFile="$subDir/$theConfigFile_basename"
theLinkToTheScript="$tmpDir/linkToScript"

mkdir -p "$subDir"
cat << EOF > "$theScript"
#!/usr/bin/env bash echo "Hello world, I've been launched with (\\\$0) : \$0" if [ -h "\$0" ]; then echo 'I am a symlink' theScriptImRunning=\$(readlink -f \${BASH_SOURCE[0]}) theScriptImRunning is the target of the link else echo 'I am a regular file' theScriptImRunning=\${BASH_SOURCE[0]} theScriptImRunning is 'myself' fi directoryOfThisScript="\$( cd "\$( dirname "\$theScriptImRunning" )" && pwd )" source "\$directoryOfThisScript/$theConfigFile_basename" echo -e "a = \$a\n"
EOF cat << EOF > "$theConfigFile"
#!/usr/bin/env bash a=12
EOF chmod +x "$theScript" ln -sf "$theScript" "$theLinkToTheScript" "$theScript" actual script, launched via its absolute path "$theLinkToTheScript" script launched via a symlink # clean before leaving cd "$tmpDir/.." [ -d "$tmpDir" ] && rm -r "$tmpDir"
Hello world, I've been launched with ($0) : /tmp/zzzMyTmpDir.2A5nws0E/subDir/script.sh
I am a regular file
a = 12

Hello world, I've been launched with ($0) : /tmp/zzzMyTmpDir.2A5nws0E/linkToScript
I am a symlink
a = 12

So the solution is...

# Include an external file even though the current script is launched
# - not from its own directory
# - via a symlink

[ -h ${BASH_SOURCE[0]} ] && theScriptImRunning="$(readlink -f ${BASH_SOURCE[0]})" || theScriptImRunning=${BASH_SOURCE[0]}
directoryOfThisScript="$( cd "$( dirname "$theScriptImRunning" )" && pwd )"

. "$directoryOfThisScript/config.sh"
mail

How to return a value from a function ?

Usage

There are several ways to do so :
return + $?
For integers only (and "not too big" ones, preferably). Actually, the returned number will be n modulo 256 :
foo() { return $i; }; for i in {254..258}; do foo $i; echo "$i $?"; done
254 254
255 255
256 0
257 1
258 2
echo + $()
For anything you like : numbers, strings, ...

Example

#!/usr/bin/env bash

functionThatReturns() {
	return 42
	}

functionThatEchoes() {
	echo 42
	}

functionThatReturns
echo "1. '$?'"

result=$(functionThatReturns)
echo "2. '$result'"

functionThatEchoes
echo "3. '$?'"

result=$(functionThatEchoes)
echo "4. '$result'"










1. '42'		return + $? = OK


2. ''		return + $() = KO
42		this is the echo made in the function

3. '0'		this only means the function ended successfully


4. '42'		echo + $() = OK
mail

How to get the position of a character in a string ?

Command When haystack has needle ...
as the 1st character 0 time exactly 1 time more than 1 time
haystack='hello world'; needle='h' haystack='hello world'; needle='z' haystack='hello world'; needle='w' haystack='hello world'; needle='o'
echo "$haystack" | grep -oab "$needle" | grep -oE '[0-9]+'
0
0-based index
UNIX_FAILURE
6
0-based index
4
7
0-based index
printf '%s\n' "$haystack" | grep -o . | grep -n "$needle" | grep -oE '[0-9]+'
1
1-based index
UNIX_FAILURE
7
1-based index
5
8
1-based index
tmp="${haystack%%$needle*}"; echo ${#tmp}
0
0-based index
11
length of $haystack
6
0-based index
4
position of 1st occurrence, 0-based index
echo "$haystack" | awk -v x="$needle" '{ print index($0, x) }'
1
1-based index
0
Looks like a reliable "not found" indicator
7
1-based index
5
position of 1st occurrence, 1-based index
mail

How to display the xth, yth and zth lines of a stream of text ?

Let's consider a command returning 10 lines of text, from which we want to display only the 1st, 3rd and 8th lines.
mail

How to keep / remove the n leading / trailing characters of a string ?

Here are a few snippets with :
keep remove
leading
trailing
mail

How to test if a variable is an integer ?

UNIX_SUCCESS=0
UNIX_FAILURE=1
isInteger() {
	[[ $1 =~ ^[-+]?[0-9]+$ ]] && echo $UNIX_SUCCESS || echo $UNIX_FAILURE
	}
[ $(isInteger "$myVariable") -eq "$UNIX_FAILURE" ] && {
	(shout how unhappy you feel)
	}
mail

How to parse a string character by character ?

You could try :
myString='hello'
length=$((${#myString}-1))
for i in $(eval echo "{0..$length}"); do
	echo $i : ${myString:i:1}
done
But this is even better (source) :
myString='hello'
for((i=0; i<${#myString}; i++)); do
	echo $i : ${myString:i:1}
done
mail

How to store a command into a variable ?

Use eval.

If you want to store
  • the result of a command
  • but not the command itself
into a variable, just use the command substitution construct $(command) :
now=$(date); echo "'now' was : $now"
mail

How to check a string against a list of values ?

With a for loop :

To make sure a command line parameter matches a value from a list :

#!/usr/bin/env bash

valueToCheck="$1"
listOfAcceptedValues='foo bar baz'

UNIX_SUCCESS=0
UNIX_FAILURE=1
valueToCheckIsInTheList=$UNIX_FAILURE

for value in $listOfAcceptedValues; do
	echo "Testing '$valueToCheck' against '$value'"
	[ "$valueToCheck" == "$value" ] && { echo "'$valueToCheck' is a valid value."; valueToCheckIsInTheList=$UNIX_SUCCESS; break; }
done

[ "$valueToCheckIsInTheList" -eq "$UNIX_FAILURE" ] && { echo "'$valueToCheck' is not a valid value."; exit $UNIX_FAILURE; }

With a case construct :

This suits cases where different input values imply different behaviors of the script. Otherwise, using case may be overkill, and the for loop method may be more adapted.

#!/usr/bin/env bash

valueToCheck="$1"

case "$valueToCheck" in
	'foo')
		echo "'$valueToCheck' is a valid value."
		# do something with 'foo'
		;;
	'bar')
		echo "'$valueToCheck' is a valid value."
		# do something different with 'bar'
		;;
	'baz')
		echo "'$valueToCheck' is a valid value."
		# do something different again with 'baz'
		;;
	*)
		echo "'$valueToCheck' is not a valid value."
		# deal with it !!!
		;;
esac

With a regular expression :

This method may return false positives if used improperly. Indeed, to check whether foo is within foo bar baz, we can "regexp match" foo bar baz against foo (it matches), but fo and f also match.

It _may_ sound more logical to do it the other way round : matching foo against foo bar baz, but this obviously can't work.

To workaround false positive matches of fo and f, we must use "word boundary detectors" :
  • (^|[[:space:]]) and ($|[[:space:]]) (which is very well explained here) (this is possibly wrong or obsolete)
  • or \b
#!/usr/bin/env bash

listOfAcceptedValues='foo bar baz'
listOfValuesToCheck="foo bar baz poo 123 bam ofo fo f ''"

for valueToCheck in $listOfValuesToCheck; do
	[[ "$listOfAcceptedValues" =~ "$valueToCheck" ]] && result1='' || result1=' not'
	[[ "$listOfAcceptedValues" =~ (^|[[:space:]])"$valueToCheck"($|[[:space:]]) ]] && result2='' || result2=' not'
	echo -e "'$valueToCheck'\tis : (1)$result1 a valid value, \t(2)$result2 a valid value."
done
'foo'	is : (1) a valid value,		(2) a valid value.
'bar'	is : (1) a valid value,		(2) a valid value.
'baz'	is : (1) a valid value,		(2) a valid value.
'poo'	is : (1) not a valid value,	(2) not a valid value.
'123'	is : (1) not a valid value,	(2) not a valid value.
'bam'	is : (1) not a valid value,	(2) not a valid value.
'ofo'	is : (1) not a valid value,	(2) not a valid value.
'fo'	is : (1) a valid value,		(2) not a valid value.
'f'	is : (1) a valid value,		(2) not a valid value.
''''	is : (1) not a valid value,	(2) not a valid value.

Other example :

validUsers='kevin stuart bob'; for user in alice bob bobby; do regex="\b$user\b"; [[ "$validUsers" =~ $regex ]] && echo "'$user' is valid" || echo "'$user': NOPE"; done
'alice': NOPE
'bob' is valid
'bobby': NOPE

Other methods :

There are other methods, but they don't all apply to the same use cases, especially if they have to check untrusted values (i.e. user input) which may contain wildcards or stuff like that.
mail

How to generate a string made of repetitions of a character / substring ?

Repeat a single character :

With Perl (source) :

perl -e 'print "X"x42; print "\n"'
Check :
generatedString=$(perl -e 'print "X"x42; print "\n"'); echo $generatedString; echo ${#generatedString}

With Python :

python -c "print('X' * 42)"
Check :
generatedString=$(python -c "print('X' * 42)"); echo $generatedString; echo ${#generatedString}

With Bash (source) :

A short and elegant solution :
printf 'X%.0s' {1..20}
Alternate solutions for a variable number of repeats :
  • printf-based :
    • n=4; printf '🍭%.0s' $(eval "echo {1..$n}")
    • n=4; printf '🧸%.0s' $(seq 1 $n)
    This construct is not able to repeat anything 0 times.
  • echo-based :

Repeat a string :

With Perl :

perl -e 'print "ABCD "x4; print "\n"'
ABCD ABCD ABCD ABCD[SPACE]

With Python :

python -c "print('ABCD ' * 4)"
ABCD ABCD ABCD ABCD[SPACE]

With Bash :

A short and elegant solution (source) :
printf 'ABCD %.0s' {1..20}
mail

How to compare a variable against a list of characters ?

Generic case :

#!/usr/bin/env bash

answerIsValid=''

while [ -z "$answerIsValid" ]; do
	echo "Continue ? [yn]"
	read answer
	[[ "$answer" == [yYnN] ]] && answerIsValid=1 || echo -e "Invalid answer\n"
done
  • This is effectively a list of characters, not a regular expression.
  • The characters list mustn't be quoted.

The list of characters can also be provided as a variable :

#!/usr/bin/env bash

answerIsValid=''
validCharacters='yYnN\[\]'

while [ -z "$answerIsValid" ]; do
	echo "Continue ? [yn]"
	read answer
	[[ "$answer" == [$validCharacters] ]] && answerIsValid=1 || echo -e "Invalid answer\n"
done
  • Special characters must be escaped.
  • The example above works either single or double-quoted.

This example script accepts [ or \[ as valid inputs, but .[ is rejected .

mail

How to read command-line arguments in a script ?

method 1

Use getopts.

method 2 : "manually"

#!/usr/bin/env bash

apples=0
bananas=0
coconuts=0

getCliParameters() {
	while [ "$#" -gt 0 ]; do
		case "$1" in
			-a | --apples)   shift; apples="$1"   ;;	1st shift after reading the option itself (letter/word)
			-b | --bananas)  shift; bananas="$1"  ;;
			-c | --coconuts) shift; coconuts="$1" ;;
			-*) echo "Unknown option: '$1'"; exit 1 ;;
		esac
		shift							2nd shift after reading its value
	done
	}

displayValues() {
	cat <<-EOF
	On my shopping list, I have :
	- $apples apples
	- $bananas bananas
	- $coconuts coconuts
	EOF
	}

main() {
	getCliParameters "$@"
	displayValues
	}

main "$@"
./test.sh -a 2 --bananas 3 -c 4
On my shopping list, I have :
- 2 apples
- 3 bananas
- 4 coconuts

./test.sh -a 5
On my shopping list, I have :
- 5 apples
- 0 bananas
- 0 coconuts

./test.sh -a 3 -b 3 -c 3 -d 4
Unknown option: '-d'

comparison

method pros cons
getopts
  • does most of the work for you : this is what it was made for
  • portability issues (on other shells/Unices ?)
  • does not support long options (--parameter value)
"manual"
  • supports long options
  • better portability
  • more work "reinventing the wheel"

things they have in common

  • none has the ability to mark an option as mandatory and warn when missing : you'll have to handle this manually
mail

How to explode a string into an array ?

The basics (details on Bash arrays) :

Explode :
IFS=', ' read -a myArray <<< "$stringToExplode"
Iterate over elements :
for element in "${myArray[@]}"; do
	echo "$element"
done
Iterate over elements using key/value pairs :
for index in "${!myArray[@]}"; do
	echo "$index ${myArray[index]}"
done

Ready-to-use one-liners :

Explode string and iterate on values :

stringToExplode='Lorem ipsum dolor sit amet, ...'; oldIfs="$IFS"; IFS=' '; read -a myArray <<< "$stringToExplode"; IFS="$oldIfs"; for element in "${myArray[@]}"; do echo "$element"; done

Lorem
ipsum
dolor
sit
amet,
...

Explode string and iterate on key / value pairs :

stringToExplode='Lorem ipsum dolor sit amet, ...'; oldIfs="$IFS"; IFS=' '; read -a myArray <<< "$stringToExplode"; IFS="$oldIfs"; for index in "${!myArray[@]}"; do echo "$index ${myArray[index]}"; done

0 Lorem
1 ipsum
2 dolor
3 sit
4 amet,
5 ...
mail

How to sum numbers ?

Numbers listed in a file :

Making sums :

Pure Bash version :
sum=0; while read value; do sumBefore=$sum; sum=$((sum+value)); echo "$sumBefore + $value = $sum"; done < fileWithNumbers; echo $sum
bc version :
sum=0; while read value; do sumBefore=$sum; sum=$(echo "$sum+$value" | bc); echo "$sumBefore + $value = $sum"; done < fileWithNumbers; echo $sum
Awk version :
awk '{ sum += $1 } END { print sum }' fileWithNumbers

Which is the fastest ?

tmpFile=$(mktemp --tmpdir tmp.numbers.XXXXXXXX); >$tmpFile; for i in {1..1000}; do echo $i >> $tmpFile; done; echo 'Bash sum'; time (sum=0; while read value; do sum=$((sum+value)); done < $tmpFile; echo "$sum"); echo 'bc sum'; time (sum=0; while read value; do sum=$(echo "$sum+$value" | bc); done < $tmpFile; echo "$sum"); echo 'Awk sum'; time (awk '{ sum += $1 } END { print sum }' $tmpFile); rm $tmpFile
From the fastest to the slowest :
  1. Awk
  2. Bash
  3. bc

Numbers returned to STDOUT :

Dummy example :
echo -e 'result1=3\nresult2=4\nresult3=5' | awk -F '=' '{ sum += $2 } END {print sum}'
Total storage, in GiB, on internal drives :
df -m | awk '/^\/dev/ { sum += $2 } END { print sum/1024 }'
Number of occurrences of a given expression within a bunch of files :
grep -c 'a given expression' * | awk -F ':' '{ sum += $2 } END {print sum}'
mail

How to check the number of arguments passed to a script ?

nbExpectedArgs=1
if [ $# -ne $nbExpectedArgs ]; then
	echo "Usage: $(basename $0) <argument>"
	exit 1
fi
To display a longer error message with a usage function :
#!/usr/bin/env bash

nbExpectedArgs=3

usage() {
	cat <<-EOF
	Usage: $(basename $0) <argument1> <argument2> <argument3>
	blah
	blah
	blah
	EOF
	}

if [ $# -ne $nbExpectedArgs ]; then
	usage
	exit 1
fi

The long error message is displayed with cat rather than echo.