Bash scripting - Loops

mail

for

With fixed parameters (start, stop, increment) :

With parameters (start, stop, increment) stored in variables :

  • ugly : start=37; stop=73; increment=7; for i in $(eval echo "{$start..$stop..$increment}"); do echo $i; done (source)
  • better : start=37; stop=73; increment=7; for((i=$start; i<$stop; i+=$increment)); do echo $i; done
  • with 0-padding : start=0001; stop=1001; increment=100; for i in $(eval "echo {$start..$stop..$increment}"); do echo $i; done

If you don't just ++ at every loop :

for ((i=1; i<1000000000000000000; i=i*10)); do
echo "$i" | sed -r ': repeat s/([0-9]+)([0-9]{3})(,|$)/\1,\2/; t repeat'
done
This C-style for loop is called an arithmetic for loop (more about for loops).

Looping on items with SPACEs

strings

  • doesn't work :
    list="a b 'c d'"; for item in $list; do echo "$item"; done
    a
    b
    'c
    d'
  • solution :
    list="a|b|c d"; oldIfs=$IFS; IFS='|'; for item in $list; do echo "$item"; done; IFS=$oldIfs
    a
    b
    c d
As a general rule, if you're hitting this kind of behavior :
  • it may be worth considering passing a list of items as a file (one item per line) rather than a string with item separators
  • this file can be easily read with a while read construct
  • people may argue temp files are BAAAD!, but readability matters

files

This works thanks to
FileBaseName='file with spaces '; tmpDir=$(mktemp -d); touch "$tmpDir/$FileBaseName"{1,2,3}

# doesn't work : for is puzzled by SPACEs
for someFile in $(find "$tmpDir" -type f); do echo "$someFile"; done

# works 
while read someFile; do echo "$someFile"; done < <(find "$tmpDir" -type f)

# clean before leaving
[ -d "$tmpDir" ] && rm -r "$tmpDir"

for vs while (source) :

  • ugly : for line in $(cat file.txt); do echo $line; done
  • ugly again (UUOC) : cat file.txt | while read line; do echo $line; done
  • better : while read line; do echo $line; done < file.txt (more)
    This is better because you don't need to spawn a sub-process with |, or with $(...), or start the external cat command.
mail

How to retry until success ?

It _may_ be possible to build a QnD hack based on watch -g

With recursivity :

#!/usr/bin/env bash

doSomething() {
	random=$RANDOM
	echo "I'm working with '$1' and '$2' and '$3' (RANDOM = $random)"
	echo
	[ $random -lt 10000 ] && return 0 || return 1	faking failure
	}

doSomethingUntilSuccess() {
	loopNumber=$((loopNumber+1))
	echo "Doing 'doSomethingUntilSuccess' (loop $loopNumber) ..."
	doSomething 'foo' 'bar' 42 || doSomethingUntilSuccess
	}

loopNumber=0
doSomethingUntilSuccess
echo 'DONE !'
#!/usr/bin/env bash

generateOneWord() {
	local length=$1
	pwgen -0 $length 1
	}

generateWordsUntilMatchRegex() {
	local regex=$1
	echo -n '.'
	generateOneWord "$wordLength" | grep -Ei "$regex" || generateWordsUntilMatchRegex "$regex"
	}

wordLength=40
generateWordsUntilMatchRegex 'h.*e.*l.*l.*o'
generateWordsUntilMatchRegex '^w.*[orl].*d$'

With an until loop :

#!/usr/bin/env bash

trap "{ echo 'CTRL-C detected... Bye-bye.'; exit 1; }" SIGINT

# This function is the job to redo if failed
doSomething() {
	random=$RANDOM
	echo "I'm working with '$1' and '$2' and '$3' (RANDOM = $random)"
	echo
	[ $random -lt 10 ] && return 0 || return 1
	}

loopNumber=1

until doSomething 'foo' 'bar' 42; do
	echo "$loopNumber loops so far, and running ..."
	loopNumber=$((loopNumber+1))
done

echo "DONE !"
echo "In $loopNumber loops "
#!/usr/bin/env bash

doTheJob() {
	echo 'hello world'
	}

checkTheJobIsDone() {
	[ $(($RANDOM % 6)) -ge 5 ]
	}

main() {
	until checkTheJobIsDone; do
		doTheJob
		sleep 1
	done
	echo 'The job is done'
	}

main
mail

while read

Typical case : reading a file line by line :

When working in a highly constrained environment where neither cat nor less are available :
while read line; do echo $line; done < fileToRead
In more details :
tmpFile=$(mktemp)
echo -e 'Bob\nKevin\nStuart' > "$tmpFile";
while read name; do
	echo "Hello, '$name' !"
done < "$tmpFile"
rm "$tmpFile"
Hello, 'Bob' !
Hello, 'Kevin' !
Hello, 'Stuart' !

Leave leading and trailing spaces with IFS= :

tmpFile=$(mktemp)
echo -e ' Bob \n Kevin \n Stuart ' > "$tmpFile";
while read name; do
	echo "Hello, '$name' !"
done < "$tmpFile"
while IFS= read name; do
	echo "Hello, '$name' !"
done < "$tmpFile"
rm "$tmpFile"
Hello, 'Bob' !
Hello, 'Kevin' !
Hello, 'Stuart' !
Hello, ' Bob ' !
Hello, ' Kevin ' !
Hello, ' Stuart ' !

Trying to understand the -r :

tmpFile=$(mktemp)
echo -e 'apples\t12\nbananas\t3\ncoconuts\t42' > "$tmpFile";
while read fruit number; do
	echo -e "Number of '$fruit' : '$number'"
done < "$tmpFile"
while read -r fruit number; do
	echo -e "Number of '$fruit' : '$number'"
done < "$tmpFile"
rm "$tmpFile"
Number of 'apples' : '12'
Number of 'bananas' : '3'
Number of 'coconuts' : '42'
Number of 'apples' : '12'
Number of 'bananas' : '3'
Number of 'coconuts' : '42'
Makes no difference (not the right use case ? I'll have to further investigate this one.)

It also works with process substitution :

while read line; do
	echo "$line"
done < <(ps -u $(whoami) | head -10)
Remember : done < <(command)

... and with heredocs too :

while read line; do
	echo "$line"
done <<< $(ps -u $(whoami) | head -10)
Remember : done <<< $(command)

... which can be used to provide a list of values :

while read line; do
	echo "$line"
done < <(cat <<-EOF
	1 apple
	2 bananas
	3 coconuts
	EOF
	)
This could be done with a for loop, but the while read construct allows listing items vertically, making the code more readable, especially if items are long or if they are made of tuples :
while read line; do
	firstName=$(echo "$line" | cut -d'|' -f1)
	lastName=$( echo "$line" | cut -d'|' -f2)
	echo "First name : '$firstName', last name : '$lastName'"
done < <(cat <<-EOF
	Bruce|Banner
	Lex|Luthor
	Lois|Lane
	Peter|Parker
	EOF
	)
The code on the left can even be simplified into (source) :
while read firstName lastName; do
	echo "First name : '$firstName', last name : '$lastName'"
done < <(cat <<-EOF
	Bruce Banner
	Lex Luthor
	Lois Lane
	Peter Parker
	EOF
	)
First name : 'Bruce', last name : 'Banner'
First name : 'Lex', last name : 'Luthor'
First name : 'Lois', last name : 'Lane'
First name : 'Peter', last name : 'Parker'
mail

if then else fi

Because I can never remember this construct :
if condition; then
	some
	commands
else
	some
	different
	commands
fi
There's no need for { } surrounding the then and else blocks.
mail

until

Usage

until condition
do
	doSomething
done
will doSomething

Snippets

  • a=5; until [ "$a" -eq 2 ]; do echo $a; a=$((a-1)); done
  • until ping -c 1 192.168.144.118 & >/dev/null; do echo -n '.'; sleep 1; done; echo OK
  • url='http://x.y.z.y/page/content/changing/upon/reload/'
    needle='some text'
    tmpFile=$(mktemp --tmpdir tmp.XXXXXXXX)
    loops=0
    until grep -i "$needle" $tmpFile; do
    	loops=$((loops+1))
    	>$tmpFile
    	wget $url -O $tmpFile
    done
    echo "Number of loops : $loops"

An infinite loop

until false; do date; sleep 1; done
mail

case esac

#!/bin/bash
something='hello world'
case $something in
	h*)
		echo 'foo'
		;;
	*)
		echo 'bar'
		;;
esac
foo
something
this is what the various patterns will be matched against
pattern)
  • shell pattern matching (not regexp), do not use quotes (details and examples)
  • before matching is attempted,
    1. tilde expansion
    2. parameter and variable expansion
    3. arithmetic expansion
    4. command substitution
    5. process substitution
    6. quote removal
    are performed
  • the first pattern that matches is executed, the following ones are skipped
  • the final semicolons ;; can not be omitted to create a "fallback" like in C or with the PHP switch
Some of the extended pattern matching operators require the extglob shell option to be enabled (and will produce shell errors otherwise) :
shopt -s extglob
;;
the double semicolon ;; is the "end of case" operator, but is not the only option available :
  • ;; : no subsequent matches are attempted after the first pattern match
  • ;& : causes execution to continue with the list associated with the next set of patterns
    #!/usr/bin/env bash
    
    myFunction() {
    	case "$1" in
    		a) echo a ;;
    		b) echo b ;&
    		c) echo c ;;
    		*) echo star ;;
    	esac
    	echo -----
    	}
    
    myFunction a
    myFunction b
    myFunction c
    myFunction z
    a	prints then hits ;; → stop
    -----
    b	prints then executes the code of the next case unconditionally
    c	prints then hits ;; → stop
    -----
    c	prints then hits ;; → stop
    -----
    star	prints then hits ;; → stop
    -----
  • ;;& : causes the shell to
    1. test the next pattern list in the statement (if any)
    2. if this next pattern matches, execute the associated code
    3. then, continue the case statement execution as if the pattern list had not matched in (1)
    The exit status is zero if no pattern matches. Otherwise, it is the exit status of the last command executed in list.
    #!/usr/bin/env bash
    
    myFunction() {
    	case "$1" in
    		a) echo a ;;
    		b) echo b ;;&
    		c) echo c ;;
    		*) echo star ;;
    	esac
    	echo -----
    	}
    
    myFunction a
    myFunction b
    myFunction c
    myFunction z
    a	prints then hits ;; → stop
    -----
    b	prints then checks whether the next case matches (here, no match)
    star	continue evaluating the following cases : here the * matches
    -----
    c	prints then hits ;; → stop
    -----
    star	prints then hits ;; → stop
    -----
For full details :
man -P 'less -p "case word"' bash
mail

How to interrupt a loop ?

An example is worth 1000 words :
#!/usr/bin/env bash

listOfCommands=': break continue return exit'
for command in $listOfCommands; do
	echo "With '$command'"

	for i in {1..3}; do
		echo -n ' ==> '
		[ "$i" -eq 2 ] && eval "$command"
		echo $i
	done
	echo -e 'This comes after the loop\n'

done
With ':'
 ==> 1
 ==> 2		nothing special, as expected
 ==> 3
This comes after the loop

With 'break'
 ==> 1
 ==> This comes after the loop	stop the whole loop block and continue the script after it

With 'continue'
 ==> 1
 ==>  ==> 3	skip the rest of the current loop and start the next loop
This comes after the loop

With 'return'
 ==> 1
 ==> ./test.sh: line 9: return: can only `return' from a function or sourced script	complain and interrupt nothing
2
 ==> 3
This comes after the loop

With 'exit'
 ==> 1
 ==> 	stop the script, return to the shell prompt