I like Bash scripting - Httqm's Docs

#!/usr/bin/env bash

myFunction() {
	while [ "$#" -gt 0 ]; do
		echo "arg: $1"
		shift
	done
	}

myFunction one
echo
myFunction one two
echo
myFunction one two three
echo
myFunction {a..z}

arg: one

arg: one
arg: two

arg: one
arg: two
arg: three

arg: a
arg: b
arg: c
arg: d

arg: x
arg: y
arg: z

with tr
with sed
with Bash parameter expansion :
- convert case
- transform or return information about parameter

Just consider this snippet :

for word in Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut gravida dolor ornare, bibendum nunc eu, ornare nisl. Integer ut diam est.; do
	echo "$word"
done

It works fine but lacks readability, especially as the list grows longer.

This _may_ help :

while read word; do
	echo "$word"
done < <(cat <<-EOLIST
Lorem
ipsum
dolor
sit
amet,
consectetur
adipiscing
elit.
Ut
gravida
dolor
ornare,
bibendum
nunc
eu,
ornare
nisl.
Integer
ut
diam
est.
EOLIST
)

but has the same readability issue when the list gets loooong .

store values in a dedicated file : values.txt

while read word; do
	echo "$word"
done < values.txt

The basic method

Let's try basic things first, it _could_ work :

myVariable='line 1
line 2
line 3'; echo $myVariable

line 1 line 2 line 3

Same with a slight change :
```
myVariable='line 1
line 2
line 3'; echo "$myVariable"
```
```
line 1
line 2
line 3
```
The magic is not in the variable itself, but in what you instruct echo to do with it.

An advanced method

This method is said to "better" support characters like quotes / special characters, but looks like it doesn't support hardcore things like that :

That long string was made with :
	pwgen -ys 64 --num-passwords=4

myVariable='dtIqDWwwA{[o>,%M:p;zW>ri$wdYM&9`:U%juyA1kB&lk"Hu<*05]|c=I?6~@0`h
Y\]DzR2{S=oO:m}@3_G}[6bc+g&{N;L1)MLKr1U9$+HIh}J{=bs%WM60i_Vq'QK"
wiy~Pg^?,A7ISC]t[UX`'*B%Nt~F3Qo,Hon1JPD9hdJ{|DOYQ%2s`nUJ;w*Ra\el
r1'CMx%*{X}:rG@C94V)0uczM#8Vh08`L@mR:Yv5|Od/[<\+$6M_J*5Gi$3)C2yf'; echo "$myVariable"

read -d '' myVariable << EOF; echo $myVariable
dtIqDWwwA{[o>,%M:p;zW>ri$wdYM&9`:U%juyA1kB&lk"Hu<*05]|c=I?6~@0`h
Y\]DzR2{S=oO:m}@3_G}[6bc+g&{N;L1)MLKr1U9$+HIh}J{=bs%WM60i_Vq'QK"
wiy~Pg^?,A7ISC]t[UX`'*B%Nt~F3Qo,Hon1JPD9hdJ{|DOYQ%2s`nUJ;w*Ra\el
r1'CMx%*{X}:rG@C94V)0uczM#8Vh08`L@mR:Yv5|Od/[<\+$6M_J*5Gi$3)C2yf
EOF

But it works fine with "normal text" :

read -d '' myVariable << EOF; echo "$myVariable"
A SQL query goes into a bar,
walks up to two tables and asks,
"Can I join you?"
EOF

A SQL query goes into a bar,
walks up to two tables and asks,
"Can I join you?"

This construct comes with a limitation : read returns 1 as its exit code, which will interrupt code running within set -e

read -d '' myVariable << EOF; echo $?; echo "$myVariable"
Hello world
EOF

1
Hello world

With set -e :

set -e; read -d '' myVariable << EOF; echo $?; echo "$myVariable"
Hello world
EOF

The script / current shell exits.
Workaround :

set -e; read -d '' myVariable << EOF || true; echo $?; echo "$myVariable"
Hello world
EOF

0
Hello world

A special case

Use this whenever you'd like a variable :

to be declared on several lines
- for readability
- to avoid the 80-column limit
but to be a single long line anyway

myVariable="A SQL query goes into a bar, \
walks up to two tables and asks, \
\"Can I join you?\""; echo "$myVariable"; echo $myVariable

A SQL query goes into a bar, walks up to two tables and asks, "Can I join you?"			echo works fine with 
A SQL query goes into a bar, walks up to two tables and asks, "Can I join you?"			 or without quotes

All these commands do the job :

a=2; a=$((a+1)); echo $a
a=2; ((a+=1)); echo $a
a=2; ((a++)); echo $a
a=2; let 'a=a+1'; echo $a
a=2; let 'a+=1'; echo $a
a=2; let 'a++'; echo $a

So far, no idea which is "better" or should be preferred —if any.

I've written myScript.sh, which outputs stuff to stdout and stderr like any regular script
Now, I'd like to :
- write all outputs into myScript.log
- still view the script output in the console while running
- make as few changes as possible to myScript.sh

solution	details	pro	con
use tee at launch	launch the script like this : ./myScript.sh \| tee myScript.log	short & simple	if you forget the tee part : no logs or you have to kill + relaunch the script
use redirections and tee in the script	replace all script lines : someCommand [arguments] with : someCommand [arguments] 2>&1 \| tee myScript.log	does the job but only for very short scripts	ugly unreadable on anything longer than a one-liner does not meet my minimal changes requirement
add a redirection at the beginning of the script	Add to the beginning of the script : #!/usr/bin/env bash exec 1>myScript.log 2>&1 [script code]	meets my minimal changes requirement effectively writes everything to myScript.log	but nothing appears in the console, albeit requested
redirection + tee hack (source)	Add to the beginning of the script : #!/usr/bin/env bash exec > >(tee myScript.log) 2>&1 see breakdown below [script code]	does the job meets all my requirements	will look like magic to many (including myself in a few weeks) I found a limitation

About this `exec > >(tee )` command :

Breakdown

What does this command do ?

exec > >(tee myScript.log) 2>&1

exec

exec alters file descriptors
the exec > someFile 2>&1 construct redirects all outputs of a script (stdout + stderr) into someFile :
- > someFile : redirects the standard output only
- 2>&1 : explicitly redirects the standard error to "where the standard input is"

tee myScript.log

tee reads data on its standard input
sends that data to :
- its standard output
- and to the myScript.log file

>()

this is a process substitution

the whole command :

STDOUT + STDERR          =====>          tee          ==+==>          write everything
of all commands                                         |             to logfile
of my script                                            |
                                                        |
                                                        +==>          send everything
                                                                      to console

Limitation

I made a script like this MWE :

#!/usr/bin/env bash

doThings() {
	echo "wait $1 second"
	echo 'ERROR' >&2
	sleep $1
	echo "done (wait $1)"
	}

main() {
	doThings 0.25s &
	doThings 0.5s &
	wait
	echo 'the end'
	}

main

wait 0.25s second
wait 0.5s second
ERROR
done (wait 0.25s)
ERROR
done (wait 0.5s)
the end

This logs nothing so far, but apart from this, everything is going extremely well.

If I add this exec > >(tee ) hack to the script :

#!/usr/bin/env bash

exec > >(tee test.log) 2>&1		just added this line

doThings() {
	echo "wait $1 second"
	echo 'ERROR' >&2
	sleep $1
	echo "done (wait $1)"
	}

main() {
	doThings 0.25s &
	doThings 0.5s &
	wait
	echo 'the end'
	}

main

wait 0.25s second
ERROR
wait 0.5s second
ERROR
done (wait 0.25s)
done (wait 0.5s)		execution is stuck here forever

this looks related to the wait : once commented, the script runs until the end, but the output becomes inconsistent :
```
the end
wait 0.25s second
wait 0.5s second
done (wait 0.25s)
done (wait 0.5s)
```
I think tee itself is seen as a background command, and wait is waiting for it to end, which won't happen while the script is still executing. Hence waiting forever.
tee staying alive looks like a known question (but no solution in these links )

Workaround :

#!/usr/bin/env bash

doThings() {
	echo "wait $1 second"
	echo 'ERROR' >&2
	sleep $1
	echo "done (wait $1)"
	}

main() {
	doThings 0.25s &
	doThings 0.5s &
	wait
	echo 'the end'
	}

main 2>&1 | tee test.log		use tee directly on main()

echo '-------------log--------------'	this is only to
cat test.log				show that output
echo '------------/log--------------'	and log are identical

wait 0.25s second
ERROR
wait 0.5s second
ERROR
done (wait 0.25s)
done (wait 0.5s)
the end
-------------log--------------
wait 0.25s second
ERROR
wait 0.5s second
ERROR
done (wait 0.25s)
done (wait 0.5s)
the end
------------/log--------------

Let's write 10 million numbers and play with the 10 first ones only !

With grep :

number=100000000; time (seq $number | grep -E '^.$'); time (seq $number | grep -E -m 9 '^.$')

1

9

real    0m1.035s
user    0m1.489s
sys     0m0.212s
1

9

real    0m0.001s
user    0m0.002s
sys     0m0.000s

Both commands :

write numbers from 1 to 100000000
output single-digit numbers
but while the 2^nd command stops after finding the 9, the 1^st command keeps reading the input until the end

While sed (read below) actually stops working after reading n lines of input, grep stops after finding n matching lines. If we specify -m 10 instead of -m 9 in the grep command above, grep will read the input until the end, searching for a 10^th match that doesn't exist.

With sed (inspired by) :

number=10000000; time (seq $number | sed -rn 's/^.$/X/'); time (seq $number | sed -rn 's/^.$/X/;10q')

real    0m0.556s
user    0m0.599s
sys     0m0.049s

real    0m0.001s
user    0m0.002s
sys     0m0.000s

Both commands :

write numbers from 1 to 10000000
replace single-digit numbers with X
but while the 2^nd command stops replacements after reading 10, the 1^st command keeps reading the input until the end

With awk :

number=10000000; time (seq $number | awk '$0 < 10'); time (seq $number | awk '$0 < 10; $0 > 10 {exit}')

1

9

real    0m1.396s
user    0m1.455s
sys     0m0.020s
1

9

real    0m0.002s
user    0m0.002s
sys     0m0.000s

Like previous examples :

the 1^st command checks ALL lines of input
while the 2^nd doesn't care once numbers are > 10

alternate solution :

number=10000000; time (seq $number | awk '$0 < 10'); time (seq $number | awk 'NR >= 10 {exit}; {print}')

If the input is made of a giant single line (concatenated/minified HTML/CSS/...) such as :

blablabla<startToken><stuffIWantToKeepOrRemove><endToken>blablabla

A workaround would be to :

replace <startToken> with \n<startToken>\n

do the same with <endToken>, so that the input becomes :

blablabla
<startToken>
<stuffIWantToKeepOrRemove>
<endToken>
blablabla

once tokens are shun on distinct lines, you may apply recipes below

from ...	until ...	keep	remove
start of file	token
token	end of file
token	token	startToken='Line 3'; endToken='Line 7' for i in {0..9}; do echo "Line $i"; done \| \ awk -v ST="$startToken" -v ET="$endToken" ' BEGIN { betweenTokens=0 } $0 ~ ST { betweenTokens=1 } $0 ~ ET { betweenTokens=0 } betweenTokens == 1 { print }' to include the "end token" line in the results, swap the `$0 ~ ET` and `betweenTokens == 1` lines see also this special case	startToken='Line 3'; endToken='Line 7' for i in {0..9}; do echo "Line $i"; done \| \ awk -v ST="$startToken" -v ET="$endToken" ' BEGIN { betweenTokens=0 } $0 ~ ST { betweenTokens=1 } $0 ~ ET { betweenTokens=0 } betweenTokens == 1 { next } { print }'
token	token + n lines		startToken='Line 3'; nbLinesToDelete=4 for i in {0..9}; do echo "Line $i"; done \| \ awk -v ST="$startToken" -v N="$nbLinesToDelete" ' BEGIN { matchingLineNumber = -9999 } $0 ~ ST { matchingLineNumber = NR; next } NR > matchingLineNumber && NR < (matchingLineNumber + N) \ { next } { print }' removes n lines including the line of the token to remove the line of the token AND the next n lines, change the highlighted `<` into `<=`

About the -9999 initialization value :

let's call y this initialization value
the NR > && NR < line becomes : if (NR > y) and (NR < y+N) then "remove line"
NR starts at 1 (NR > 0), so NR > y for all values of y such as y < 1
as for NR < y+N :
- with y=0 : true if NR < N
- with y=-1 : true if NR < N-1
- with y=-2 : true if NR < N-2
meaning : lines from the start of the input until the (N+y)^th are also removed (by mistake )
any value of y is safe until NR < y+N is true again
- considering NR > 0, this becomes : y+N > NR > 0
- then y+N > 1, and so N > 1-y
- -9999 works until we want to delete a block of 10000 lines

special case

Hypotheses (differences with the generic case above) :

the analyzed file may contain several occurrences of "start token / lines / end token", so we'll display each matching block followed by a spacer : =====
the "end token" may be generic and appear many times in the file, while not being related to the "start token"

startToken='block 2'; endToken='common block end'
for i in A B; do for j in {1..3}; do echo -e "block $j\nFOO - $i\n$endToken\nBAR - $i"; done; done | \
awk -v ST="$startToken" -v ET="$endToken" '
	BEGIN			{ betweenTokens=0 }
	$0 ~ ST			{ betweenTokens=1 }
	betweenTokens == 1	{ print }
	$0 ~ ET			{ if(betweenTokens == 1) print "====="; betweenTokens=0 }	avoid repeating the spacer
	'

an asterisk *
a TAB

an asterisk `*`

select matching lines

echo -e 'a*\nb\nc*' | grep '*'
echo -e 'a*\nb\nc*' | grep -E '\*'

a*
c*

select non-matching lines

echo -e 'a*\nb\nc*' | grep -v '*'
echo -e 'a*\nb\nc*' | grep -E v '\*'

Please note that, according to POSIX:

in basic regular expressions (BRE) :
	the asterisk looses its special meaning when used as the first character of the BRE or right after the caret (^).

When using Extended regular expressions (ERE) :
	this creates undefined behaviour. So grep '*foo' file searches for the substring "*foo" while grep -E '*foo' file is undefined.

(source)

a `TAB`

Just paste a TAB in the terminal :
- echo -e "line 1\nline 2\n\tline 3 has a [TAB]\nline 4" | grep "CTRL-v TAB"
- ```
echo -e "foo bar space1\nfoo\tbar\ttab1\nfoo	bar	tab2\nfoo bar space2" | grep 'o	b'
```
```
foo     bar     tab1
foo     bar     tab2
```
Sounds ok and not very reliable at the same time. This is our quick-n-dirty solution.
With character classes :
echo -e "line_1\nline_2\n\tline_3_has_a_[TAB]\nline_4" | grep '[[:blank:]]'

[:blank:] matches both TAB and SPACE, so it's helpless when it comes to match TAB only.

Use grep hacks :

with grep -P (GNU grep only) :

echo -e "line 1\nline 2\n\tline 3 has a [TAB]\nline 4" | grep -P '\t'

echo -e "foo bar space1\nfoo\tbar\ttab1\nfoo	bar	tab2\nfoo bar space2" | grep -P 'o\tb'

foo     bar     tab1
foo     bar     tab2

with the $-escape syntax (works only with single quotes) :

echo -e "line 1\nline 2\n\tline 3 has a [TAB]\nline 4" | grep $'\t'

echo -e "foo bar space1\nfoo\tbar\ttab1\nfoo	bar	tab2\nfoo bar space2" | grep $'o\tb'

foo     bar     tab1
foo     bar     tab2

before='fo'; after='ar'; echo -e "foo bar space1\nfoo\tbar\ttab1\nfoo	bar	tab2\nfoo bar space2" | grep $before$'o\tb'$after

foo     bar     tab1
foo     bar     tab2

With Awk :
echo -e "line 1\nline 2\n\tline 3 has a [TAB]\nline 4" | awk '/\t/'

The is the most portable solution, works on all shells and platforms.

set -u; a='whatever'; [ "$a" ] && echo foo || echo bar; [ -z "$a" ] && echo foo || echo bar; [ -n "$a" ] && echo foo || echo bar
```
foo
bar
foo
```
What matters here is that this causes no error, whatever the test results.
set -u; a=''; [ "$a" ] && echo foo || echo bar; [ -z "$a" ] && echo foo || echo bar; [ -n "$a" ] && echo foo || echo bar
```
bar
foo
bar
```
Different results but still no error .
set -u; unset a; [ "$a" ] && echo foo || echo bar; [ -z "$a" ] && echo foo || echo bar; [ -n "$a" ] && echo foo || echo bar
```
bash: a: unbound variable
```
No matter what you try to detect and handle the unset variable : Bash catches it before you and causes an error.

Define a default value :

set -u; unset a; b=${a:-}; [ "$b" ] || echo 'a is unset'; [ -z "$b" ] && echo 'a is unset'; [ -n "$b" ] || echo 'a is unset'

a is unset
a is unset
a is unset

Now you can catch the error and handle it yourself the way you like.

maxLength=10; echo {a..z} | tr -d ' ' | awk "{print substr(\$0,1,$maxLength)}"
maxLength=10; echo {a..z} | tr -d ' ' | colrm "$((maxLength+1))"
maxLength=10; echo {a..z} | tr -d ' ' | cut -c -"$maxLength"
cut -c counts bytes, not characters :
echo '1é2è3ê4ë' | cut -c -5
```
1é2�
```
maxLength=10; echo {a..z} | tr -d ' ' | grep -Eo "^.{$maxLength"}
maxLength=10; echo {a..z} | tr -d ' ' | sed -r "s/^(.{$maxLength}).*$/\1/"

Let's consider :

a command that outputs results in a single line :
echo 'value1 value2 value3'
- using echo to mimic any other command
- values are SPACE-separated
and I'd like these values stored in variables myVar1, myVar2 and myVar3

copy-paste-edit solution

output=$(echo 'value1 value2 value3')		intermediate variable because the command is supposed to be run only once
myVar1=$(echo "$output" | cut -d' ' -f1)
myVar2=$(echo "$output" | cut -d' ' -f2)
myVar3=$(echo "$output" | cut -d' ' -f3)

echo "$myVar1 $myVar2 $myVar3"

value1 value2 value3

alternate solution

read myVar1 myVar2 myVar3 < <(echo 'value1 value2 value3')

echo "$myVar1 $myVar2 $myVar3"

value1 value2 value3

What if values are not `SPACE`-separated and the output has many lines and , i.e. not-so-basic situation

The not-so-basic command that generates a multiline output with values having a leading text :
echo -e 'not interesting output\nboring\nboring again\nthe values are : value1, value2, value3.\nBORING !!!\nwill this end ?'
```
not interesting output
boring
boring again
the values are : value1, value2, value3.
BORING !!!
will this end ?
```
The same read hack can work here. All we have to do is mangle this output so that we get values (and values only) separated by SPACE. To do so, one of the weapons of choice is awk :
echo -e 'not interesting output\nboring\nboring again\nthe values are : value1, value2, value3.\nBORING !!!\nwill this end ?' | awk -F ':' '/the values are/ { $1=""; gsub(/[,\.]/, " ", $2); print; }'

Depending on the situation, different awk hacks will be necessary.

Putting it all together :

read myVar1 myVar2 myVar3 < <(echo -e 'not interesting output\nboring\nboring again\nthe values are : value1, value2, value3.\nBORING !!!\nwill this end ?' | awk -F ':' '/the values are/ { $1=""; gsub(/[,\.]/, " ", $2); print; }')

echo "$myVar1 $myVar2 $myVar3"

Operating systems differ on the way they handle the newline character :

OS family	newline character	displayed character	Character code
Unix (Linux, *BSD, Mac OS X)	LF	`\n`	`0x0a`
Apple (before Mac OS X)	CR	`\r`	`0x0d`
MS-DOS, Windows	CR+LF	`\r\n`	`0x0d0a`

Files can be converted with utilities :

dos2unix / unix2dos : these utilities are usually installed by default on Red Hat systems
tofrodos : this is available as a Debian package (man page)

Convert files from DOS to Unix :

convert and replace file :
- dos2unix file
- fromdos file
convert and create new file :
- dos2unix -o originalFile newFile
- fromdos -b file (file will be renamed as file.bak)

Here, a pin code is a digits-only string (some people call this a "number" ) designed to be used as a password. It must be non-obvious (hence not your birth date nor your car plate / phone number / health insurance ID), and since we (humans) really suck with randomness, let's get help from a computer :

pinCodeLength=10; pinCode=''; while [ "${#pinCode}" -le "$pinCodeLength" ]; do pinCode+=$RANDOM; done; echo "$pinCode" | head -c "$pinCodeLength"

I have :

My favorite fruit is :
apples (love the red ones ;-)

On the back of my phone, you can see an...
apple-shaped logo.

New-York is nicknamed :
the "big apple" since the 1970s.

I want :

My favorite fruit is :
apples (love the red ones ;-)

On the back of my phone, you can see an...
android-shaped logo.

New-York is nicknamed :
the "big apple" since the 1970s.

echo -e "My favorite fruit is :\napples (love the red ones ;-)\n\nOn the back of my phone, you can see an...\napple-shaped logo.\n\nNew-York is nicknamed :\nthe \"big apple\" since the 1970s." | awk 'BEGIN { matchingLineNumber=-1 }; /phone/ { matchingLineNumber=NR }; NR == matchingLineNumber+1 { gsub(/apple/, "android", $0) }; { print }'

The overall concept is pretty similar to this example. Consider the unfolded Awk code :

BEGIN				{ matchingLineNumber=-1 };		just initializing a variable
/phone/				{ matchingLineNumber=NR };		upon finding our needle, just set this variable with the current line number
NR == matchingLineNumber+1	{ gsub(/apple/, "android", $0) };	on the next line, do the substitution
				{ print }				and print the current line

It is sadly frequent that we have to dive into old + cluttered + hardly readable code, that is not even properly indented (or not indented at all). One of the first things to do, then, is to improve readability with proper indentation. This won't magically turn lead into gold, but it helps anyway. Especially if it can be done effortlessly .

This is what reIndentCode.sh is about. Turning this :

a ( b
c d [ e ] f [
g h { j (
k ) } l m
n ] o ) p
q r

into this :

a ( b
	c d [ e ] f [
		g h { j (
				k ) } l m
		n ] o ) p
q r

This shell script is a quick-n-dirty solution I implemented when dealing with a 50KiB Perl script going awry. I bet it has many limitations, but it can be a starting point for the next time or anybody in a similar situation.

Sometimes you end in a situation where you can not write into /this/is/a/very/long/pathTo/myFile, even though myFile has the proper bits set. So it would be interesting to have a clear picture of owner + group + permissions, from the top, down to myFile.

The solution below works fine, so I'll leave it here —maybe it can become an inspiration for future needs— but there's actually a single command to do the very same thing : namei .

fullPathToStudy='/this/is/a/very/long/pathTo/myFile'; nbFields=$(echo "$fullPathToStudy" | grep -o '/' | wc -l); output=''; for ((i=1; i<=nbFields+1; i++)); do currentPath="$(echo "$fullPathToStudy" | cut -d '/' -f 1-$i)/"; [ -d "$currentPath" ] && output+="\n$(ls -ld "$currentPath")"; done; [ -f "$fullPathToStudy" ] && output+="\n$(ls -l "$fullPathToStudy")"; echo -e "$output" | awk '{print $1" "$3" "$4" "$NF}' | column -s ' ' -t

drwxr-xr-x		root		/
drwxr-xr-x		root		/this/
drwxr-xr-x	bob	developers	/this/is/
drwxr-xr-x	bob	developers	/this/is/a/
drwxr-xr-x	bob	developers	/this/is/a/very/
drwx------	bob	developers	/this/is/a/very/long/
drwx------	bob	developers	/this/is/a/very/long/pathTo/
-rw-------	bob	developers	/this/is/a/very/long/pathTo/myFile

Let's consider a file such as :

Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Etiam mollis viverra ligula,
Lorem ipsum dolor sit amet,
ut luctus magna imperdiet eget.
Ut consectetur laoreet venenatis.
Nulla euismod sapien nec sodales tempor.
Lorem ipsum dolor sit amet,
Suspendisse sagittis odio eu urna imperdiet,
vitae sollicitudin ante mattis.

How can I spot the duplicated lines ?

echo -e "Lorem ipsum dolor sit amet,\nconsectetur adipiscing elit.\nEtiam mollis viverra ligula,\nLorem ipsum dolor sit amet,\nut luctus magna imperdiet eget.\nUt consectetur laoreet venenatis.\nNulla euismod sapien nec sodales tempor.\nLorem ipsum dolor sit amet,\nSuspendisse sagittis odio eu urna imperdiet,\nvitae sollicitudin ante mattis." | sort | uniq -c | sort -n | awk '$1 > 1 && $2 !~ "^(#.*)?$" {print}'

Same as above, with contents stored in a file :

sort fileWithDuplicateLines | uniq -c | sort -n | awk '$1 > 1 && $2 !~ "^(#.*)?$" {print}'

Considering this piece of code below, how can I remove the comments ?

not commented
/*comment part 1
comment part 2
comment part 3*/ but I want to keep the end of this line
not commented either

This snippet will be fed into lines hereafter :

echo -e 'not commented\n/*comment part 1\ncomment part 2\ncomment part 3*/ but I want to keep the end of this line\nnot commented either'

There are plenty of different / context-specific / incompatible / incomplete ways to do this

tr | sed | tr method :

The idea is to :

turn the whole input into a single giant line
remove the comments
turn the remainings back into distinct lines

echo -e 'not commented\n/*comment part 1\ncomment part 2\ncomment part 3*/ but I want to keep the end of this line\nnot commented either' | tr '\n' 'X' | sed -r 's|(.*)/\*.*?\*/(.*)|\1\2|g' | tr 'X' '\n'

This won't work if there are several distinct multi-line comments :

echo -e 'not commented\n/*comment 1 part 1\ncomment 1 part 2\ncomment 1 part 3*/ but I want to keep the end of this line\nnot commented either\n/* comment 2 part 1\ncomment 2 part 2\ncomment 2 part 3*/, keep this\nnot commented either.' | tr '\n' 'X' | sed -r 's|(.*)/\*.*?\*/(.*)|\1\2|g' | tr 'X' '\n'

Other (better !) method (inspired by) :

cat << EOF | sed 's|/\*|\n&|g; s|*/|&\n|g' | sed '/\/\*/,/*\//d' | sed '/^$/d'
not commented
/*comment 1 part 1
comment 1 part 2
comment 1 part 3*/ but I want to keep the end of this line
not commented either
/* comment 2 part 1
comment 2 part 2
comment 2 part 3*/, keep this
not commented either.
EOF

Details :

's|/\*|\n&|g

change whatever_before/*comment into

whatever_before
/*comment

making the /* the first characters of a new line

s|*/|&\n|g'

change comment*/whatever_after into

comment*/
whatever_after

sed '/\/\*/,/*\//d'

delete block of lines starting with /* and ending with */ (i.e. the comments we just isolated on distinct lines)

sed '/^$/d'

delete empty lines

echo 'Hello world' | grep -o 'l' | wc -l

DON'T :
- 'foo\'bar'
- ... (there are plenty of other creative ways of failing at this )
DO : 'foo'"'"'bar'

Explanation of how `'"'"'` is interpreted as just `'` :

There are 3 successive quoted strings : aaa, b and ccc :

 aaa  b  ccc
'foo'"'"'bar'
^   ^^^^^   ^
1   23456   7

1 (') : start 1^st quotation (aaa) using single quotes
2 (') : end 1^st quotation
3 (") : start 2^nd quotation (b) using double-quotes
4 (') : quoted character, the one we wanted to escape
5 (") : end 2^nd quotation
6 (') : start 3^rd quotation (ccc) using single quotes
7 (') : end 3^rd quotation

The good ol'method :

myString='hello'; myString="$myString world"; echo "$myString"

hello world

The `+=` method :

This is a Bash-specific construct (i.e. /bin/bash only. It will fail in scripts starting with #!/bin/sh & al.)
Not supported in 3.x Bash versions (exact version not found, feel free to investigate the Bash source code)

myString='hello'; myString+=" world"; echo "$myString"
```
hello world
```
Whatever the variable type (even though they look like integers), += concatenates :
value=42; value+=31; echo "$value"
```
4231
```
To workaround this and actually sum, use let :
value=42; let value+=31; echo "$value"
```
73
```
You can also subtract, multiply and divide :
value=42; let value-=40; echo "$value"; value=42; let value*=4; echo "$value"; value=42; let value/=2; echo "$value"
```
2
168
21
```

Please, please, PLEASE : don't parse the output of ls !!!

There are dedicated + convenient + reliable + simple commands to get a file modification date :

stat -c %y someFile
2018-11-21 10:37:05.683199311 +0100
date -r someFile
Wed Nov 21 10:37:05 CET 2018
date -r someFile +%F_%H-%M-%S
2018-11-21_10-37-05

Solution 1 (badness=1) :

command1 > "/path/to/logFile" 2>&1
command2 >> "/path/to/logFile" 2>>&1
command3 >> "/path/to/logFile" 2>>&1

Does the job but will clutter your script making it barely readable. Ok for very short scripts.

Solution 2 (badness=100) :

(
command1
command2
command3
) > "/path/to/logFile" 2>&1

This hack looks so awkward I honestly couldn't have imagined that myself. I saw that in a 1000+ lines non-indented shell script full of UUoC, backticks value=`command`, parsing the output of ls, ...

jump to the solution

more about compound commands
more about output redirection
more about exec

#!/usr/bin/env bash

showResult() {
	local blockType=$1
	echo "======== cat '$logFile' after '$blockType' block"
	cat "$logFile"
	echo '======== /cat'
	rm "$logFile"
	echo
	}


logFile="$0.log"
myVariable='initial value'

########## with '()' ##########
#	==> creates a subshell
echo "before () : $myVariable"
(
	myVariable='changed inside ()'
	echo 'hello'
	echo 'world'
	echo "in () : $myVariable"
) > "$logFile"
echo "after () : $myVariable"

showResult '()'


########## with '{}' ##########
#	==> no subshell
echo "before {} : $myVariable"
{
	myVariable='changed inside {}'
	echo 'hello'
	echo 'world'
	echo "in () : $myVariable"
} > "$logFile"
echo "after {} : $myVariable"

showResult '{}'


########## with 'exec' ##########
#	==> uses file descriptors
echo "before exec : $myVariable"
exec 10>&1 20>&2 1>"$logFile" 2>&1

myVariable='changed inside exec'
echo 'hello'
echo 'world'
echo "in exec : $myVariable"

exec 2>&20 20>&- 1>&10 10>&-
echo "after exec : $myVariable"

showResult 'exec'

before () : initial value
after () : initial value
======== cat './test.sh.log' after '()' block
hello
world
in () : changed inside ()
======== /cat

before {} : initial value
after {} : changed inside {}
======== cat './test.sh.log' after '{}' block
hello
world
in () : changed inside {}
======== /cat

before exec : changed inside {}
after exec : changed inside exec
======== cat './test.sh.log' after 'exec' block
hello
world
in exec : changed inside exec
======== /cat

exec can be used to redirect all the outputs. If, at some point of the script, we want to "stop redirecting the outputs", we have to (source) :

step	description	command	file descriptors
0	before any redirection	(n/a)	1 : /dev/stdout
1	prepare for the recovery, then redirect	exec 3>&1 1>logFile	3 : /dev/stdout 1 : logFile
2	use the output redirection : anything that should normally be written to screen goes to logFile	any command you like	3 : /dev/stdout 1 : logFile
3	recover the original standard output (i.e. "stop redirecting")	exec 1>&3 3>&-	1 : /dev/stdout 3 : closed. nowhere ?

Named file descriptors (aka "automatic file descriptor allocation")

Based on the example above :

I don't know whether there are restrictions (or risk of collisions) on file descriptor numbers...
file descriptors 3 to 9 are available (source)
starting from Bash 4.1 (May 2010) (source), it is possible (recommended?) to use "automatic file descriptor allocation" instead of picking numbers manually (source) :

#!/usr/bin/env bash

outFile=$(mktemp)

echo 'foo'

exec {fileDescriptor}>&1 1>"$outFile"
echo 'bar'

exec 1>&${fileDescriptor} {fileDescriptor}>&-
echo 'baz'

cat "$outFile"
rm "$outFile"
echo "The chosen file descriptor was : '$fileDescriptor'"

foo
baz
bar
The chosen file descriptor was : '10'

x&<y and x&>y both mean "make x a copy of y". The only difference is that they respectively refer to an input and output file descriptor.
if y is -, x will be closed.

source

Snippet to copy-paste :

exec {previousStdout}>&1 {previousStderr}>&2 1>"/path/to/logFile" 2>&1

(script content goes here)

exec 2>&${previousStderr} {previousStderr}>&- 1>&${previousStdout} {previousStdout}>&-

myScript.sh is doing its job fine
I'd like it to have a slightly different behavior depending on whether it's been fired
- interactively : i.e. from a terminal
- non-interactively : by at or cron

As most (if not all) questions found here, this question arose from a real-life situation. That time, I had to fix an old script written by someone who's not even part of the company anymore...
My readings and the tests I made lead me to the conclusion that asking this question is the sign of poor script design. Functionalities like :

stdout + stderr on the shell side
the MAILTO crontab variable

should be enough to design a script that works fine —exactly the same way— whatever method is used to start it : manually / at / cron.
For best practices, have a look at :

The Unix philosophy
Some basic design rules and especially the rule of silence (which is generally broken by such badly designed scripts).

Let's try some methods and check whether they answer our question :

with $PS1
with $-
with [ -t fd ]

#!/usr/bin/env bash
#	To fire this script via 'at' :
#	at $(date --date "now +1 minutes" '+%H%M') -f myScript.sh

exec 1> output.txt

########################################## ##########################################################

# source
[ -z "$PS1" ] && interactive='no' || interactive='yes'; echo "interactive 1 : '$interactive'"

# when run :
#	manually			==>	yes
# 	in a script			==>	no
# 	in a script fired by 'at'	==>	yes
# 	in a script fired by 'cron'	==>	no


########################################## ##########################################################

# source
case $- in *i*) interactive='yes' ;; *) interactive='no' ;; esac; echo "interactive 2 : '$interactive'"

# when run :
#	manually			==>	yes
# 	in a script			==>	no
# 	in a script fired by 'at'	==>	no
# 	in a script fired by 'cron'	==>	no


########################################## ##########################################################

echo "dollarDash : '$-'"
# when run :
#	manually			==>	himBHs
# 	in a script			==>	hB
# 	in a script fired by 'at'	==>	s
# 	in a script fired by 'cron'	==>	hB

There is no easy / standard / reliable way to distinguish use cases, but there are some workarounds :

set a variable in the crontab and check it in your script (source)
investigate the process tree until you find (or not) cron's PID (source)

	Line to column	Column to line
"Single" input	fieldSeparator=';'; echo "foo${fieldSeparator}bar${fieldSeparator}baz" \| tr "$fieldSeparator" '\n'	echo -e 'foo\nbar\nbaz' \| xargs fieldSeparator=';'; echo -e 'foo\nbar\nbaz' \| xargs \| tr ' ' "$fieldSeparator"
"Multiple" input	See https://unix.stackexchange.com/questions/520031/pivot-file-values#answer-520047

Append these functions to ~/.bash_aliases :

getFileSnippet() {
	local fileToInspect=$1
	local startLine=$2
	local stopLine=$3
	tmpFile=$(mktemp --tmpdir='/run/shm' tmp.XXXXXXXX)
	sed -n "$startLine,${stopLine}p" "$fileToInspect" > "$tmpFile"
	echo "$tmpFile"
	}

compareSnippets() {
	[ $# -ne 6 ] && { echo 'Wrong number of arguments, 6 expected.'; return $UNIX_FAILURE; }
	local fileToInspect1=$1
	local startLine1=$2
	local stopLine1=$3
	local fileToInspect2=$4
	local startLine2=$5
	local stopLine2=$6

	for argumentToCheck in fileToInspect1 fileToInspect2; do
		[ -f "${!argumentToCheck}" ] || { echo "Argument '${!argumentToCheck}' is not a file."; return $UNIX_FAILURE; }
	done

	snippet1=$(getFileSnippet "$fileToInspect1" "$startLine1" "$stopLine1")
	snippet2=$(getFileSnippet "$fileToInspect2" "$startLine2" "$stopLine2")
	diff "$snippet1" "$snippet2"
	rm "$snippet1" "$snippet2"
	}

compare code snippets with :
compareSnippets fileA 117 124 fileB 159 166

I have :

foo1
bar1
foo2 REMOVE THIS LINE AND THE FOLLOWING ONE
bar2
foo3
bar3

I want :

foo1
bar1
foo3
bar3

echo -e 'foo1\nbar1\nfoo2 REMOVE THIS LINE AND THE FOLLOWING ONE\nbar2\nfoo3\nbar3' | awk 'BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}'

Let's consider the awk '' part remembering :

it's made of ;-separated rules. Each rule is evaluated separately against each input line
rules are formatted like : condition {action}. condition can be a regular expression match or a variable check

BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}

becomes :

BEGIN				{ matchingLineNumber = -1 }
/REMOVE/			{ matchingLineNumber = NR; next }
NR == matchingLineNumber + 1	{ next }
				{ print }

line 1 : just initializing a variable. Setting it to 0 would make our 3^rd awk line affect the line 1 of the input. Hence the -1
line 2 : if the current input line matches the 'REMOVE' regexp, keep the line number, stop processing rules and continue to the next input line
line 3 : if the current input line directly follows a line matching the regexp, stop processing rules and continue to the next input line
line 4 : no condition, just print the current input line

It works whatever the position of the matching line within the input :

echo -e 'foo2 REMOVE THIS LINE AND THE FOLLOWING ONE\nbar2\nfoo1\nbar1\nfoo3\nbar3' | awk 'BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}'

echo -e 'foo1\nbar1\nfoo3\nbar3\nfoo2 REMOVE THIS LINE AND THE FOLLOWING ONE\nbar2' | awk 'BEGIN {matchingLineNumber=-1}; /REMOVE/ {matchingLineNumber=NR; next}; NR==matchingLineNumber+1 {next}; {print}'

While building a one-liner, one of the commands outputs :

a b

whereas the next command expects :

b a

How may I swap them ?

echo 'a b' | awk '{print $2" "$1}'
echo 'a b' | sed -r 's/(.+) +(.+)/\2 \1/'

myVariable=$(echo {a..c} | tr ' ' '\n'); echo "myVariable : '$myVariable'"

myVariable : 'a
b			definitely a multiline variable
c'

echo "$myVariable" | while read aSingleLineOfMyVariable; do echo "a single line : '$aSingleLineOfMyVariable'" done; echo "last value : '$aSingleLineOfMyVariable'"

a single line : 'a'	works fine inside the loop
a single line : 'b'
a single line : 'c'
last value : ''		undefined variable since it only exists in the subshell created while piping

tmpFile=$(mktemp); echo "$myVariable" > "$tmpFile"; while read aSingleLineOfMyVariable; do echo "a single line : '$aSingleLineOfMyVariable'" done < "$tmpFile"; echo "last value : '$aSingleLineOfMyVariable'"; rm "$tmpFile"

a single line : 'a'
a single line : 'b'
a single line : 'c'
last value : ''		$aSingleLineOfMyVariable is lost when leaving the loop

Nice try but :

no improvement since the previous test
creating a temporary file is not very elegant and can have a performance cost if repeated numerous times
still can't catch the last value and use it outside of the loop

while IFS= read aSingleLineOfMyVariable; do echo "a single line : '$aSingleLineOfMyVariable'" done < <(printf '%s\n' "$myVariable"); echo "last value : '$aSingleLineOfMyVariable'"

a single line : 'a'
a single line : 'b'
a single line : 'c'
last value : ''		$aSingleLineOfMyVariable is lost again, continue reading

Whatever command is (including something like echo "$someVariable"), the <(command) construct (aka process substitution) expands to a file and, as such, can be fed into anything with < or >.

previousIfs="$IFS"; IFS=$'\n'; for aSingleLineOfMyVariable in $myVariable; do echo "a single line : '$aSingleLineOfMyVariable'" done; echo "last value : '$aSingleLineOfMyVariable'"; IFS="$previousIfs"

a single line : 'a'
a single line : 'b'
a single line : 'c'
last value : 'c'	$aSingleLineOfMyVariable is not lost this time

This comment says this is because while loops create a subshell, whereas for loops don't. I'm afraid this is wrong... (it is !)

Let's check this :

Test #1 :

tmpFile=$(mktemp); echo -e 'a\nb\nc' > "$tmpFile"; while read item; do echo "item (during loop) : $item"; done < "$tmpFile"; echo "item (after loop) : $item"; rm "$tmpFile"

item (during loop) : a
item (during loop) : b
item (during loop) : c
item (after loop) :		empty variable

for item in {a..c}; do echo "item (during loop) : $item"; done; echo "item (after loop) : $item"

item (during loop) : a
item (during loop) : b
item (during loop) : c
item (after loop) : c		still exist outside of the loop

Looks like it was right, after all ?

Test #2 :

for i in 'in for loop'; do myVariable='foo'; echo "$i"; done; echo "myVariable : $myVariable"

in for loop
myVariable : foo

A variable set within a for loop still exists after the loop.

while true; do myVariable='bar'; echo 'in while loop (1)'; break; done; echo "myVariable : $myVariable"; while [ -z "$i" ]; do myVariable='baz'; echo 'in while loop (2)'; i=1; done; echo "myVariable : $myVariable"

in while loop (1)
myVariable : bar
in while loop (2)
myVariable : baz

A variable set within a while loop still exists after the loop, whichever way the loop ends : break or regular exit. This proves the comment linked above WRONG !

while read item; do myVariable=meu; echo "item : '$item', myVariable : '$myVariable'"; done < <(echo -e 'ga\nbu\nzo'); echo "item : '$item', myVariable : '$myVariable'"

item : 'ga', myVariable : 'meu'
item : 'bu', myVariable : 'meu'
item : 'zo', myVariable : 'meu'
item : '', myVariable : 'meu'

Trying to workaround with a process substitution makes no difference

unset myVariable; for i in whatever; do read myVariable; echo "myVariable : '$myVariable'"; done; echo "myVariable : '$myVariable'" or : unset myVariable i; while [ -z "$i" ]; do read myVariable; echo "myVariable : '$myVariable'"; i=foo; done; echo "myVariable : '$myVariable'" + any +

myVariable : 'any'
myVariable : 'any'

A variable set with read, both in for and while loops, survives the end of the loop.

tmpFile=$(mktemp); echo -e 'ga\nbu\nzo' > "$tmpFile"; while read item; do myVariable=meu; echo "item : '$item', myVariable : '$myVariable'"; done < "$tmpFile"; echo "item : '$item', myVariable : '$myVariable'"; rm "$tmpFile"

item : 'ga', myVariable : 'meu'
item : 'bu', myVariable : 'meu'
item : 'zo', myVariable : 'meu'
item : '', myVariable : 'meu'

A variable set with a while read construct doesn't survive the end of the loop. Would that be the reason ?

The revelation (source) :

Indeed, the while read construct is the explanation of this behavior. While consuming lines of input, while read myVariable...

catches a line of input and stores it in myVariable
if it succeeds (i.e. not the end of the input), the while loop continues normally
otherwise (i.e. no more input data) :
1. myVariable gets an empty value
2. this makes read return a UNIX_FAILURE
3. the loop condition is false : the while loop ends
once outside of the while loop, myVariable looks empty. It's actually been overwritten with an empty value by the last read

Consider script.sh that sources functions.sh like this :

. ./functions.sh

This implies script.sh and functions.sh are in the same directory, and it works only if script.sh is launched from its own directory. Otherwise, the relative path ./functions.sh can not be resolved because ./ is interpreted as "the directory from which the command is launched".

To workaround this, you can automatically translate the relative path into an absolute path before source-ing :

# Include an external file even though the current script is not launched from its own directory
directoryOfThisScript="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
. "$directoryOfThisScript/functions.sh"

Full snippet for a script that has several files to source + error management :

######################################### includes ##################################################
nameOfThisScript=$(basename "${BASH_SOURCE[0]}")
directoryOfThisScript="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

configFile="$directoryOfThisScript/$nameOfThisScript.conf"
functionsFile="$directoryOfThisScript/functions.sh"

for fileToSource in "$configFile" "$functionsFile"; do
	source "$fileToSource" 2>/dev/null || {
		echo "File '$fileToSource' not found"
		exit 1
		}
done
######################################### /includes #################################################

This works fine until the script is symlinked and launched via a link.

jump to the solution

#!/usr/bin/env bash

tmpDir=$(mktemp -d --tmpdir zzzMyTmpDir.XXXXXXXX)

subDir="$tmpDir/subDir"
theScript="$subDir/script.sh"
theConfigFile_basename='config.sh'; theConfigFile="$subDir/$theConfigFile_basename"
theLinkToTheScript="$tmpDir/linkToScript"

mkdir -p "$subDir"
cat << EOF > "$theScript"
#!/usr/bin/env bash

echo "Hello world, I've been launched with (\\\$0) : \$0"

if [ -h "\$0" ]; then
	echo 'I am a symlink'
	theScriptImRunning=\$(readlink -f \${BASH_SOURCE[0]})	theScriptImRunning is the target of the link
else
	echo 'I am a regular file'
	theScriptImRunning=\${BASH_SOURCE[0]}			theScriptImRunning is 'myself'
fi

directoryOfThisScript="\$( cd "\$( dirname "\$theScriptImRunning" )" && pwd )"
source "\$directoryOfThisScript/$theConfigFile_basename"
echo -e "a = \$a\n"
EOF

cat << EOF > "$theConfigFile"
#!/usr/bin/env bash
a=12EOF

chmod +x "$theScript"
ln -sf "$theScript" "$theLinkToTheScript"

"$theScript"			actual script, launched via its absolute path
"$theLinkToTheScript"		script launched via a symlink

# clean before leaving
cd "$tmpDir/.."
[ -d "$tmpDir" ] && rm -r "$tmpDir"

Hello world, I've been launched with ($0) : /tmp/zzzMyTmpDir.2A5nws0E/subDir/script.sh
I am a regular file
a = 12

Hello world, I've been launched with ($0) : /tmp/zzzMyTmpDir.2A5nws0E/linkToScript
I am a symlink
a = 12

So the solution is...

# Include an external file even though the current script is launched
# - not from its own directory
# - via a symlink

[ -h ${BASH_SOURCE[0]} ] && theScriptImRunning="$(readlink -f ${BASH_SOURCE[0]})" || theScriptImRunning=${BASH_SOURCE[0]}
directoryOfThisScript="$( cd "$( dirname "$theScriptImRunning" )" && pwd )"

. "$directoryOfThisScript/config.sh"

There are several ways to do so :

return + $?

For integers only (and "not too big" ones, preferably). Actually, the returned number will be n modulo 256 :

foo() { return $i; }; for i in {254..258}; do foo $i; echo "$i $?"; done

echo + $()

For anything you like : numbers, strings, ...

#!/usr/bin/env bash

functionThatReturns() {
	return 42
	}

functionThatEchoes() {
	echo 42
	}

functionThatReturns
echo "1. '$?'"

result=$(functionThatReturns)
echo "2. '$result'"

functionThatEchoes
echo "3. '$?'"

result=$(functionThatEchoes)
echo "4. '$result'"











1. '42'		return + $? = OK


2. ''		return + $() = KO
42		this is the echo made in the function

3. '0'		this only means the function ended successfully


4. '42'		echo + $() = OK

Command	When `haystack` has `needle` ...
	as the 1^st character	0 time	exactly 1 time	more than 1 time
	haystack='hello world'; needle='h'	haystack='hello world'; needle='z'	haystack='hello world'; needle='w'	haystack='hello world'; needle='o'
echo "$haystack" \| grep -o a b "$needle" \| grep -o E '[0-9]+'	0 0-based index	UNIX_FAILURE	6 0-based index	4 7 0-based index
printf '%s\n' "$haystack" \| grep -o . \| grep -n "$needle" \| grep -o E '[0-9]+'	1 1-based index	UNIX_FAILURE	7 1-based index	5 8 1-based index
tmp="${haystack%%$needle*}"; echo ${#tmp}	0 0-based index	11 length of `$haystack`	6 0-based index	4 position of 1st occurrence, 0-based index
echo "$haystack" \| awk -v x="$needle" '{ print index($0, x) }'	1 1-based index	0 Looks like a reliable "not found" indicator	7 1-based index	5 position of 1st occurrence, 1-based index

Let's consider a command returning 10 lines of text, from which we want to display only the 1^st, 3^rd and 8^th lines.

With grep :
- let's ask grep to add line numbers, then select the lines to display (again with grep), then hide the line numbers :
  for i in {a..j}; do echo "Line $i"; done | grep -n '' | grep -E '^(1|3|8):' | sed -r 's/^[0-9]+://'
- as above, but this time, the numbering is made by nl (the end of the command also changes because nl formats its output with TABs) :
  for i in {a..j}; do echo "Line $i"; done | nl | grep -E '(1|3|8)[[:blank:]]' | sed -r 's/^ *[0-9]+\t//'
Even simpler, with sed :
for i in {a..j}; do echo "Line $i"; done | sed -n '1p; 3p; 8p'

Here are a few snippets with :

myString='0123456789'
n=3

This does not always come easy in the command lines :

sometimes the right value is length of string - 3
or n has to be incremented / decremented because
- the start character is either included or excluded
- or the string is 0/1-indexed
or both

Some of the commands shown below are pretty awkward and appear here for the sake of completeness only.

	keep	remove
leading	echo ${myString::3} echo $myString \| cut -c -3 echo $myString \| sed -r 's/^(.{3}).*/\1/'	echo ${myString:3} echo $myString \| cut -c 4- echo $myString \| sed -r 's/.*(.{3})$/\1/'
trailing	echo ${myString:(-3):3} echo $myString \| cut -c 8- echo $myString \| sed -r 's/.*(.{3})$/\1/'	echo ${myString:0:-3} echo $myString \| cut -c -7 echo $myString \| sed -r 's/(.*).{3}$/\1/'

UNIX_SUCCESS=0
UNIX_FAILURE=1

isInteger() {
	[[ $1 =~ ^[-+]?[0-9]+$ ]] && echo $UNIX_SUCCESS || echo $UNIX_FAILURE
	}

[ $(isInteger "$myVariable") -eq "$UNIX_FAILURE" ] && {
	(shout how unhappy you feel)
	}

You could try :

myString='hello'
length=$((${#myString}-1))
for i in $(eval echo "{0..$length}"); do
	echo $i : ${myString:i:1}
done

But this is even better (source) :

myString='hello'
for((i=0; i<${#myString}; i++)); do
	echo $i : ${myString:i:1}
done

Use eval.

If you want to store

the result of a command
but not the command itself

into a variable, just use the command substitution construct $(command) :

now=$(date); echo "'now' was : $now"

With a for loop
With a case construct
With a regular expression

With a `for` loop :

To make sure a command line parameter matches a value from a list :

#!/usr/bin/env bash

valueToCheck="$1"
listOfAcceptedValues='foo bar baz'

UNIX_SUCCESS=0
UNIX_FAILURE=1
valueToCheckIsInTheList=$UNIX_FAILURE

for value in $listOfAcceptedValues; do
	echo "Testing '$valueToCheck' against '$value'"
	[ "$valueToCheck" == "$value" ] && { echo "'$valueToCheck' is a valid value."; valueToCheckIsInTheList=$UNIX_SUCCESS; break; }
done

[ "$valueToCheckIsInTheList" -eq "$UNIX_FAILURE" ] && { echo "'$valueToCheck' is not a valid value."; exit $UNIX_FAILURE; }

With a `case` construct :

This suits cases where different input values imply different behaviors of the script. Otherwise, using case may be overkill, and the for loop method may be more adapted.

#!/usr/bin/env bash

valueToCheck="$1"

case "$valueToCheck" in
	'foo')
		echo "'$valueToCheck' is a valid value."
		# do something with 'foo'
		;;
	'bar')
		echo "'$valueToCheck' is a valid value."
		# do something different with 'bar'
		;;
	'baz')
		echo "'$valueToCheck' is a valid value."
		# do something different again with 'baz'
		;;
	*)
		echo "'$valueToCheck' is not a valid value."
		# deal with it !!!
		;;
esac

With a regular expression :

This method may return false positives if used improperly. Indeed, to check whether foo is within foo bar baz, we can "regexp match" foo bar baz against foo (it matches), but fo and f also match.

It _may_ sound more logical to do it the other way round : matching foo against foo bar baz, but this obviously can't work.

To workaround false positive matches of fo and f, we must use "word boundary detectors" :

(^|[[:space:]]) and ($|[[:space:]]) (which is very well explained here) (this is possibly wrong or obsolete)
or \b

#!/usr/bin/env bash

listOfAcceptedValues='foo bar baz'
listOfValuesToCheck="foo bar baz poo 123 bam ofo fo f ''"

for valueToCheck in $listOfValuesToCheck; do
	[[ "$listOfAcceptedValues" =~ "$valueToCheck" ]] && result1='' || result1=' not'
	[[ "$listOfAcceptedValues" =~ (^|[[:space:]])"$valueToCheck"($|[[:space:]]) ]] && result2='' || result2=' not'
	echo -e "'$valueToCheck'\tis : (1)$result1 a valid value, \t(2)$result2 a valid value."
done

'foo'	is : (1) a valid value,		(2) a valid value.
'bar'	is : (1) a valid value,		(2) a valid value.
'baz'	is : (1) a valid value,		(2) a valid value.
'poo'	is : (1) not a valid value,	(2) not a valid value.
'123'	is : (1) not a valid value,	(2) not a valid value.
'bam'	is : (1) not a valid value,	(2) not a valid value.
'ofo'	is : (1) not a valid value,	(2) not a valid value.
'fo'	is : (1) a valid value,		(2) not a valid value.
'f'	is : (1) a valid value,		(2) not a valid value.
''''	is : (1) not a valid value,	(2) not a valid value.

Other example :

validUsers='kevin stuart bob'; for user in alice bob bobby; do regex="\b$user\b"; [[ "$validUsers" =~ $regex ]] && echo "'$user' is valid" || echo "'$user': NOPE"; done

'alice': NOPE
'bob' is valid
'bobby': NOPE

Other methods :

There are other methods, but they don't all apply to the same use cases, especially if they have to check untrusted values (i.e. user input) which may contain wildcards or stuff like that.

Repeat a single character :

With Perl (source) :

perl -e 'print "X"x42; print "\n"'

Check :

generatedString=$(perl -e 'print "X"x42; print "\n"'); echo $generatedString; echo ${#generatedString}

With Python :

python -c "print('X' * 42)"

Check :

generatedString=$(python -c "print('X' * 42)"); echo $generatedString; echo ${#generatedString}

With Bash (source) :

A short and elegant solution :

printf 'X%.0s' {1..20}

Alternate solutions for a variable number of repeats :

printf-based :
- n=4; printf '🍭%.0s' $(eval "echo {1..$n}")
- n=4; printf '🧸%.0s' $(seq 1 $n)
This construct is not able to repeat anything 0 times.
echo-based :
- n=4; for((i=0;i<$n;i++)); do echo -n 'X'; done

Repeat a string :

With Perl :

perl -e 'print "ABCD "x4; print "\n"'

ABCD ABCD ABCD ABCD[SPACE]

With Python :

python -c "print('ABCD ' * 4)"

ABCD ABCD ABCD ABCD[SPACE]

With Bash :

A short and elegant solution (source) :: printf 'ABCD %.0s' {1..20}

Generic case :

#!/usr/bin/env bash

answerIsValid=''

while [ -z "$answerIsValid" ]; do
	echo "Continue ? [yn]"
	read answer
	[[ "$answer" == [yYnN] ]] && answerIsValid=1 || echo -e "Invalid answer\n"
done

This is effectively a list of characters, not a regular expression.
The characters list mustn't be quoted.

The list of characters can also be provided as a variable :

#!/usr/bin/env bash

answerIsValid=''
validCharacters='yYnN\[\]'

while [ -z "$answerIsValid" ]; do
	echo "Continue ? [yn]"
	read answer
	[[ "$answer" == [$validCharacters] ]] && answerIsValid=1 || echo -e "Invalid answer\n"
done

Special characters must be escaped.
The example above works either single or double-quoted.

This example script accepts [ or \[ as valid inputs, but .[ is rejected .

method 1

Use getopts.

method 2 : "manually"

#!/usr/bin/env bash

apples=0
bananas=0
coconuts=0

getCliParameters() {
	while [ "$#" -gt 0 ]; do
		case "$1" in
			-a | --apples)   shift; apples="$1"   ;;	1^st shift after reading the option itself (letter/word)
			-b | --bananas)  shift; bananas="$1"  ;;
			-c | --coconuts) shift; coconuts="$1" ;;
			-*) echo "Unknown option: '$1'"; exit 1 ;;
		esac
		shift							2^nd shift after reading its value
	done
	}

displayValues() {
	cat <<-EOF
	On my shopping list, I have :
	- $apples apples
	- $bananas bananas
	- $coconuts coconuts
	EOF
	}

main() {
	getCliParameters "$@"
	displayValues
	}

main "$@"

./test.sh -a 2 --bananas 3 -c 4
On my shopping list, I have :
- 2 apples
- 3 bananas
- 4 coconuts

./test.sh -a 5
On my shopping list, I have :
- 5 apples
- 0 bananas
- 0 coconuts

./test.sh -a 3 -b 3 -c 3 -d 4
Unknown option: '-d'

comparison

method	pros	cons
getopts	does most of the work for you : this is what it was made for	portability issues (on other shells/Unices ?) does not support long options (--parameter value)
"manual"	supports long options better portability	more work "reinventing the wheel"

things they have in common

none has the ability to mark an option as mandatory and warn when missing : you'll have to handle this manually

The basics (details on Bash arrays) :

Explode :

IFS=', ' read -a myArray <<< "$stringToExplode"

Iterate over elements :

for element in "${myArray[@]}"; do
	echo "$element"
done

Iterate over elements using key/value pairs :

for index in "${!myArray[@]}"; do
	echo "$index ${myArray[index]}"
done

Ready-to-use one-liners :

Explode string and iterate on values :

stringToExplode='Lorem ipsum dolor sit amet, ...'; oldIfs="$IFS"; IFS=' '; read -a myArray <<< "$stringToExplode"; IFS="$oldIfs"; for element in "${myArray[@]}"; do echo "$element"; done

Lorem
ipsum
dolor
sit
amet,
...

Explode string and iterate on key / value pairs :

stringToExplode='Lorem ipsum dolor sit amet, ...'; oldIfs="$IFS"; IFS=' '; read -a myArray <<< "$stringToExplode"; IFS="$oldIfs"; for index in "${!myArray[@]}"; do echo "$index ${myArray[index]}"; done

0 Lorem
1 ipsum
2 dolor
3 sit
4 amet,
5 ...

Numbers listed in a file :

Making sums :

Pure Bash version :: sum=0; while read value; do sumBefore=$sum; sum=$((sum+value)); echo "$sumBefore + $value = $sum"; done < fileWithNumbers; echo $sum
bc version :: sum=0; while read value; do sumBefore=$sum; sum=$(echo "$sum+$value" | bc); echo "$sumBefore + $value = $sum"; done < fileWithNumbers; echo $sum
Awk version :: awk '{ sum += $1 } END { print sum }' fileWithNumbers

Which is the fastest ?

tmpFile=$(mktemp --tmpdir tmp.numbers.XXXXXXXX); >$tmpFile; for i in {1..1000}; do echo $i >> $tmpFile; done; echo 'Bash sum'; time (sum=0; while read value; do sum=$((sum+value)); done < $tmpFile; echo "$sum"); echo 'bc sum'; time (sum=0; while read value; do sum=$(echo "$sum+$value" | bc); done < $tmpFile; echo "$sum"); echo 'Awk sum'; time (awk '{ sum += $1 } END { print sum }' $tmpFile); rm $tmpFile

From the fastest to the slowest :

Awk
Bash
bc

Numbers returned to STDOUT :

Dummy example :: echo -e 'result1=3\nresult2=4\nresult3=5' | awk -F '=' '{ sum += $2 } END {print sum}'
Total storage, in GiB, on internal drives :: df -m | awk '/^\/dev/ { sum += $2 } END { print sum/1024 }'
Number of occurrences of a given expression within a bunch of files :: grep -c 'a given expression' * | awk -F ':' '{ sum += $2 } END {print sum}'

nbExpectedArgs=1
if [ $# -ne $nbExpectedArgs ]; then
	echo "Usage: $(basename $0) <argument>"
	exit 1
fi

To display a longer error message with a usage function :

#!/usr/bin/env bash

nbExpectedArgs=3

usage() {
	cat <<-EOFcat is more convenient than numerous echos, or a single echo with line breaks	Usage: $(basename $0) <argument1> <argument2> <argument3>
	blah
	blah
	blah	EOF
	}

if [ $# -ne $nbExpectedArgs ]; then
	usage
	exit 1
fi

How to pass a list (i.e. any number) of arguments to a function ?

How to convert a string to lowercase / uppercase ?

How to loop over a long list of values ?

Situation

Details

Solution

How to declare a multi-line string variable ?

The basic method

An advanced method

A special case

How to increment a variable ?

How to write script logs to stdout and to stderr and to myScript.log ?

Situation

Solution

About this exec > >(tee ) command :

Breakdown

Limitation

How to stop processing the input after reading n lines ?

With grep :

With sed (inspired by) :

With awk :

alternate solution :

How to keep / remove lines of text from start / until end of file, between tokens, starting at token / line x and for the next n lines ?

About the -9999 initialization value :

special case

How to grep special characters : *, TAB ?

Table of contents

an asterisk *

a TAB

How to handle unset variables when the -u / nounset flag is raised ?

Situation

Solution

How to trim a line to a specific length ?

How to feed several variables at once ?

copy-paste-edit solution

alternate solution

What if values are not SPACE-separated and the output has many lines and , i.e. not-so-basic situation

Convert files between DOS and Unix formats

Convert files from DOS to Unix :

How to generate a pin code with a specific length ?

How to alter a line when the previous line matches ?

Situation

Solution

Details

How to (re-)indent code automatically ?

How to get a cascading list of owner + group + permissions for a specified file ?

Situation

Solution

How to find duplicate lines in a file ?

Situation

Solution

How to remove multi-line comments ?

Situation

Solution

tr | sed | tr method :

Alternate solution

Other (better !) method (inspired by) :

How to count occurrences of a character in a string ?

How to escape single quotes within single-quoted strings ?

Solution

Details

Explanation of how '"'"' is interpreted as just ' :

How to concatenate strings ?

The good ol'method :

The += method :

How to get a file modification date ?

How to write all the outputs of a script into a file ?

Situation

Solution 1 (badness=1) :

Solution 2 (badness=100) :

Details

Solution

Details

Named file descriptors (aka "automatic file descriptor allocation")

Snippet to copy-paste :

How to detect whether a script is run interactively or not ?

Situation

Details

Solution

How to transpose line to column / column to line ?

About this `exec > >(tee )` command :

How to grep special characters : `*`, `TAB` ?

an asterisk `*`

a `TAB`

What if values are not `SPACE`-separated and the output has many lines and , i.e. not-so-basic situation

Explanation of how `'"'"'` is interpreted as just `'` :

The `+=` method :

How to display the x^th, y^th and z^th lines of a stream of text ?

With a `for` loop :

With a `case` construct :