Shell - (on my way to) Mastering the command line

WARNING: terminal is not fully functional

Situation :

When running man, less, ..., in some specific conditions (e.g. within an SSH session nested inside a screen session), you may get this error message :
WARNING: terminal is not fully functional
-  (press RETURN)

Details :

This is because the TERM environment variable is not / improperly set.

Solution :

export TERM=xterm-256color
Or fix it permanently.

-bash: !whatever: event not found, exclamation marks, events, what are these ?

Long story short :
  1. this has to do with the shell history
  2. history entries are named events
  3. the shell interprets inputs like !something as commands to recall an event (details)

So, if you just want to use an exclamation mark (not followed by a space) in a string, just simple-quote it :

echo Hello world !
Hello world !
echo Hello world ! Hello everybody !
Hello world ! Hello everybody !
echo ah!ah!ah!
bash: !ah!ah!: event not found
echo "ah!ah!ah!"
bash: !ah!ah!: event not found
echo 'ah!ah!ah!'
ah!ah!ah!
echo Hello everybody !!
Surprise... but remember !! can express more than just extreme happiness

Bash shortcut keys

Bash has Emacs (default, see table below) and Vi shortcut sets. To define your favorite shortcut style :
ALT CTRL
a move to the beginning of the line
conflicts with screen's CTRL-a-... shortcuts
c send the SIGINT signal to stop the current process
d delete everything after the cursor exits the current shell
e move to the end of the line
l clear the screen
q resume from CTRL-s
r search history of commands
s pause output to the screen. Useful when running a verbose command :
for i in {1..1000}; do echo $i; sleep 0.1; done
x-x Jump back and forth between the cursor position and the beginning of the line
z send the SIGTSTP signal to the current foreground process to send it to the background. Use fg to bring it back to the foreground.
_ Undo the last keystroke. Can be repeated to undo several keys back
. paste the last argument of the previous command. Repeat to cycle back in previous commands

How to perform search / replace on a specific line of a file ?

Let's consider the test string :
testString='I like bananas.\nbananas are great.\nbananas are what I prefer.\nMy favorite fruit is bananas.'

With awk (source) :

echo -e "$testString" | awk '/prefer/ { gsub("bananas","apples",$0); print $0 }'

apples are what I prefer.

This only returns the processed line. Needs a little more work

With sed :

echo -e "$testString" | sed -r '/prefer/ s/bananas/apples/'

I like bananas.
bananas are great.
apples are what I prefer.
My favorite fruit is bananas.

testFile='./testFile'; echo -e "$testString" > "$testFile"; echo 'BEFORE :'; cat "$testFile"; sed -i '/prefer/ s/bananas/apples/' "$testFile"; echo -e '\nAFTER :'; cat "$testFile"; rm "$testFile"

BEFORE :
I like bananas.
bananas are great.
bananas are what I prefer.
My favorite fruit is bananas.
AFTER :
I like bananas.
bananas are great.
apples are what I prefer.
My favorite fruit is bananas.

At first sight, sed looked simpler to me than awk, but awk is also able to work on a specific field of every line, or on a specific field of a specific line.

Wildcards and quotes

Quoting shell variables is mandatory to protect scripts from unexpected effects should unquoted variables contain SPACE characters (or any other character considered by the shell as a word separator, such as TAB or NEWLINE. Read more about IFS). Indeed, the SPACE character is one of the common argument separators for the shell. Here come the quotes (but they must be set wisely ).
Let's play :
fileName='my file'; touch $fileName; ls -1
file
my
rm $fileName; ls -1
(both gone)
touch "$fileName"; ls -1
my file
rm "$fileName"; ls -1
(gone)
touch ${fileName}; ls -1
file
my
rm ${fileName}; ls -1
(both gone)
touch ${fileName}1 ${fileName}2; ls -1
file1
file2
my
ls -1 $fileName*
file1
file2
my
ls -1 "$fileName*"
ls: cannot access my file*: No such file or directory
rm ${fileName}1 ${fileName}2; ls -1
(all gone)
touch "${fileName}1" "${fileName}2"; ls -1
my file1
my file2
ls -1 $fileName*
no double quotes
ls: cannot access my: No such file or directory
ls: cannot access file*: No such file or directory
ls -1 "$fileName*"
* inside double quotes
ls: cannot access my file*: No such file or directory
ls -1 "$fileName"*
* outside double quotes
my file1
my file2

How commands are read by Bash (source) :

  1. read input
  2. break it up into words and operators, obeying the quoting rules (escape characters, simple and double quotes) :
    • backslash \ : preserve the literal value of the following character (except newline)
    • single quotes '...' : preserve the literal value of each character enclosed within the quotes. A single quote may not occur between single quotes, even when preceded by a \.
    • double quotes "..." : preserve the literal value of all characters enclosed within the quotes, except for $, `...` and \.
    No * within quotes.
  3. perform alias expansion
  4. substitute the tokens into simple and compound commands (e.g. if, for, while, [[, case, ... constructs. More)
  5. perform shell expansions (source) :
    1. brace expansion : {...}
    2. tilde expansion : ~
    3. variables expansion : $myVariable is substituted with its value
    4. commands substitution : $(command) (or `command`) is substituted with its output
    5. arithmetic expansion : $((arithmeticExpression)) is substituted with its result
    6. process substitution : ???
    7. word split : split the result of previous expansions by SPACE (actually, each character of $IFS is a delimiter).
    8. file name expansion : if any word contains *, ? or[, it is considered as a pattern and replaced with an alphabetically sorted list of file names matching the pattern (if any. Otherwise, leave the special character as-is)
  6. redirections, if any
  7. execute the command
  8. wait for the command to complete and collect its exit status

Examples :

Considering we have run : fileName='my file'; touch "${fileName}1" "${fileName}2", (so we actually have files my file1 and my file2), let's list files :

ls $fileName*

  1. variable expansion : ls my file*
  2. word split (already split up) : ls my file*
  3. file name expansion : no file found match the pattern file*, so no change : ls my file*
  4. result

ls "$fileName*"

  1. word/operator breakup : $fileName is recognized as a variable and will be substituted
  2. variable expansion : ls "my file*"
  3. word split : because of the quotes, the [SPACE] between my and file can not be used to split : ls "my file*"
  4. file name expansion : because of the quotes again, the * is taken literally, nothing to expand. No file named exactly my file* (i.e. having a * in the file name) found, so no change : ls "my file*"
  5. result

ls "$fileName"*

  1. word/operator breakup : $fileName is recognized as a variable and will be substituted
  2. variable expansion : ls "my file"*
  3. word split : (as above) nothing to split : ls "my file"*
  4. file name expansion : files my file1 and my file2 are found and substituted : ls my file1 my file2
  5. result

Regular Expressions in shell context

Since I already dealt with that topic earlier and spread information in many places, here's a collection of hyperlinks to the corresponding articles / sections until I clean this up :

On UTF-8-capable systems (and generally speaking : extended charsets), characters lists such as a-z include special characters like à, é or î. Thus, characters lists (a-z) can not be used in regular expressions to discriminate ASCII/non-ASCII characters. To do so, the solution is to build a complete list of all characters to match against with a regular expression : abcdefghijklmnopqrstuvwxyz (source).

Unset variables in Bash / Ksh

For any further reference, here's what happens during an if statement in Bash and in Ksh while the tested variable is actually unset :
Shell Output
if [ ${undefinedVariable} -eq 0 ]; then echo 'this is the "THEN" part'; else echo 'this is the "ELSE" part'; fi
Bash
-bash: [: -eq: unary operator expected
this is the "ELSE" part
Ksh
ksh: [: argument expected
this is the "ELSE" part
if [ ${undefinedVariable} -ne 0 ]; then echo 'this is the "THEN" part'; else echo 'this is the "ELSE" part'; fi
Bash
-bash: [: -ne: unary operator expected
this is the "ELSE" part
Ksh
ksh: [: argument expected
this is the "ELSE" part

We note that :

Heredocs

Send several lines of text to another command via a | :

cat << EOL | grep base
Roses are #ff0000
Violets are #0000ff
All my base are belong to you
EOL

Create a new file :

Step-by-step version :
  1. cat << EOF > newFile.txt
  2. some text, line 1
  3. some text, line 2
  4. some text, line n
  5. EOF
Big-bang version :
cat << EOF > newFile.txt
some text, line 1
some text, line 2
some text, line n
EOF
  • Data is written to newFile.txt
  • EOF is the "end of file" tag (aka stop token). Any string can be used instead.
cat << EOF > newFile.txt
line 1, not indented
    line 2, space-indented
	line 3, TAB-indented
EOF
cat newFile.txt; rm newFile.txt
  • Indentation works only with spaces, not with TAB.
  • The stop token must have no leading whitespace.

Stop token hacks

Single-quoted stop token (source)
This disables :
  • variable expansion (changing $USER into )
  • command substitution (changing $(command) into the result of executing command)
  • arithmetic expansion (changing $((1+1)) into 2)
cat << EOF
current user : $USER
today : $(date +"%a %b %d")
2 apples + 1 banana is $((2+1)) fruits
EOF
current user : 
today : Fri Nov 17
2 apples + 1 banana is 3 fruits
cat << 'EOF'
simple quotes : $USER
today : $(date +"%a %b %d")
2 apples + 1 banana is $((2+1)) fruits
EOF
simple quotes : $USER
today : $(date +"%a %b %d")
2 apples + 1 banana is $((2+1)) fruits
Dash (-) -prefixed stop token (source) :

This is a cosmetic hack improving readability of scripts since its allows indenting the heredocs too. The - in the stop token suppresses leading tabs in the output.

  • This has no effect on lines indented with spaces, including the line of the stop token itself which must be TAB-indented. If space-indented, the stop token line becomes invisible which causes an error :
    ./myScript.sh: line 63: warning: here-document at line 23 delimited by end-of-file (wanted `EOF')
    ./myScript.sh: line 64: syntax error: unexpected end of file
  • There must be no space bettween << and - (Can't explain why, but if you try, it'll fail )
cat << EOF
no indent
 indent one space
	indent one TAB
EOF
no indent
 indent one space
indent one TAB
cat <<-EOF
no indent
 indent one space
	indent one TAB
	 indent one TAB + one space
		indent 2 TAB's
	EOF
no indent
 indent one space
indent one TAB
 indent one TAB + one space
indent 2 TAB's

What if I want to output some special characters without disabling variable expansion ?

value=42; exampleFile='/tmp/myExampleFile.txt'; cat << EOF > "$exampleFile"
if (\$something > $value)
    blah
if (\$anything < ($value/2))
    pooh
else
    nomnom
if (true || false || whatever)
    who_cares
EOF
cat "$exampleFile"; rm "$exampleFile"
outputs :
if ($something > 42)
    blah
if ($anything < (42/2))
    pooh
else
    nomnom
if (true || false || whatever)
    who_cares

How to pipe a multiline heredoc into a command ?

You can do things like :
cat << EOSQL | sqlplus -s / as sysdba | grep -Ev '^$'
SELECT DISTINCT(TRUNC(last_refresh)) FROM dba_snapshot_refresh_times;
query1;
query2;
EOSQL
Or even :
echo -e "query1;\nquery2;" | sqlplus -s / as sysdba | grep -Ev '^$'
But it is simpler to do :
sqlplus -s / as sysdba << EOSQL | grep -Ev '^$'
query1;
query2;
EOSQL
Remember :
  • The cat << EOF construct is fine when redirecting into a file
  • when redirecting to a command, command << EOF looks more appropriate

How to remove the header line from a command output ?

Let's imagine a command (such as a DB query) that outputs something like :
HEADER
data line 1
data line 2
data line 3
You can retrieve all lines except the header ...
... with grep :
echo -e "HEADER\ndata line 1\ndata line 2\ndata line 3" | grep -v 'HEADER'
Requires to know how to match the header line, i.e. knowing HEADER.
... with sed :
  • echo -e "HEADER\ndata line 1\ndata line 2\ndata line 3" | sed -n '2,$ p'
  • or even simpler : echo -e "HEADER\ndata line 1\ndata line 2\ndata line 3" | sed 1d
... with tail :
echo -e "HEADER\ndata line 1\ndata line 2\ndata line 3" | tail -n +2

Why does this {start..stop..step} output {start..stop..step} instead of a sequence of numbers ?

The "step" feature of the brace expansion is a new feature of Bash 4 (source). To get Bash version :

How to display the n leading / trailing characters from each line of a file ?

Leading characters :

Given the data file :
for i in {1..1000}; do echo $RANDOM >> data.txt; done
sed can do it :
sed -r 's/(^.{3}).*$/\1/g' data.txt
But it's overkill as cut can do it way easier :
cut -c -3 data.txt

Trailing characters :

No such option in cut, so let's use sed :
echo hello | sed -r 's/.*(.{3})$/\1/g'

Job control

Job control is nothing but the ability to stop / suspend / resume the execution of processes. A jobId is displayed when starting a process in the background :
user@host $ emacs & vlc &
[1] 10367
[2] 10368
Here, emacs is the 1st command we've launched, 1 is its jobId and 10367 is its PID.

List the current jobs :

jobs
[1]-	Running	emacs &
[2]+	Running	vlc &
jobs -l
[1]-	10367	Running	emacs &
[2]+	10368	Running	vlc &
Field Value Description Example
[n] Job ID
To be used with fg, bg, wait, kill, ... The job ID must be prefixed by a %
fg %1
kill %5
+ or - + : current job
- : previous job
10367 PID
Job status :
  • Running : currently running (not stopped / suspended)
  • Stopped : job is suspended

The %jobId syntax used to refer to a job is also known as jobspec.

Suspend a running job :

Resume a suspended job :

  • In the foreground : fg %2
  • In the background : bg %5

Curly brackets & shell Brace Expansion

Output a list of values (details) :

myCommand {value1,value2,value3}
is equivalent to :
myCommand value1; myCommand value2; myCommand value3

Output a sequence of characters or numbers (details):

ascending numbers :
echo {2..8} : 2 3 4 5 6 7 8
reverse order letters :
echo {z..a} : z y x w v u t s r q p o n m l k j i h g f e d c b a
descending numbers with leading zeros (source) :
echo {100..00..10} : 100 090 080 070 060 050 040 030 020 010 000

Other brace expansions :

Syntax Description Example
{value1,value2,value_n}
String generation
Generate as many strings as the number of parameters, including a prefix and/or a suffix (examples)
echo foo{1,2,3}bar
foo1bar foo2bar foo3bar
{start..stop}
{start..stop..step}
Sequence generation
Generate a string for each parameter from the specified interval, including a prefix and/or a suffix
The step parameter appears with Bash 4 (details, examples).
This construct should be preferred to seq because it won't start subprocesses.
echo test_{1..2}{a..b}_
test_1a_ test_1b_ test_2a_ test_2b_
echo {a..z..7}
a h o v
${parameter:-default}
Use default value
If parameter is unset or null, default (which may be an expansion) is substituted. Otherwise, the value of parameter is substituted.
Return the value of parameter, or default if unset.
key='value'; echo ${key:-'nothing'}; unset key; echo ${key:-'nothing'}
value
nothing
key='value'; default=42; echo ${key:-$default}; unset key; echo ${key:-$default}
value
42
${parameter:=default}
Assign default value
If parameter is unset or null, default (which may be an expansion) is assigned to parameter. The value of parameter is then substituted.
In other words : this stores default in parameter if unset. Leave as is otherwise.
key='value'; result=${key:='nothing'}; echo $key; unset key; result=${key:='nothing'}; echo $key
value
nothing
${parameter:+value}
Use value if parameter exists
If parameter exists, substitute (e.g. return) value (which may be an expansion).
Otherwise (parameter is null or unset), return nothing.
key='value'; echo ${key:+'key exists'}; unset key; echo ${key:+'key exists'}
key exists
(empty line)
${parameter?message}
Return parameter, or display message if unset
value=42; echo ${value?this variable is unset.}; unset value; echo ${value?this variable is unset.}
42
bash: value: this variable is unset.
${parameter:offset:length}
Substring Expansion
Expands to up to length characters of parameter starting at the character specified by offset (0-indexed). If :length is omitted, go all the way to the end. If offset is negative (use parentheses!), count backward from the end of parameter instead of forward from the beginning.
The last character of a string : ${parameter:(-1):1}
myString='0123456789'; echo ${myString:2}; echo ${myString:4:2}; echo ${myString:(-4):2};
23456789
45
67
myString='azerty'; echo ${myString:(-1):1}
y
${#myString}
${#myArray}
Number of sub-elements
  • length of the string myString
  • number of items in the array myArray Don't use this!
myString='foo bar'; echo ${#myString}
7
${parameter#pattern}
Remove pattern from the beginning of parameter
The pattern is matched against the beginning of parameter. The result is the expanded value of parameter with the shortest match deleted.
This can be used to retrieve the extension of a file name.
myString='abcdef'; echo ${myString#abc}; echo ${myString#[abc]}; myString='aaabbbccc'; echo ${myString#a*b}
def
bcdef
bbccc
name=file.txt; echo ${name#*.}
txt
${parameter##pattern} As above, but the longest match is deleted. myString='aaabbbccc'; echo ${myString##a*b}
ccc
${parameter%pattern}
Remove pattern from the end of parameter
The pattern is matched against the end of parameter. The result is the expanded value of parameter with the shortest match deleted.
This can be used to retrieve a file name without extension. If the extension is known, basename can do it very easily.

This can be used to build a logfile name based on the script name :

script='/path/to/my script.sh'; logFile=$(basename "$script"); logFile=${logFile%.*}'.log'; echo "$logFile"
my script.log

myString='aaabbbccc'; echo ${myString%a*b}; echo ${myString%b*c}
aaabbbccc
aaabb
name=file.txt; echo ${name%.*}
file
${parameter%%pattern} As above, but the longest match is deleted. myString='aaabbbccc'; echo ${myString%%b*c}
aaa
${parameter/search/replace} Results in the expanded value of parameter with the first match of search replaced by replace. myString='abcd abcd'; echo ${myString/cd/CD}
abCD abcd
${parameter//search/replace} As above, but every match of search is replaced. myString='abcd abcd'; echo ${myString//cd/CD}
abCD abCD
  1. ${parameter^}
  2. ${parameter^^}
  3. ${parameter,}
  4. ${parameter,,}
Return parameter with :
  1. 1st character uppercase
  2. all characters uppercase
  3. 1st character lowercase
  4. all characters lowercase
(source)
myString='hello, world'; echo ${myString^}; echo ${myString^^}; myString=${myString^^}; echo ${myString,}; echo ${myString,,}
Hello, world
HELLO, WORLD
hELLO, WORLD
hello, world

Bash command-line tips

Enable / disable verbose mode (source) :

  • Enable :
    • bash -v
    • set -v
    • set -o verbose
  • Disable :

Replay and amend the previous command (source) :

!! replays the previous command :
$ apt-get install package
$ sudo !!
You can even modify and replay a previous command : !!:s/wrong/right

Re-use a previous (long-typed) argument (source, related) :

  1. Let's imagine you just typed : anyCommand a/very/long/and/complex/path/to/a/file
  2. To re-use the argument (filename, here) with another command, just type the command and press ALT-. to paste the argument.
  3. Press ALT-. again to scroll back the arguments list.

Edit and replay the previous command (source) :

If you typed a command that failed, such as :
cd /rome
you can fix it with the ^wrongString^rightString^ syntax :
^rome^home^
This will perform strings substitution and execute :
cd /home

Shell exit codes

Code Meaning
0 Success
(aka "UNIX_SUCCESS" in my scripts)
1 Catchall for general errors
(aka "UNIX_FAILURE" in my scripts)
2 Misuse of shell builtins
126 Command invoked cannot execute
127 command not found
128 Invalid argument to exit
128 + n Fatal error signal n
130 Script terminated by CTRL-c
255* Exit status out of range

Exit codes over 255

On some special cases (such as programs launching shell commands as a child process), the exit code may be shown on 2 bytes :
Bits Meaning
15-8 shell command (child process) exit code
7 =1 if a core dump was produced
6-0 signal number that killed the process

So, given the 32512 exit code, which gives 0111 1111 0000 0000 in binary, we can deduce that the shell exit code was 127.

Faster solution : final exit code = exit code modulo 255