Shell - (on my way to) Mastering the command line

• options :
  • affect the actions of command :
    • ls : (no option) list files
    • ls -l : list files using the long display format
  • can be specified with :
    • a single character with a leading hyphen : ls -a
    • a word with 2 leading hyphens : ls --all
  • may require a value (sometimes referred to as argument, which is rather confusing) :
    • for single-character options : with a [space] between the option and its value : tail -n 5 someFile
    • for "word" options : with a = between the option and its value : tail --lines=5 someFile
  • can have their short versions combined when they have no argument : ls -lh
  •  --  (i.e. [space]+--+[space]) marks the end of options, everything after this is an argument (even though it looks like a valid option)
• arguments :
  • are what command works on :
    • for cp : the source and destination values
    • for cd : the directory to enter
    • for emacs : the file to edit
    • for vlc : the file to play

More definitions (sources : 1, 2) :

!!

previous command (source : man -P 'less -p "\\!\\!"' history)

Technically, this is not a variable but a command. It's listed here because this article is where I'll start searching when facing such cryptic things .

!$

last argument of the preceding command. Said to be equivalent to $_ (see also)

$!

PID of last job run in background. Try it : pwd & echo $!

$#

number of parameters passed to the current script or function

Check it :

#!/usr/bin/env bash

tmpScript=$(mktemp tmpScript.XXXXXXXX)

cat << 'EOF' > "$tmpScript"
#!/usr/bin/env bash
main() { echo $# : $@; }
main "$@"
EOF

chmod +x "$tmpScript"
./"$tmpScript"
./"$tmpScript" foo
./"$tmpScript" foo bar
./"$tmpScript" 'foo bar'
rm "$tmpScript"

0 :
1 : foo
2 : foo bar
1 : foo bar

$$

PID of current process

• It _may_ sound like a good idea to use this as part of a temporary file name so that concurrent executions of a script won't collide, but mktemp will do that better.
• It's very convenient to detect which flavor of shell is being used : ps $$

  PID	TTY	STAT	TIME	COMMAND
  18060	pts/1	Ss	0:01	/bin/bash

$*

all the parameters passed to the current script, as a space-separated string (source) :

• $* and $@ hold a b c d (4 distinct elements)
• "$*" holds a b c d (a single string having spaces). This is not iterable because of the explicit double-quotes.
• "$@" holds a b c d (the 3 script parameters). The double quotes protect spaces when they are part of the parameter itself.

Check it :

#!/usr/bin/env bash
# run :	./script.sh a b 'c d'

echo -e '\nwith $* :'
for i in $*; do echo $i; done

echo -e '\nwith $@ :'
for i in  $@; do echo $i; done

echo -e '\nwith "$*" :'
for i in  "$*"; do echo $i; done

echo -e '\nwith "$@" :'
for i in  "$@"; do echo $i; done

with $* :
a
b
c
d

with $@ :
a
b
c
d

with "$*" :
a b c d

with "$@" :
a
b
c d

$-

current options set for the shell, the doc says it is not completely reliable

This variable is often used to check whether the current script is run interactively or not (details).

$0

name of the shell or current shell script (see also ${BASH_SOURCE[0]})

$?

decimal exit code of previous process (aka "most recent pipeline").

This handles the value returned by return.

$@

all the parameters passed to the current script or function, as separate words. This is an "iterable list of strings" (source, see $* for details)

You should practically always use double quotes around "$@".

$_

final argument of previous command executed :

testFile='./test'; echo $_; echo hello >"$testFile"; echo $_; rm "$testFile"; echo $_

hello	this is the last argument of echo, even if there are some extra directives on the command line
./test	this refers to "$testFile" coming after rm : the value has been substituted

BASH_SOURCE[0]

this is almost a synonymous of $0 but has slight differences (source) :

${BASH_SOURCE[0]}	$0
holds the current script name, when executed or sourced is read-only is Bash-specific can be shortened to $BASH_SOURCE (Bash allows referring to the element 0 of an array with the array name itself)	holds the current script name only when executed (something else when sourced) can be overwritten is POSIX-compliant

Check it with this script :

#!/usr/bin/env bash
echo -e "
\$0:$0
\$BASH_SOURCE:$BASH_SOURCE
\${BASH_SOURCE[0]}:${BASH_SOURCE[O]}
" | column -s ':' -t

• ./test.sh

  $0			./test.sh
  $BASH_SOURCE		./test.sh
  ${BASH_SOURCE[0]}	./test.sh

• source test.sh

  $0			/bin/bash
  $BASH_SOURCE		test.sh
  ${BASH_SOURCE[0]}	test.sh

DISPLAY

• indicates where the X server is. Format :
  host:displayNumber.screenNumber
  Numbers start at 0 !
• To temporarily connect to a specific X server, just type :
  command_line -display (or --display ?) host:displayNumber.screenNumber
• If you're connecting to the remote host via SSH, consider using the X11 forwarding option.

EDITOR

set the program to use to edit :

• crontabs
• Git commit messages, rebases, patches, ...
• files viewed with less
• ...

http_proxy

Set it with :

export http_proxy=http://host:port

Same goes on for https_proxy and ftp_proxy.

HISTFILE

Upon Bash process termination, all typed commands are written into an history file — ~/.bash_history — which absolute path is stored in HISTFILE.

You may disable recording the history of commands.

HOME

current user's home directory, and default for the cd builtin command

About unquoted tilde expressions : ~whatever (source) :

• if whatever is a valid user name — like ~bob— the tilde expression is expanded into Bob's home directory
• if whatever is an empty string, ~ is expanded into $HOME
• otherwise, whatever is left unchanged

HOSTNAME

The name of the current host

IFS

input field separator : used to split string into substrings, defaults to "[space][tab][newline]"

LANG (1, 2, 3)

• sets the default locale, i.e. the locale used when no more specific setting (LC_COLLATE, LC_NUMERIC, LC_TIME, ) is provided; it doesn’t override any setting, it provides the base value.
• prefixing a shell command with LANG=C is a common hack to ensure the output message will be displayed in the default language (i.e. english), which is important on commands that select/filter text
• list the available locales

LANGUAGE

• not listed in the Bash reserved variables names (???)
• usage :

  LANGUAGE=en find charlie
  	find: ‘charlie’: No such file or directory
  LANGUAGE=fr find charlie
  	find: ‘charlie’: Aucun fichier ou dossier de ce type

LC_NUMERIC

defines whether numbers should be displayed with a point or a comma as a decimal separator

LOGNAME

Depending on the Unix / Linux flavor :

• either the name of the user that initially logged in, even though this user executed su afterwards (source : 1, 2)
• or a synonymous for USER

OLDPWD

The previous working directory as set by the cd builtin (see PWD)

PATH

Path to binaries. Since this is a colon (:)-separated list, to append a new value :

• DON'T :
  PATH=path/to/some/directory
  this would overwrite the whole content of the variable
• DO :
  PATH=$PATH:path/to/some/directory

PIPESTATUS

Array holding exit statuses of last executed foreground pipe. Indexes start at 0.

true | true | false | true | false | false | true; echo ${PIPESTATUS[*]}

0 0 1 0 1 1 0

true | true | false | true | false | false | true; echo "${PIPESTATUS[0]} ${PIPESTATUS[1]} ${PIPESTATUS[2]} ${PIPESTATUS[3]} ${PIPESTATUS[4]} ${PIPESTATUS[5]} ${PIPESTATUS[6]}"

0 0 1 0 1 1 0

PPID

The shell's parent process identifier (or parent's pid if you like)

PROMPT_COMMAND

The contents of this variable are executed as a regular Bash command just before Bash displays a prompt.

This can be used to display the current Git branch in the prompt (inspired by).

PS1

is the shell prompt (details)

PWD

The current working directory as set by cd (see OLDPWD). Using currentDir=$(pwd) is double work since PWD is already available.

RANDOM

• returns a pseudorandom integer in the range 0 - 32767
• to roll a dice :
  echo $(($RANDOM % 6))
  • this is a 0-based dice, with numbers 0, 1, 2, 3, 4, 5 and no 6
  • you may find articles mentioning the $[] construct : this is an old syntax that has been obsoleted by $(()) (source)
• see also shuf

SHELL

• full pathname to the shell
• if not set when the shell starts, Bash assigns to it the full pathname of the current user's login shell

SHLVL

number of nested subshells (starts at 1). Try it : echo -n $SHLVL; bash -c 'echo -n $SHLVL; bash -c "echo -n \$SHLVL"; echo -n $SHLVL; exit; '; echo -n $SHLVL will output :

12321

TMPDIR

directory in which Bash creates temporary files for its own use

USER

the current user name, replaces an overkill $(whoami) in scripts

This variable is often set as an environment variable by Bash login startup files, but it is not actually a Bash builtin (source).

• WARNING: terminal is not fully functional
• Error opening terminal: screen.xterm-256color.

1. this has to do with the shell history
2. history entries are named events
3. the shell interprets inputs like !something as commands to recall an event (details)

So, if you just want to use an exclamation mark (not followed by a space) in a string, just simple-quote it :

echo Hello world !

Hello world !

echo Hello world ! Hello everybody !

Hello world ! Hello everybody !

echo ah!ah!ah!

bash: !ah!ah!: event not found

echo "ah!ah!ah!"

bash: !ah!ah!: event not found

echo 'ah!ah!ah!'

ah!ah!ah!

echo Hello everybody !!

Surprise... but remember !! can express more than just extreme happiness

• set -o vi
• set -o emacs

I have :

testString='I like bananas.\nbananas are great.\nbananas are what I prefer.\nMy favorite fruit is bananas.'; echo -e "$testString"

I like bananas.
bananas are great.
bananas are what I prefer.
My favorite fruit is bananas.

I want :

I like bananas.
bananas are great.
oranges are what I prefer.
My favorite fruit is bananas.

With awk (source) :

oranges are what I prefer.

With sed :

I like bananas.
bananas are great.
oranges are what I prefer.
My favorite fruit is bananas.

BEFORE :
I like bananas.
bananas are great.
bananas are what I prefer.
My favorite fruit is bananas.

AFTER :
I like bananas.
bananas are great.
oranges are what I prefer.
My favorite fruit is bananas.

At first sight, sed looked simpler to me than Awk, but Awk is also able to work on a specific field of every line, or on a specific field of a specific line.

fileName='my file'; touch $fileName; ls -1

file
my

rm $fileName; ls -1

(both gone)

touch "$fileName"; ls -1

my file

rm "$fileName"; ls -1

(gone)

touch ${fileName}; ls -1

file
my

rm ${fileName}; ls -1

(both gone)

touch ${fileName}1 ${fileName}2; ls -1

file1
file2
my

ls -1 $fileName*

file1
file2
my

ls -1 "$fileName*"

ls: cannot access my file*: No such file or directory

rm ${fileName}1 ${fileName}2; ls -1

(all gone)

touch "${fileName}1" "${fileName}2"; ls -1

my file1
my file2

ls -1 $fileName*

no double quotes

ls: cannot access my: No such file or directory
ls: cannot access file*: No such file or directory

ls -1 "$fileName*"

* inside double quotes

ls: cannot access my file*: No such file or directory

ls -1 "$fileName"*

* outside double quotes

my file1
my file2

How commands are read by Bash (source) :

Examples :

• Regex operators
• Substrings and regular expressions in a Awk / Bash / Perl / sed context
• Wildcards in Bash

On UTF-8-capable systems (and generally speaking : extended charsets), characters lists such as a-z include special characters like à, é or î. Thus, characters lists (a-z) can not be used in regular expressions to discriminate ASCII/non-ASCII characters. To do so, the solution is to build a complete list of all characters to match against with a regular expression : abcdefghijklmnopqrstuvwxyz (source).

-bash: [: -eq: unary operator expected
this is the "ELSE" part

ksh: [: argument expected
this is the "ELSE" part

-bash: [: -ne: unary operator expected
this is the "ELSE" part

ksh: [: argument expected
this is the "ELSE" part

We note that :

• The error messages differ depending on the shell being used
• Whatever a -eq or a -ne test is made, the else part is executed.

Send several commands via SSH :

ssh bob@sshServer << EOC
command1
command2
EOC

Send several lines of text to a command via a | :

cat << EOL | grep base
Roses are #ff0000
Violets are #0000ff
All my base are belong to you
EOL

Create a new file :

cat << EOF > newFile.txt
some text, line 1
some text, line 2
some text, line n
EOF

cat << EOF > newFile.txt
line 1, not indented
    line 2, space-indented
	line 3, TAB-indented
EOF
cat newFile.txt; rm newFile.txt

Stop token hacks

cat << EOF
indent : none
  indent : SPACE * 2
	indent : TAB * 1

EOF

cat <<-EOF
indent : none
  indent : SPACE * 2
	indent : TAB * 1

EOF

indent : none
  indent : SPACE * 2
	indent : TAB * 1

indent : none
  indent : SPACE * 2
indent : TAB * 1

What if I want to output some special characters without disabling variable expansion ?

value=42; exampleFile='/tmp/myExampleFile.txt'; cat << EOF > "$exampleFile"
if (\$something > $value)
    blah

if (\$anything < ($value/2))
    pooh
else
    nomnom

if (true || false || whatever)
    who_cares
EOF
cat "$exampleFile"; rm "$exampleFile"

if ($something > 42)
    blah

if ($anything < (42/2))
    pooh
else
    nomnom

if (true || false || whatever)
    who_cares

How to pipe a multiline heredoc into a command ?

cat << EOSQL | sqlplus -s / as sysdba | grep -Ev '^$'
SELECT DISTINCT(TRUNC(last_refresh)) FROM dba_snapshot_refresh_times;
query1;
query2;
EOSQL

echo -e "query1;\nquery2;" | sqlplus -s / as sysdba | grep -Ev '^$'

sqlplus -s / as sysdba << EOSQL | grep -Ev '^$'
query1;
query2;
EOSQL

HEADER
data line 1
data line 2
data line 3

... with grep :

echo -e "HEADER\ndata line 1\ndata line 2\ndata line 3" | grep -v 'HEADER'

Requires to know how to match the header line, i.e. knowing HEADER.

... with sed :

• echo -e "HEADER\ndata line 1\ndata line 2\ndata line 3" | sed -n '2,$ p'
• or even simpler : echo -e "HEADER\ndata line 1\ndata line 2\ndata line 3" | sed 1d

... with tail :

echo -e "HEADER\ndata line 1\ndata line 2\ndata line 3" | tail -n +2

• bash --version
• echo $BASH_VERSION

Leading characters :

for i in {1..1000}; do echo $RANDOM >> data.txt; done

sed -r 's/(^.{3}).*$/\1/g' data.txt

cut -c -3 data.txt

Trailing characters :

echo hello | sed -r 's/.*(.{3})$/\1/g'

user@host $ emacs & vlc &
[1] 10367
[2] 10368

List the current jobs :

Suspend a running job :

Resume a suspended job :

Output a list of values (details) :

Output a sequence of characters or numbers (details):

Other brace expansions :

foo1bar foo2bar foo3bar

test_1a_ test_1b_ test_2a_ test_2b_

a h o v

myFunction() {
	local myVariable=$1
	local myOtherVariable=${2:-42}
	
	}

value
nothing

value
42

value
nothing

key exists
(empty line)

42
bash: value: this variable is unset.

23456789
45
67

y

7

☹☺	2

def
bcdef
bbccc

txt

ccc

log

aaabbbccc
aaabb

aaa

abcdef

abCD abcd

abCD abCD

Hello, world
HELLO, WORLD
hELLO, WORLD
hello, world

Exit codes over 255