Awk - Description, flags and examples

checkPageExists() {
	local page=$1
	curl -sI "$page" | awk '/^HTTP\/1.1/ {
		if ($2=="200")
			exit 0
		else
			exit 1
		 }'
	}

main() {
	
	for page in $pageList; do
		checkPageExists "$page" || continue
		
	done
	}

Tutorials & basic examples :

List "enabled" repositories :

[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://download.docker.com/linux/centos/$releasever/source/stable
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

curl -s https://download.docker.com/linux/centos/docker-ce.repo | awk ' \
	BEGIN		{ repoId=""; } \
	/^ *$/		{ repoId=""; } \
	/^\[.*\]/	{ repoId=$0; } \
	/^enabled=1/	{ if (repoId!="") { result=gensub(/[\[\]]/, "", "g", repoId); print result; } }'

docker-ce-stable

Find who's the link / who's the target in a symlink :

while read line; do
	echo -e "\t$line"
	echo "$line" | awk '{ link=$1; target=""; for(i=2; i<=NF; i++) { if($i!="->") link=link" "$i; else break; } for(j=i+1; j<NF; j++) target=target""$j" "; target=target""$NF; print " LINK: \047"link"\047, TARGET: \047"target"\047"; }'
	echo
done < <(find -type l -exec ls -l {} + | awk '{ for (i=1; i<9; i++) $i=""; print $0; }')

• By default, Awk prints the matching lines, so anything like { print $0 } or even { print } works, but is redundant.
• There are some slight differences between print and printf
• There are many ways of specifying which fields from the input should be output :
  • list them explicitly :
    echo {a..z} | awk '{ print $8, $5, $12, $12, $15 }'

    h e l l o

  • list them in a loop :
    echo {a..z} | awk '{ for(i=3; i<10; i++) printf $i }'

    cdefghi

  • list them counting from the end of the input line, using NF :
    echo {a..z} | awk '{ print $NF, $(NF-1), $(NF-2) }'

    z y x

print :

printf :

sprintf :

echo 'abc' | awk '{
	switch ($0) {
		case /[[:lower:]]+/:
			print "lowercase"
			break
		case /[[:upper:]]+/:
			print "uppercase"
			break
		}
	}'

echo 'abc' | awk '{ switch ($0) { case /[[:lower:]]+/: print "lowercase"; break; case /[[:upper:]]+/: print "uppercase"; break; } }'

#!/usr/bin/env bash

value1=120
# initial value : 160
# tested values : 200, 140, 100
#	decreasing from the initial value : more bass sounds

value2=0.5678
# initial value : 0.87055
# tested values : 0.747, 0.777, 0.789
#	increasing values above 3.xxx : extreme bass sounds (?), hardly audible
#	around 0.5xxxxx : nice chime sounds

value3=13
# initial value : 10
# tested values : 13, 17, 26
#	increasing values : more high-pitched sounds
#	26 makes some 'D2-R2' blips

value4=128	# no effect so far :-(
# initial value : 128

awk "function wl() {
		rate=64000;
		return (rate/$value1)*($value2^(int(rand()*$value3)))};
	BEGIN {
		srand();
		wla=wl();
		while(1) {
			wlb=wla;
			wla=wl();
			if (wla==wlb)
				{wla*=2;};
			d=(rand()*10+5)*rate/4;
			a=b=0; c=$value4;
			ca=40/wla; cb=20/wlb;
			de=rate/10; di=0;
			for (i=0;i<d;i++) {
				a++; b++; di++; c+=ca+cb;
				if (a>wla)
					{a=0; ca*=-1};
				if (b>wlb)
					{b=0; cb*=-1};
				if (di>de)
					{di=0; ca*=0.9; cb*=0.9};
				printf(\"%c\",c)};
			c=int(c);
			while(c!=$value4) {
				c<$value4?c++:c--;
				printf(\"%c\",c)};};}" | aplay -r 64000

BEGIN

a BEGIN rule is executed once only, before the first input record is read (example)

BEGINFILE

see ENDFILE

END

an END rule is executed once only, after all the input is read (example)

ENDFILE

This is a gawk extension. The ENDFILE rule :

• is called when gawk has finished processing the last record in an input file. For the last input file, it will be called before any END rules
• is executed even for empty input files
• allows to catch errors

A basic example

How to store the result of a system command into a variable ? (source)

awk 'BEGIN {
	myCommand = "date --iso-8601=seconds"
	myCommand | getline myResultVariable
	close(myCommand)
	print "Current date = "myResultVariable
	}'

Mixing Awk and Bash tests :

[ -e someFile ] && rm someFile; for i in 1 2; do
	echo -e 'hello someFile world' | awk '{ print "\ninput : "$0; if(system("[ -f "$2" ]")) { print $2" exists" } }'
	ls someFile; touch someFile
done; rm someFile

input : hello someFile world
someFile exists							mmmkay
ls: cannot access 'someFile': No such file or directory		make up your mind !

input : hello someFile world
someFile							

• doesn't support the .{n} syntax
• Only gawk supports {} (source)

ls -l $(which awk)

lrwxrwxrwx 1 root root 21 Oct 11 2016 /usr/bin/awk -> /etc/alternatives/awk*

md5sum /etc/alternatives/awk $(which gawk)

23a5b5a3d9ba0d2c6277dbdaf2557033	/etc/alternatives/awk
23a5b5a3d9ba0d2c6277dbdaf2557033	/usr/bin/gawk

Once gawk is installed, it can be invoked with awk.

$n

the n^th element of the current line ($0 being the whole line itself) :

for i in {1..4}; do echo 'a b c d' | awk '{print "Item '$i' of line \""$0"\" is "$'$i'"."}'; done

FILENAME

• name of the current input file
• - when reading from standard input
• empty string inside a BEGIN rule

FS

Field Separator. Can be set with -F

NF

• number of fields in the current line :
  echo 'a b c' | awk '{print NF}'; echo 'joe jack william averell' | awk '{print NF}';

  3
  4

• It is often used to refer to the last field of a line :
  echo 'a b c' | awk '{print $NF}'; echo 'joe jack william averell' | awk '{print $NF}';

  c
  averell

NR

number of records processed so far (which can be approximated to the number of the current row, starting at 1) :

for i in {a..e}; do echo $i; done | awk '{ print "line "NR":\t" $0}'

line 1: a
line 2: b
line 3: c
line 4: d
line 5: e

OFS

Output Field Separator. It is automatically inserted between fields by print. Defaults to a single space.

This is not a CLI flag, it goes into the "action" part :

• echo {a..z} | awk '{OFS="."; print $1,$3,$5,$7}'

  a.c.e.g

• No need to repeat the definition for every line of input :
  echo {a..z} | awk 'BEGIN{OFS="PLOP"} {print $1,$3,$5,$7}'

  aPLOPcPLOPePLOPg

RS

Records Separator

• defaults to \n (NEWLINE) : by default Awk considers 1 record == 1 line of input
• gawk also accepts regular expression

awk

Bash (source)

PERL

sed

Awk is a programmable text filter. Its input can be :

• a file : awk [options] myFile
• the standard input stdin : stream of text from previous command | awk [options]

• by default : the standard output stdout (source)
• any file you like with output redirection : awk [options] inputFile > outputFile
• Don't forget that awk's default action is to print matching lines. So any action limited to { print $0 }, albeit correct, is redundant .
• details on print

An Awk script is made of 3 blocks :

1. pre-process : BEGIN
2. process
3. post-process : END

Awk reads the input line by line, then applies the specified filter(s) to detect whether or not to process the current line. Before starting processing a line, Awk splits it into fields and stores fields values in $1 (1^st field), $2, ..., $NF (last field). $0 is the whole input line. The fields separator (specified with FS) defaults either to [SPACE] or [TAB] (details).

There is no need to use grep together with Awk as Awk "RegExp matches" lines to process.

Filters :

awk 'NR==n {doSomething}'

c

a
b
d

awk '!/regEx/ {doSomething}'

foo

field1	field2

bar	34
baz	56

awk '$n == "value" {doSomething}'

foo2 bar2 baz2

awk '$n != "value" {doSomething}'

foo1 bar1 baz1
foo3 bar3 baz3

awk '$n ~ /regEx/ {doSomething}'

awk '$n !~ /regEx/ {doSomething}'

foo2 bar2 baz2
foo3 bar3 baz3

NO MATCH
MATCH : bar2
NO MATCH

awk 'condition1 logicalOperator condition2 logicalOperator ... conditionN {doSomething}'

foo	8U	ooops !
bar	34U
baz	56U

bar	34U
baz	56U

awk 'BEGIN {
	print "trailing unit (single letter) : " strtonum("123U")
	print "trailing unit (word) : " strtonum("123potatoes")
	print "leading unit (single letter) : " strtonum("Y123")
	print "leading unit (word) : " strtonum("banana123")
	}'

trailing unit (single letter) : 123	OK
trailing unit (word) : 123		OK
leading unit (single letter) : 0	KO
leading unit (word) : 0			KO

• If Awk's exit statement is invoked with a numeric value, this numeric value is used as the exit status code.
• Otherwise (source) :

  Exit status	Description
  0 aka UNIX_SUCCESS	No problem during execution, including when no match was found. Check it : for char in a b; do echo "$char" | awk '/a/'; echo $?; echo; done a 0 0
  1 aka UNIX_FAILURE	An error occurred
  2	Fatal error

Process log files :

[13-Nov-2013 03:03:35 Europe/Paris] PHP Warning: Memcached::touch(): ... in ....php on line 45
[13-Nov-2013 03:04:42 Europe/Paris] PHP Warning: file_get_contents(http://...): HTTP/1.0 404 Not Found in ...php on line 202
...

extra examples to illustrate the above :

echo -e 'fruit: apple\nfruit: banana\nvegetable: carrot' | awk -v stuffToMatch='b' '/stuffToMatch/'
(nothing)
echo -e 'fruit: apple\nfruit: banana\nvegetable: carrot' | awk -v stuffToMatch='b' '$0 ~ /stuffToMatch/'
(nothing)
⇒ no match found, as said above


echo -e 'fruit: apple\nfruit: banana\nvegetable: carrot' | awk -v stuffToMatch='b' 'stuffToMatch'
fruit: apple
fruit: banana
vegetable: carrot
⇒ matches everything


echo -e 'fruit: apple\nfruit: banana\nvegetable: carrot' | awk -v stuffToMatch='b' '$0 ~ stuffToMatch'
fruit: banana
vegetable: carrot

echo -e 'fruit: apple\nfruit: banana\nvegetable: carrot' | awk -v stuffToMatch='b' '$1 ~ stuffToMatch'
vegetable: carrot

echo -e 'fruit: apple\nfruit: banana\nvegetable: carrot' | awk -v stuffToMatch='b' '$2 ~ stuffToMatch'
fruit: banana
⇒ 3 examples work as expected

Selecting PID's :

Specifying the field separator :

List all ports and PIDs on which a Mongodb instance is listening :

Select non-empty lines :

C
G
K

Dark wizardry ?

value1 value2