Awk - Description, flags and examples

awk: not an option: -i

Tutorials & basic examples :

convert sleep durations into human-readable durations (related: numbers with trailing unit letter)

for unit in '' s m; do
	for duration in 0 1 2 10 100; do
		value="$duration$unit"
		echo -n "'$value'."
		awk '
			/^[0-9]+$/	{ if($0 < 2) print $0 " second"; else print $0 " seconds"; }
			/^[0-9]+[ms]$/	{
				number=strtonum($0)
				unit=gensub(/^[0-9]+([ms])/, "\\1", "g");

				switch (unit) {
					case /m/:
						longUnit="minute"
						break
					case /s/:
						longUnit="second"
						break
					}
				 if(number > 1) print number" "longUnit"s"; else print number" "longUnit   ;
				 }
			' <<< "$value"
	done
done | column -s '.' -t

'0'     0 second
'1'     1 second
'2'     2 seconds
'10'    10 seconds
'100'   100 seconds
'0s'    0 second
'1s'    1 second
'2s'    2 seconds
'10s'   10 seconds
'100s'  100 seconds
'0m'    0 minute
'1m'    1 minute
'2m'    2 minutes
'10m'   10 minutes
'100m'  100 minutes

a dummy script with gensub, if else, length and logical operators

cat << EOF | awk '
BEGIN	{ followingWord=""; myVariable=""; }
/FOO/	{ followingWord=gensub(/.*FOO ([^ ]+).*/, "\\1", "g") }
/BAR/	{ myVariable="A" }
/BAZ/	{ myVariable="B" }
		{ if((length(followingWord) > 0) && (length(myVariable) > 0)) {
			print followingWord" ("length(followingWord)") "myVariable" ("length(myVariable)")."
			followingWord=""; myVariable="";
			}
		}
'
Lorem ipsum dolor sit amet, consectetur adipiscing elit,
sed do eiusmod tempor incididunt ut FOO labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco
laboris nisi ut aliquip ex BAR ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit
esse cillum dolore BAZ eu fugiat nulla pariatur.
EOF

labore (6) A (1).

List "enabled" repositories :

[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://download.docker.com/linux/centos/$releasever/source/stable
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

curl -s https://download.docker.com/linux/centos/docker-ce.repo | awk ' \
	BEGIN		{ repoId=""; } \
	/^ *$/		{ repoId=""; } \
	/^\[.*\]/	{ repoId=$0; } \
	/^enabled=1/	{ if (repoId != "") { result=gensub(/[\[\]]/, "", "g", repoId); print result; } }'

docker-ce-stable

Find who's the link / who's the target in a symlink :

while read line; do
	echo -e "\t$line"
	echo "$line" | awk '{ link=$1; target=""; for(i=2; i<=NF; i++) { if($i!="->") link=link" "$i; else break; } for(j=i+1; j<NF; j++) target=target""$j" "; target=target""$NF; print " LINK: \047"link"\047, TARGET: \047"target"\047"; }'
	echo
done < <(find -type l -exec ls -l {} + | awk '{ for (i=1; i<9; i++) $i=""; print $0; }')

#!/usr/bin/env bash

value1=120
# initial value : 160
# tested values : 200, 140, 100
#	decreasing from the initial value : more bass sounds

value2=0.5678
# initial value : 0.87055
# tested values : 0.747, 0.777, 0.789
#	increasing values above 3.xxx : extreme bass sounds (?), hardly audible
#	around 0.5xxxxx : nice chime sounds

value3=13
# initial value : 10
# tested values : 13, 17, 26
#	increasing values : more high-pitched sounds
#	26 makes some 'D2-R2' blips

value4=128	# no effect so far :-(
# initial value : 128

awk "function wl() {
		rate=64000;
		return (rate/$value1)*($value2^(int(rand()*$value3)))};
	BEGIN {
		srand();
		wla=wl();
		while(1) {
			wlb=wla;
			wla=wl();
			if (wla==wlb)
				{wla*=2;};
			d=(rand()*10+5)*rate/4;
			a=b=0; c=$value4;
			ca=40/wla; cb=20/wlb;
			de=rate/10; di=0;
			for (i=0;i<d;i++) {
				a++; b++; di++; c+=ca+cb;
				if (a>wla)
					{a=0; ca*=-1};
				if (b>wlb)
					{b=0; cb*=-1};
				if (di>de)
					{di=0; ca*=0.9; cb*=0.9};
				printf(\"%c\",c)};
			c=int(c);
			while(c!=$value4) {
				c<$value4?c++:c--;
				printf(\"%c\",c)};};}" | aplay -r 64000

BEGIN

a BEGIN rule is executed once only, before the first input record is read (example)

BEGINFILE

see ENDFILE

END

an END rule is executed once only, after all the input is read (example)

ENDFILE

This is a gawk extension. The ENDFILE rule :

• is called when gawk has finished processing the last record in an input file. For the last input file, it will be called before any END rules
• is executed even for empty input files
• allows to catch errors

• doesn't support the .{n} syntax
• Only gawk supports {} (source)

ls -l $(which awk)

lrwxrwxrwx 1 root root 21 Oct 11 2016 /usr/bin/awk -> /etc/alternatives/awk*

md5sum /etc/alternatives/awk $(which gawk)

23a5b5a3d9ba0d2c6277dbdaf2557033	/etc/alternatives/awk
23a5b5a3d9ba0d2c6277dbdaf2557033	/usr/bin/gawk

Once gawk is installed, it can be invoked with awk.

$n

the n^th element of the current line ($0 being the whole line itself) :

for i in {1..4}; do echo 'a b c d' | awk '{print "Item '$i' of line \""$0"\" is "$'$i'"."}'; done

FILENAME

• name of the current input file
• - when reading from standard input
• empty string inside a BEGIN rule

FS

Field Separator. Can be set with -F

NF

• number of fields in the current line :
  echo 'a b c' | awk '{print NF}'; echo 'joe jack william averell' | awk '{print NF}';

  3
  4

• It is often used to refer to the last field of a line :
  echo 'a b c' | awk '{print $NF}'; echo 'joe jack william averell' | awk '{print $NF}';

  c
  averell

NR

number of records processed so far (which can be approximated to the number of the current row, starting at 1) :

for i in {a..e}; do echo $i; done | awk '{ print "line "NR":\t" $0}'

line 1: a
line 2: b
line 3: c
line 4: d
line 5: e

OFS

Output Field Separator. It is automatically inserted between fields by print. Defaults to a single space.

This is not a CLI flag, it goes into the "action" part :

• echo {a..z} | awk '{OFS="."; print $1,$3,$5,$7}'

  a.c.e.g

• No need to repeat the definition for every line of input :
  echo {a..z} | awk 'BEGIN{OFS="PLOP"} {print $1,$3,$5,$7}'

  aPLOPcPLOPePLOPg

RS

Records Separator

• defaults to \n (NEWLINE) : by default Awk considers 1 record == 1 line of input
• gawk also accepts regular expression

awk

Bash (source)

PERL

sed

Awk is a programmable text filter. Its input can be :

• a file : awk [options] myFile
• the standard input stdin : stream of text from previous command | awk [options]

• by default : the standard output stdout (source)
• any file you like with output redirection : awk [options] inputFile > outputFile
• Don't forget that awk's default action is to print matching lines. So any action limited to { print $0 }, albeit correct, is redundant .
• details on print

An Awk script is made of 3 blocks :

1. pre-process : BEGIN
2. process
3. post-process : END

Awk reads the input line by line, then applies the specified filter(s) to detect whether or not to process the current line. Before starting processing a line, Awk splits it into fields and stores fields values in $1 (1^st field), $2, ..., $NF (last field). $0 is the whole input line. The fields separator (specified with FS) defaults either to [SPACE] or [TAB] (details).

There is no need to use grep together with Awk as Awk "RegExp matches" lines to process.

Filters :

awk 'NR==n {doSomething}'

c

a
b
d

awk '!/regEx/ {doSomething}'

foo

field1	field2

bar	34
baz	56

awk '$n == "value" {doSomething}'

foo2 bar2 baz2

awk '$n != "value" {doSomething}'

foo1 bar1 baz1
foo3 bar3 baz3

awk '$n ~ /regEx/ {doSomething}'

awk '$n !~ /regEx/ {doSomething}'

foo2 bar2 baz2
foo3 bar3 baz3

NO MATCH
MATCH : bar2
NO MATCH

awk 'condition1 logicalOperator condition2 logicalOperator ... conditionN {doSomething}'

foo	8U	ooops !
bar	34U
baz	56U

bar	34U
baz	56U

awk 'BEGIN {
	print "trailing unit (single letter) : " strtonum("123U")
	print "trailing unit (word) : " strtonum("123potatoes")
	print "leading unit (single letter) : " strtonum("Y123")
	print "leading unit (word) : " strtonum("banana123")
	}'

trailing unit (single letter) : 123	OK
trailing unit (word) : 123		OK
leading unit (single letter) : 0	KO
leading unit (word) : 0			KO

• If Awk's exit statement is invoked with a numeric value, this numeric value is used as the exit status code.
• Otherwise (source) :

  Exit status	Description
  0 aka UNIX_SUCCESS	No problem during execution, including when no match was found. Check it : for char in a b; do echo "$char" | awk '/a/'; echo $?; echo; done a 0 0
  1 aka UNIX_FAILURE	An error occurred
  2	Fatal error

Process log files :

[13-Nov-2013 03:03:35 Europe/Paris] PHP Warning: Memcached::touch(): ... in ....php on line 45
[13-Nov-2013 03:04:42 Europe/Paris] PHP Warning: file_get_contents(http://...): HTTP/1.0 404 Not Found in ...php on line 202
...

Selecting PID's :

Specifying the field separator :

List all ports and PIDs on which a Mongodb instance is listening :

Select non-empty lines :

C
G
K

Dark wizardry ?

value1 value2