Awk - Description, flags and examples

Tutorials & basic examples :

List "enabled" repositories :

[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

[docker-ce-stable-source]
name=Docker CE Stable - Sources
baseurl=https://download.docker.com/linux/centos/$releasever/source/stable
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg

curl -s https://download.docker.com/linux/centos/docker-ce.repo | awk ' \
	BEGIN		{ repoId=""; } \
	/^ *$/		{ repoId=""; } \
	/^\[.*\]/	{ repoId=$0; } \
	/^enabled=1/	{ if (repoId!="") { result=gensub(/[\[\]]/, "", "g", repoId); print result; } }'

docker-ce-stable

• Awk loops on all lines of input
• at each iteration, Awk stores the contents of each line into its $0 variable
• THEN —when present— the END clause is executed. At that time, the value of $0 is the value of the previous line, i.e. the last line of input.

display the first line (remember : { print $0 } is the default action)

seq 0 9 | awk 'NR==1'

display the last line

seq 0 9 | awk 'END {print $0}'

our test matrix

for i in {1..5}; do for j in {1..5}; do echo -n "L${i}C$j "; done; echo; done

display the full line 1

for i in {1..5}; do for j in {1..5}; do echo -n "L${i}C$j "; done; echo; done | awk 'NR==1'

display the line 1, field 3

for i in {1..5}; do for j in {1..5}; do echo -n "L${i}C$j "; done; echo; done | awk 'NR==1 {print $3}'

display the full last line

for i in {1..5}; do for j in {1..5}; do echo -n "L${i}C$j "; done; echo; done | awk 'END {print $0}'

display the last line, field 4

for i in {1..5}; do for j in {1..5}; do echo -n "L${i}C$j "; done; echo; done | awk 'END {print $4}'

• the replacement string can refer to subparts of the matched string using callbacks
• gensub returns the modified string as the result of the function. The original target string is not changed. So you'll have to explicitly store the result in a variable and print it.

• regexp : explicit
• replacement : callbacks are \1 to \9. Since replacement is a quoted string, the \ characters need to be escaped, and callbacks will appear as \\n
• how :
  • if it starts with g or G (i.e. "global") : replace all matches
  • otherwise, consider it as a number indicating which match of regexp to replace. If < 1, assume 1.
• target : the field to search+replace into. If missing, assume $0.

Basic examples :

A not-so-basic example :

alpha brXvo
charlie delta
echo foXtrot

replace all o with 0 :

echo 'hello world' | awk '{ gsub(/o/, "0"); print }'

hell0 w0rld

replace all e with E in the 2^nd word only :

echo 'happy halloween' | awk '{ gsub(/e/, "E", $2); print }'

happy hallowEEn

remove square brackets [] :

echo '[foo]' | awk '{ gsub(/[\[\]]/, ""); print }'

foo

• By default, Awk prints the matching lines, so anything like { print $0 } or even { print } works, but is redundant.
• There are some slight differences between print and printf
• There are many ways of specifying which fields from the input should be output :
  • list them explicitly :
    echo {a..z} | awk '{ print $8, $5, $12, $12, $15 }'

    h e l l o

  • list them in a loop :
    echo {a..z} | awk '{ for(i=3; i<10; i++) printf $i }'

    cdefghi

  • list them counting from the end of the input line, using NF :
    echo {a..z} | awk '{ print $NF, $(NF-1), $(NF-2) }'

    z y x

The basic command everybody knows :

grep -E '<article id=‌"[^"]' *xml | wc -l

This construct is relevant for several files only. Even though "it works", applying it to a single file is a UUOW .

An alternate awk-based "pipe-less" solution :

awk 'BEGIN {i=0;} /<article id="[^"]/ {i++;} END { print i;}' *xml

print :

printf :

sprintf :

echo 'abc' | awk '{
	switch ($0) {
		case /[[:lower:]]+/:
			print "lowercase"
			break
		case /[[:upper:]]+/:
			print "uppercase"
			break
		}
	}'

echo 'abc' | awk '{ switch ($0) { case /[[:lower:]]+/: print "lowercase"; break; case /[[:upper:]]+/: print "uppercase"; break; } }'

1

0

Explanations

(unknown number of lines before)

(some text before)EXPECTED_TAG1(some text after)

(unknown number of lines between)

(some other text before)EXPECTED_TAG2(some other text after)

(unknown number of lines after)

echo -e 'ga\nbu\nzo\nmeu' | awk -v RS='u' '/a.b/ {print $0}'
ga
b

echo -e 'ga\nbu\nzo\nmeu' | awk 'BEGIN {RS="u"} /a.b/ {print $0}'
ga
b


echo -e 'Super\ncali\nfragi\nlisti\ncexpia\nlido\ncious' | awk 'BEGIN {RS=".i"} {print $0}'
Super
ca

fra


s

cex
a

do

ous

1. convert the input into a single giant line of text with tr
2. search strings as usual with your favorite tool

#!/usr/bin/env bash

value1=120
# initial value : 160
# tested values : 200, 140, 100
#	decreasing from the initial value : more bass sounds

value2=0.5678
# initial value : 0.87055
# tested values : 0.747, 0.777, 0.789
#	increasing values above 3.xxx : extreme bass sounds (?), hardly audible
#	around 0.5xxxxx : nice chime sounds

value3=13
# initial value : 10
# tested values : 13, 17, 26
#	increasing values : more high-pitched sounds
#	26 makes some 'D2-R2' blips

value4=128	# no effect so far :-(
# initial value : 128

awk "function wl() {
		rate=64000;
		return (rate/$value1)*($value2^(int(rand()*$value3)))};
	BEGIN {
		srand();
		wla=wl();
		while(1) {
			wlb=wla;
			wla=wl();
			if (wla==wlb)
				{wla*=2;};
			d=(rand()*10+5)*rate/4;
			a=b=0; c=$value4;
			ca=40/wla; cb=20/wlb;
			de=rate/10; di=0;
			for (i=0;i<d;i++) {
				a++; b++; di++; c+=ca+cb;
				if (a>wla)
					{a=0; ca*=-1};
				if (b>wlb)
					{b=0; cb*=-1};
				if (di>de)
					{di=0; ca*=0.9; cb*=0.9};
				printf(\"%c\",c)};
			c=int(c);
			while(c!=$value4) {
				c<$value4?c++:c--;
				printf(\"%c\",c)};};}" | aplay -r 64000

I have :

1;unique1
2;duplicate1
3;unique2
4;duplicate1
5;unique3
6;duplicate2
7;duplicate2
8;unique4
9;duplicate1

I want :

Anything stating whether some values are duplicated across lines

duplicate1
duplicate2
duplicate1

BEGIN

a BEGIN rule is executed once only, before the first input record is read (example)

BEGINFILE

see ENDFILE

END

an END rule is executed once only, after all the input is read (example)

ENDFILE

This is a gawk extension. The ENDFILE rule :

• is called when gawk has finished processing the last record in an input file. For the last input file, it will be called before any END rules.
• is executed even for empty input files.
• allows to catch errors.

next

immediately stop processing the current record and go on to the next record

I have :

4;unique_1
3;duplicate
2;duplicate
1;unique_2

I want :

4;unique_1
3;duplicate
1;unique_2

echo -e '4;unique_1\n3;duplicate\n2;duplicate\n1;unique_2' | awk -F ';' '!seen[$2]++'

• $2 is the 2^nd field of each processed line (here : the one having duplicates)
• seen[n] is the item with index n within the seen array (the array name is arbitrary and was chosen for readability)
• ++ is performed after ! (source)

• echo -e '4;unique_1\n3;duplicate\n2;duplicate\n1;unique_2' | sort -t ';' -uk2 | sort -nrk1
• To keep the original lines order, prepend line numbers (with cat -n), and remove them (with cut -f 2-) :

  • cat -n << EOF | sort -t ';' -uk2 | sort -nrk1 | cut -f 2-
    keepMe;unique_1
    keepMe;duplicate
    keepMe;duplicate
    keepMe;unique_2
    EOF

  • tmpFile=$(mktemp); echo -e 'keepMe;unique_1\nkeepMe;duplicate\nkeepMe;duplicate\nkeepMe;unique_2' > "$tmpFile"; cat -n "$tmpFile" | sort -t ';' -uk2 | sort -nrk1 | cut -f 2-; rm "$tmpFile"

A basic example

How to store the result of a system command into a variable ? (source)

awk 'BEGIN {
	myCommand = "date --iso-8601=seconds"
	myCommand | getline myResultVariable
	close(myCommand)
	print "Current date = "myResultVariable
	}'

echo -e 'A B C D\nE F G H\nI J K L' | awk '{ for (i=2; i<=NF; i++) printf $i" "; print "" }'

• easy when the field to skip is the first or the last one, tough otherwise
• leaves a trailing SPACE

  echo -e 'A B C D\nE F G H\nI J K L' | awk '{ for (i=2; i<=NF; i++) printf $i" "; print "" }' | xargs -I truc echo "'truc'"
  'B C D '
  'F G H '
  'J K L '
  ==> this comes from :
  	printf $i" "

echo -e 'A B C D\nE F G H\nI J K L' | awk '{ $1=""; printf $0; print "" }'

• prints a SPACE after every field, including the removed ones.
  To get rid of the leading space after removing the 1^st field :

  echo -e 'A B C D\nE F G H\nI J K L' | awk '{ $1=""; printf $0; print "" }' | sed -r 's/^ *(.*$)/\1/'
• works with any field (first, latest, middle), and any number of fields :
  echo -e 'A B C D\nE F G H\nI J K L' | awk '{ $1=$3=""; printf $0; print "" }'

• doesn't support the .{n} syntax
• Only gawk supports {} (source)

ls -l $(which awk)

lrwxrwxrwx 1 root root 21 Oct 11 2016 /usr/bin/awk -> /etc/alternatives/awk*

md5sum /etc/alternatives/awk $(which gawk)

23a5b5a3d9ba0d2c6277dbdaf2557033	/etc/alternatives/awk
23a5b5a3d9ba0d2c6277dbdaf2557033	/usr/bin/gawk

Once gawk is installed, it can be invoked with awk.

awk '/a regular expression/ {deal with it}' myFile

echo -e '/a/\n/1/\n/b/\n/2/\n/c/\n/3/' | awk '/\/[a-z]\// {print}'

It it sometimes possible to workaround this :

echo -e '/foo/foo/\n/foo/bar/\n/bar/bar/\n/foo/baz/' | awk 'BEGIN {myRegex = "/foo/.a./"} $0 ~ myRegex {print}'

Here's a very basic example showing the amount of RAM installed, in GiB :

1. dpkg -l lsb-*

   Desired=Unknown/Install/Remove/Purge/Hold
   | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
   |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
   ||/	Name		Version		Architecture	Description
   +++===================================================================================================
   ii	lsb-base	9.20161125	all		Linux Standard Base init script functionality
   un	lsb-core	<none>		<none>		(no description available)				not installed
   ii	lsb-release	9.20161125	all		Linux Standard Base version reporting utility

2. I want installed packages only, so let's start filtering : dpkg -l lsb-* | awk '/^(ii|\|\|)/ {print $0}'

   ||/	Name		Version		Architecture	Description
   ii	lsb-base	9.20161125	all		Linux Standard Base init script functionality
   ii	lsb-release	9.20161125	all		Linux Standard Base version reporting utility

3. Looks better. Now I'd like to get rid of the leading ii . Here come the magic : fieldSeparator='|'; dpkg -l lsb-* | awk '/^(ii|\|\|)/ { for (i=2; i<=4; i++) printf $i"'$fieldSeparator'"; for (i=5; i<=NF; i++) printf $i" "; print ""}' | column -s "$fieldSeparator" -t

   Name		Version		Architecture	Description
   lsb-base	9.20161125	all		Linux Standard Base init script functionality
   lsb-release	9.20161125	all		Linux Standard Base version reporting utility

   How it works :
   • All lines returned by dpkg are made of several fields : field 1 is ii , field 2 is the name, field 3 is the version, field 4 is the architecture and fields 5 to NF are the description
   • All description fields have a different number of words, so NF is different for each returned line
   • the for (i=2; i<=4; i++) printf $i"'$fieldSeparator'" command prints name + version + architecture (with a separator)
   • the for (i=5; i<=NF; i++) printf $i" " command prints the description as a single data field (no separator added)
   • all of this is fed into column for a nice tabular display and voilà!

Display only the n^th line after the one matching pattern (source) :

line 7

Display only the n^th line before the one matching pattern :

$n

the n^th element of the current line ($0 being the whole line itself) :

for i in {1..4}; do echo 'a b c d' | awk '{print "Item '$i' of line \""$0"\" is "$'$i'"."}'; done

FS

Field Separator. Can be set with -F

NF

• number of fields in the current line :
  echo 'a b c' | awk '{print NF}'; echo 'joe jack william averell' | awk '{print NF}';

  3
  4

• It is often used to refer to the last field of a line :
  echo 'a b c' | awk '{print $NF}'; echo 'joe jack william averell' | awk '{print $NF}';

  c
  averell

NR

number of records processed so far (which can be approximated to the number of the current row, starting at 1) :

for i in {a..e}; do echo $i; done | awk '{ print "line "NR":\t" $0}'

line 1: a
line 2: b
line 3: c
line 4: d
line 5: e

OFS

Output Field Separator. It is automatically inserted between fields by print. Defaults to a single space.

This is not a CLI flag, it goes into the "action" part :

• echo {a..z} | awk '{OFS="."; print $1,$3,$5,$7}'

  a.c.e.g

• No need to repeat the definition for every line of input :
  echo {a..z} | awk 'BEGIN{OFS="PLOP"} {print $1,$3,$5,$7}'

  aPLOPcPLOPePLOPg

RS

Records Separator

• defaults to \n (NEWLINE) : by default Awk considers 1 record == 1 line of input
• gawk also accepts regular expression

awk

Bash (source)

PERL

sed

Awk is a programmable text filter. Its input can be :

• a file : awk [options] myFile
• the standard input stdin : stream of text from previous command | awk [options]

• by default : the standard output stdout (source)
• any file you like with output redirection : awk [options] inputFile > outputFile
• Don't forget that awk's default action is to print matching lines. So any action limited to { print $0 }, albeit correct, is redundant .
• details on print

An Awk script is made of 3 blocks :

1. pre-process : BEGIN
2. process
3. post-process : END

Awk reads the input line by line, then applies the specified filter(s) to detect whether or not to process the current line. Before starting processing a line, Awk splits it into fields and stores fields values in $1 (1^st field), $2, ..., $NF (last field). $0 is the whole input line. The fields separator (specified with FS) defaults either to [SPACE] or [TAB] (details).

There is no need to use grep together with Awk as Awk "RegExp matches" lines to process.

Filters :

awk 'NR==n {doSomething}'

c

a
b
d

awk '!/regEx/ {doSomething}'

foo

field1	field2

bar	34
baz	56

awk '$n == "value" {doSomething}'

foo2 bar2 baz2

awk '$n != "value" {doSomething}'

foo1 bar1 baz1
foo3 bar3 baz3

awk '$n ~ /regEx/ {doSomething}'

awk '$n !~ /regEx/ {doSomething}'

foo2 bar2 baz2
foo3 bar3 baz3

NO MATCH
MATCH : bar2
NO MATCH

awk 'condition1 logicalOperator condition2 logicalOperator ... conditionN {doSomething}'

foo	8U	ooops !
bar	34U
baz	56U

bar	34U
baz	56U

awk 'BEGIN {
	print "trailing unit (single letter) : " strtonum("123U")
	print "trailing unit (word) : " strtonum("123potatoes")
	print "leading unit (single letter) : " strtonum("Y123")
	print "leading unit (word) : " strtonum("banana123")
	}'

trailing unit (single letter) : 123	OK
trailing unit (word) : 123		OK
leading unit (single letter) : 0	KO
leading unit (word) : 0			KO

• If Awk's exit statement is invoked with a numeric value, this numeric value is used as the exit status code.
• Otherwise (source) :

  Exit status	Description
  0 aka UNIX_SUCCESS	No problem during execution, including when no match was found. Check it : for char in a b; do echo "$char" | awk '/a/'; echo $?; echo; done a 0 0
  1 aka UNIX_FAILURE	An error occurred
  2	Fatal error

Process log files :

[13-Nov-2013 03:03:35 Europe/Paris] PHP Warning: Memcached::touch(): ... in ....php on line 45
[13-Nov-2013 03:04:42 Europe/Paris] PHP Warning: file_get_contents(http://...): HTTP/1.0 404 Not Found in ...php on line 202
...

Selecting PID's :

Specifying the field separator :

List all ports and PIDs on which a Mongodb instance is listening :

Select non-empty lines :

C
G
K

Dark wizardry ?

value1 value2