An example is worth one thousand words., so enjoy your reading !
[docker-ce-stable] name=Docker CE Stable - $basearch baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable enabled=1 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg [docker-ce-stable-source] name=Docker CE Stable - Sources baseurl=https://download.docker.com/linux/centos/$releasever/source/stable enabled=0 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpgI want to list the IDs of those having
enabled=1
:
curl -s https://download.docker.com/linux/centos/docker-ce.repo | awk ' \ BEGIN { repoId=""; } \ /^ *$/ { repoId=""; } \ /^\[.*\]/ { repoId=$0; } \ /^enabled=1/ { if (repoId!="") { result=gensub(/[\[\]]/, "", "g", repoId); print result; } }'
docker-ce-stable
$0
variable$0
is the value of the previous line, i.e. the last line of input.{ print $0 }
is the default action)\1
to \9
. Since replacement is a quoted string, the \
characters need to be escaped, and callbacks will appear as \\n
g
or G
(i.e. "global") : replace all matches1
.$0
.o
/, "O
", 1); print result; }'hellO world
o
/, "O
", "g"); print result; }'hellO wOrld
The variable name passed to print does NOT need a leading $
.
X
:
^(..).(.*)
/, "\\1X\\2
", 1, $2); print $1" "result; next; } { print }'alpha brXvo charlie delta echo foXtrot
o
with 0
:o
/, "0
"); print }'e
with E
in the 2
nd word only :e
/, "E
", $2); print }'[]
:{ print $0 }
or even { print }
works, but is redundant.h e l l o
cdefghi
z y x
line 1 line 2 line 3
1 2 3
SPACE
(default) or the specified OFS1 line 2 line 3 line 1line 2line 3line
\n
to the output (printf doesn't) :
\n
printf(string, expression list)
syntax :
echo 'abc' | awk '{ switch ($0) { case /[[:lower:]]+/: print "lowercase" break case /[[:upper:]]+/: print "uppercase" break } }'As a one-liner that can be pasted into the shell :
echo 'abc' | awk '{ switch ($0) { case /[[:lower:]]+/: print "lowercase"; break; case /[[:upper:]]+/: print "uppercase"; break; } }'
1
0
found=1
(use whatever name and value you like )0
, we negate the flag value with !
(unknown number of lines before) (some text before)EXPECTED_TAG1(some text after) (unknown number of lines between) (some other text before)EXPECTED_TAG2(some other text after) (unknown number of lines after)
echo -e 'ga\nbu\nzo\nmeu' | awk -v RS='u' '/a.b/ {print $0}'ga becho -e 'ga\nbu\nzo\nmeu' | awk 'BEGIN {RS="u"} /a.b/ {print $0}'ga becho -e 'Super\ncali\nfragi\nlisti\ncexpia\nlido\ncious' | awk 'BEGIN {RS=".i"} {print $0}'Super ca fra s cex a do ous
#!/usr/bin/env bash value1=120 # initial value : 160 # tested values : 200, 140, 100 # decreasing from the initial value : more bass sounds value2=0.5678 # initial value : 0.87055 # tested values : 0.747, 0.777, 0.789 # increasing values above 3.xxx : extreme bass sounds (?), hardly audible # around 0.5xxxxx : nice chime sounds value3=13 # initial value : 10 # tested values : 13, 17, 26 # increasing values : more high-pitched sounds # 26 makes some 'D2-R2' blips value4=128 # no effect so far :-( # initial value : 128 awk "function wl() { rate=64000; return (rate/$value1)*($value2^(int(rand()*$value3)))}; BEGIN { srand(); wla=wl(); while(1) { wlb=wla; wla=wl(); if (wla==wlb) {wla*=2;}; d=(rand()*10+5)*rate/4; a=b=0; c=$value4; ca=40/wla; cb=20/wlb; de=rate/10; di=0; for (i=0;i<d;i++) { a++; b++; di++; c+=ca+cb; if (a>wla) {a=0; ca*=-1}; if (b>wlb) {b=0; cb*=-1}; if (di>de) {di=0; ca*=0.9; cb*=0.9}; printf(\"%c\",c)}; c=int(c); while(c!=$value4) { c<$value4?c++:c--; printf(\"%c\",c)};};}" | aplay -r 64000
1;unique1 2;duplicate1 3;unique2 4;duplicate1 5;unique3 6;duplicate2 7;duplicate2 8;unique4 9;duplicate1
duplicate1 duplicate2 duplicate1
BEGIN
rule is executed once only, before the first input record is read (example)BEGINFILE
END
rule is executed once only, after all the input is read (example)ENDFILE
rule :
END
rules.4;unique_1 3;duplicate 2;duplicate 1;unique_2
4;unique_1 3;duplicate 1;unique_2
$2
is the 2nd field of each processed line (here : the one having duplicates)seen[n]
is the item with index n within the seen array (the array name is arbitrary and was chosen for readability)++
is performed after !
(source)!
) previously set.
;
' -uk2 | sort -nrk1cat -n << EOF | sort -t ';
' -uk2 | sort -nrk1 | cut -f 2-
keepMe;unique_1
keepMe;duplicate
keepMe;duplicate
keepMe;unique_2
EOF
;
' -uk2 | sort -nrk1 | cut -f 2-; rm "$tmpFile"system(myCommand)
myVariable=system(myCommand)
stores the return code of myCommand into myVariable
0 1
myCommand | getline
awk 'BEGIN { myCommand = "date --iso-8601=seconds" myCommand | getline myResultVariable close(myCommand) print "Current date = "myResultVariable }'
SPACE
echo -e 'A B C D\nE F G H\nI J K L' | awk '{ for (i=2; i<=NF; i++) printf $i" "; print "" }' | xargs -I truc echo "'truc'" 'B C D ' 'F G H ' 'J K L ' ==> this comes from : printf $i" "
SPACE
after every field, including the removed ones.
23a5b5a3d9ba0d2c6277dbdaf2557033 /etc/alternatives/awk 23a5b5a3d9ba0d2c6277dbdaf2557033 /usr/bin/gawk
Once gawk is installed, it can be invoked with awk.
awk '/a regular expression/ {deal with it}' myFile
Since the regular expression is/
-delimited, we have to escape those found in the regular expression itself :
echo -e '/a/\n/1/\n/b/\n/2/\n/c/\n/3/' | awk '/\/[a-z]\// {print}'
It it sometimes possible to workaround this :
echo -e '/foo/foo/\n/foo/bar/\n/bar/bar/\n/foo/baz/' | awk 'BEGIN {myRegex = "/foo/.a./
"} $0 ~ myRegex {print}'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++===================================================================================================
ii lsb-base 9.20161125 all Linux Standard Base init script functionality
un lsb-core <none> <none> (no description available) not installed
ii lsb-release 9.20161125 all Linux Standard Base version reporting utility
ii
|\|
\|
)/ {print $0}'
||/ Name Version Architecture Description ii lsb-base 9.20161125 all Linux Standard Base init script functionality ii lsb-release 9.20161125 all Linux Standard Base version reporting utility
ii
|\|
\|
)/ { for (i=2; i<=4; i++) printf $i"'$fieldSeparator'"; for (i=5; i<=NF; i++) printf $i" "; print ""}' | column -s "$fieldSeparator" -t
Name Version Architecture Description lsb-base 9.20161125 all Linux Standard Base init script functionality lsb-release 9.20161125 all Linux Standard Base version reporting utilityHow it works :
for (i=2; i<=4; i++) printf $i"'$fieldSeparator'"
command prints name + version + architecture (with a separator)for (i=5; i<=NF; i++) printf $i" "
command prints the description as a single data field (no separator added)awk '/pattern/ { x = NR + n } NR == x' someFile
{ print $0 }
is the default action.for i in {0..9}; do echo "line $i"; done | awk '/line 4/ {lineToDisplay = NR + 3} NR == lineToDisplay'
line 7
for i in {0..9}; do echo "line $i"; done | grep -B3 'line 4' | head -1
$0
being the whole line itself) :3 4
c averell
line 1: a line 2: b line 3: c line 4: d line 5: e
a.c.e.g
aPLOPcPLOPePLOPg
\n
(NEWLINE
) : by default Awk considers 1 record == 1 line of input~ is the Awk operator to match a regular expression.
logFile='/var/log/lighttpd/www.example.com.log'; resultFile='./result.csv'; tmpFile=$(mktemp --tmpdir tmp.result.XXXXXXXX); grep '" 500 ' $logFile | sed -r 's/^.*GET ([^ ]*).*HTTP\/1\.." ([0-9]*).*$/\1;\2;/' > $tmpFile; cat $tmpFile | sort | uniq -c | sort -nr > $resultFile; rm $tmpFile
Using grep 1st because sed can't find a match on every line, as we're reporting only on HTTP 500 errors.
Awk is a programmable text filter. Its input can be :
{ print $0 }
, albeit correct, is redundant .An Awk script is made of 3 blocks :
Awk reads the input line by line, then applies the specified filter(s) to detect whether or not to process the current line. Before starting processing a line, Awk splits it into fields and stores fields values in $1
(1st field), $2
, ..., $NF
(last field). $0
is the whole input line. The fields separator (specified with FS) defaults either to [SPACE] or [TAB] (details).
There is no need to use grep together with Awk as Awk "RegExp matches" lines to process.
Criteria | select matching lines | select not matching lines |
---|---|---|
line number within input |
awk 'NR==n {doSomething}' c |
a b d |
line vs regular expression |
awk '/regEx/ {doSomething}'
|
awk '!/regEx/ {doSomething}'echo -e 'foo\nbar\nbaz' | awk '!/a/ {print $0}' foo(source, example) |
line vs number of fields |
echo -e 'field1\nfield1\tfield2\nfield1\tfield2\tfield3' | awk 'NF == 2 {print $0}'
field1 field2 |
|
field vs number |
echo -e 'foo\t12\nbar\t34\nbaz\t56' | awk '$2 > 25 {print $0}'
bar 34 baz 56 Awk is smart enough to strip leading zeroes :
Trying to filter data based on line numbers returned by grep -n with a construct like :may fail because of the returned color codes.
|
|
field vs string |
awk '$n == "value" {doSomething}'for i in {1..3}; do echo "foo$i bar$i baz$i"; done | awk '$2 == "bar2" {print $0}' foo2 bar2 baz2 |
awk '$n != "value" {doSomething}'for i in {1..3}; do echo "foo$i bar$i baz$i"; done | awk '$2 != "bar2" {print $0}' foo1 bar1 baz1 foo3 bar3 baz3 |
field vs regular expression |
awk '$n ~ /regEx/ {doSomething}'
|
awk '$n !~ /regEx/ {doSomething}'for i in {1..3}; do echo "foo$i bar$i baz$i"; done | awk '$2 !~ /a.1/ {print $0}' foo2 bar2 baz2 foo3 bar3 baz3 |
field vs regular expression with if / else construct (source) |
for i in {1..3}; do echo "foo$i bar$i baz$i"; done | awk '{ if($2 ~ "a.2") {print "MATCH : "$2 } else {print "NO MATCH"} }'
NO MATCH MATCH : bar2 NO MATCH |
|
several conditions |
awk 'condition1 logicalOperator condition2 logicalOperator ... conditionN {doSomething}'logicalOperator can be (source) :
|
If the numerical value has a unit letter, it doesn't work anymore :
foo 8U ooops !
bar 34U
baz 56U
solution :
bar 34U baz 56U
Try it :
%
/, "", $5)} {if(strtonum($5) > 50) {print $0}}'
awk 'BEGIN { print "trailing unit (single letter) : " strtonum("123U") print "trailing unit (word) : " strtonum("123potatoes") print "leading unit (single letter) : " strtonum("Y123") print "leading unit (word) : " strtonum("banana123") }'
trailing unit (single letter) : 123 OK trailing unit (word) : 123 OK leading unit (single letter) : 0 KO leading unit (word) : 0 KO
Flag | Usage |
---|---|
-F x | use x as the input Field separator
|
-v variable=value | declare a variable (example) |
Exit status | Description |
---|---|
0 aka UNIX_SUCCESS |
No problem during execution, including when no match was found. Check it : a 0 0 |
1 aka UNIX_FAILURE |
An error occurred |
2 |
Fatal error |
printf adds no carriage return after printing. print does.
Then count occurrences :[13-Nov-2013 03:03:35 Europe/Paris] PHP Warning: Memcached::touch(): ... in ....php on line 45 [13-Nov-2013 03:04:42 Europe/Paris] PHP Warning: file_get_contents(http://...): HTTP/1.0 404 Not Found in ...php on line 202 ...let's say you'd like to remove the date/time field to group and count similar errors. To do so :
awk '{ for (i=1;i<=3;i++) $i="";print }' file.log | awk '{sub(/^[ \t]+/, ""); print}' | sort | uniq -c | sort -nr
Looks like you can't use a variable name within the //
operator to select the matching a line :
echo -e 'A\tB\tC\tD\nE\tF\tG\tH\n\nI\tJ\tK\tL' | awk '!/^$/ {print $3}'
C G K
value1 value2
/key1|key2/
part of the awk command is a "normal" regular expression alternationprintf $2 " "
part simply prints the 2nd field of each matching line, followed by a space