Bash Index : J - The 'J' Bash commands : description, flags and examples

mail

jq

Usage

jq is like sed for JSON data — you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

While working / experimenting with JSON files, remember they can be opened + viewed + folded / unfolded with Firefox .

Flags

Flag Usage
-c --compact-output By default, jq pretty-prints JSON output. Using this option will result in more compact output by instead putting each JSON object on a single line.
-M --monochrome-output disable color : by default, jq outputs colored JSON if writing to a terminal
-r --raw-output output string without quotes

Example

see The Ultimate Interactive JQ Guide 
resultFile='./result.json'
taigaApiUrl='https://taiga.example.com/api/v1'
taigaProjectId='2949'

username='bob'
password='123456789'

authToken=$(curl -s --noproxy "*" --insecure -X POST -H "Content-Type: application/json" -d '{ "username": "'$username'", "password": "'$password'", "type": "normal" }' "$taigaApiUrl/auth" | jq -r '.auth_token')

[ -f "$resultFile" ] && rm "$resultFile"
curl -s --noproxy "*" --insecure -X GET -H "Content-Type: application/json" -H "Authorization: Bearer ${authToken}" "$taigaApiUrl/userstories?project=$taigaProjectId" > "$resultFile"

# source : http://bigdatums.net/2017/05/13/count-json-array-elements-with-jq/
nbUserStories=$(jq '. | length' "$resultFile")
echo "'$nbUserStories' user stories"

# List the user stories IDs :
# jq '.[29].ref' result.json
# for i in {0..29}; do jq ".[$i].ref" result.json; done

# [] is an iterator, no need of 'for' loop \o/
# https://stedolan.github.io/jq/manual/#Basicfilters
#jq '.[].ref' result.json
#jq ".[].subject" result.json

# to get both :
# source : https://stackoverflow.com/questions/28164849/using-jq-to-parse-and-display-multiple-fields-in-a-json-serially#answer-31418194
#jq -r '.[] | "\(.ref) ==> \(.subject)"' result.json

# get tags :
# thanks to : https://stackoverflow.com/questions/41712363/basic-jq-usage-how-to-get-nested-value
#jq -r '.[].tags | .[] | .[0]' result.json

echo 'Tags :'
jq -r '.[].tags | .[] | .[]' "$resultFile" | sed '/^null$/d' | sort | uniq -c | sort -nr
mail

journalctl

Usage

Query the systemd journal. It is possible to filter results with :
Wrapping long lines is not very intuitive.

Permissions

  • All users are granted access to their private per-user journals.
  • Only and users who are members of a few special groups are granted access to :
    • the system journal
    • journals of other users
  • Members of the groups systemd-journal, adm, and wheel can read all journal files. Note that the two latter groups traditionally have additional privileges specified by the distribution. Members of the wheel group can often perform administrative tasks.
To read all journals as a non- user, you can either :

FIELD=VALUE CLI arguments

List of available fields : man 7 systemd.journal-fields
Field Usage example Comments
MESSAGE=message The human-readable message string for this entry. journalctl MESSAGE='Failed unmounting /var.' Looks like message has to be the exact message to be matched. No pattern or partial message seems to match.
PRIORITY severity or verbosity level, based on syslog priority concept journalctl PRIORITY=3 syslog's severity levels
_COMM
_EXE
_CMDLINE		 The name, the executable path, and the command line of the process the journal entry originates from. journalctl _COMM=pulseaudio
_SYSTEMD_UNIT systemd unit name (if any)
  • journalctl _SYSTEMD_UNIT=mysql
  • journalctl -u mysql
 Looks pretty similar to -u / --unit, but gives slightly different results
_TRANSPORT=value How the entry was received by the journal service. value is one of :
  • audit : for those read from the kernel audit subsystem
  • driver : for internally generated messages
  • journal : for those received via the native journal protocol
  • kernel : for those read from the kernel
  • stdout : for those read from a service's standard output or error output
  • syslog : for those received via the local syslog socket with the syslog protocol

Advanced filtering

logical AND
several FIELD=VALUE arguments on the same command are combined into a logical AND, displaying logs entries matching ALL criteria :
journalctl _SYSTEMD_UNIT=mysql.service PRIORITY=3
logical OR
A + between terms will combine the one before and the one after into a logical OR
journalctl _SYSTEMD_UNIT=mysql.service + PRIORITY=3
alternatives
If two matches apply to the same field, then they are automatically matched as alternatives, i.e. the resulting output will show entries matching any of the specified matches for the same field :
journalctl _SYSTEMD_UNIT=mysql.service PRIORITY=2 PRIORITY=3

Flags

Flag Usage
--disk-usage Shows the current disk usage of all journal files : the sum of the disk usage of all archived and active journal files.
-f --follow Show only the most recent journal entries, and continuously print new entries as they are appended to the journal (i.e. like tail -f)
--no-pager Do not pipe output into a pager
a pager is a utility used to display text, scroll up/down, scroll 1line/1page, search, ..., such as less
-o outputOption
--output=outputOption		 Controls the formatting of the journal entries that are shown. Among available values : verbose
-r --reverse reverse output : newest entries first
-S 'YYYY-MM-DD hh:mm:ss'
--since='YYYY-MM-DD hh:mm:ss'		 Show entries on/newer than the specified timestamp. Missing time will be interpreted as 00:00:00, missing seconds will be interpreted as :00
-u unit --unit=unit Show messages for the specified systemd unit unit (list available units)
-U 'YYYY-MM-DD hh:mm:ss'
--until='YYYY-MM-DD hh:mm:ss'		 Show entries on/older than the specified timestamp. Missing time will be interpreted as 00:00:00, missing seconds will be interpreted as :00
--vacuum-files=finalNumberOfLogfiles Remove archived journal files until there remain only finalNumberOfLogfiles logfiles
this will leave the active journal files untouched
--vacuum-size=finalLogsSize
  • Remove archived journal files until the disk space they use falls below finalLogsSize
  • finalLogsSize can be specified with a suffix such as K, M, G or T
this will leave the active journal files untouched
--vacuum-* functions appeared with systemd v2.18 (source). On prior versions, this will cause errors such as : 
journalctl: unrecognized option '--vacuum-size=1G' will appear
In such situation, to clear logs, you'll have to run, with privileges :
  1. find /var/log/journal -name '*journal' -a -ctime +180 -delete
  2. journalctl --disk-usage
--vacuum-time=timespan
  • Remove archived journal files until they contain no data older than timespan
  • timespan can be specified with a suffix such as s, m, h, days, weeks, months or years

Example

Some commands :

journalctl -u haproxy --since '2017-01-03 11:15' --until '2017-01-03 11:20' --output=verbose
journalctl -u haproxy --since '2017-01-03 11:15' --until '2017-01-03 11:20' --output=verbose | grep antispam
journalctl -u haproxy --since '2017-01-03 11:15' --until '2017-01-03 11:20' | awk '{print $6}'

How to wrap long lines ?

Lines output by journalctl are often longer than the display width, and the interesting part of messages is generally at the end of the line (i.e. not visible )

  • use the right arrow key → to view long lines
  • or (details below) : journalctl -u networking --no-pager | less
  • man journalctl is not very helpful when it comes to wrapping long lines (search wrap and pager for details). To view long lines, you may run :
    journalctl -u networking --no-pager
    ...which will throw everything to the screen, making it impossible to read. So you'll end up with :
    journalctl -u networking --no-pager | less
    i.e. asking no pager, please in the first place, then piping everything into ... a pager .
  • this construct is not necessary when greping something that is hidden by the default pager :
    journalctl -u networking --output=verbose | grep foo
    will actually find foo, even though foo is beyond the right screen border when running :
    journalctl -u networking --output=verbose

journalctl errors

Failed to determine timestamp: Cannot assign requested address
I observed this when running commands like :
journalctl -u haproxy --until '2018-03-24 10:00'
whereas this returns no log entry :
journalctl -u haproxy
Looks like "over-filtering" with -u + --until causes it.
mail

join

Usage

Join 2 files on their matching records, as does the SQL operator.

Example

The Dalton's :

Considering 2 files :

  • names.txt
    Joe Dalton
Jack Dalton
William Dalton
Averell Dalton
  • ages.txt
    Joe 23
Jack 22
William 21
Averell 20

join names.txt ages.txt returns :

Joe Dalton 23
Jack Dalton 22
William Dalton 21
Averell Dalton 20