Bash Index : J - The 'J' Bash commands : description, flags and examples

jq

Usage :

jq is like sed for JSON data — you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

While working / experimenting with JSON files, remember they can be opened + viewed + folded / unfolded with Firefox .
resultFile='./result.json'
taigaApiUrl='https://taiga.example.com/api/v1'
taigaProjectId='2949'

username='bob'
password='123456789'


# jq -r : outputs string without quotes
authToken=$(curl -s --noproxy "*" --insecure -X POST -H "Content-Type: application/json" -d '{ "username": "'$username'", "password": "'$password'", "type": "normal" }' "$taigaApiUrl/auth" | jq -r '.auth_token')


[ -f "$resultFile" ] && rm "$resultFile"
curl -s --noproxy "*" --insecure -X GET -H "Content-Type: application/json" -H "Authorization: Bearer ${authToken}" "$taigaApiUrl/userstories?project=$taigaProjectId" > "$resultFile"


# source : http://bigdatums.net/2017/05/13/count-json-array-elements-with-jq/
nbUserStories=$(jq '. | length' "$resultFile")
echo "'$nbUserStories' user stories"


#  List the user stories IDs :
# jq '.[29].ref' result.json
# for i in {0..29}; do jq ".[$i].ref" result.json; done


#  [] is an iterator, no need of 'for' loop \o/
#  https://stedolan.github.io/jq/manual/#Basicfilters
#jq '.[].ref' result.json
#jq ".[].subject" result.json


#  to get both :
#  source : https://stackoverflow.com/questions/28164849/using-jq-to-parse-and-display-multiple-fields-in-a-json-serially#answer-31418194
#jq -r '.[] | "\(.ref) ==> \(.subject)"' result.json

#  get tags :
#  thanks to : https://stackoverflow.com/questions/41712363/basic-jq-usage-how-to-get-nested-value
#jq -r '.[].tags | .[] | .[0]' result.json

echo 'Tags :'
jq -r '.[].tags | .[] | .[]' "$resultFile" | sed '/^null$/d' | sort | uniq -c | sort -nr

Flags :

Flag Usage
-r --raw-output output string without quotes

journalctl

Usage :

Query the systemd journal. It is possible to filter results with :
Wrapping long lines is not very intuitive.

Permissions

  • All users are granted access to their private per-user journals.
  • Only root and users who are members of a few special groups are granted access to :
    • the system journal
    • journals of other users
  • Members of the groups systemd-journal, adm, and wheel can read all journal files. Note that the two latter groups traditionally have additional privileges specified by the distribution. Members of the wheel group can often perform administrative tasks.

To read all journals as a non-root user, you can either :

FIELD=VALUE CLI arguments

List of available fields : man 7 systemd.journal-fields
Field Usage example Comments
MESSAGE=message The human-readable message string for this entry. journalctl MESSAGE='Failed unmounting /var.' Looks like message has to be the exact message to be matched. No pattern or partial message seems to match.
PRIORITY severity or verbosity level, based on syslog priority concept journalctl PRIORITY=3 syslog's severity levels
_COMM
_EXE
_CMDLINE
The name, the executable path, and the command line of the process the journal entry originates from. journalctl _COMM=pulseaudio
_SYSTEMD_UNIT systemd unit name (if any)
  • journalctl _SYSTEMD_UNIT=mysql
  • journalctl -u mysql
Looks pretty similar to -u / --unit, but gives slightly different results
_TRANSPORT=value How the entry was received by the journal service. value is one of :
  • audit : for those read from the kernel audit subsystem
  • driver : for internally generated messages
  • journal : for those received via the native journal protocol
  • kernel : for those read from the kernel
  • stdout : for those read from a service's standard output or error output
  • syslog : for those received via the local syslog socket with the syslog protocol
Advanced filtering
logical AND
several FIELD=VALUE arguments on the same command are combined into a logical AND, displaying logs entries matching ALL criteria :
journalctl _SYSTEMD_UNIT=mysql.service PRIORITY=3
logical OR
A + between terms will combine the one before and the one after into a logical OR
journalctl _SYSTEMD_UNIT=mysql.service + PRIORITY=3
alternatives
If two matches apply to the same field, then they are automatically matched as alternatives, i.e. the resulting output will show entries matching any of the specified matches for the same field :
journalctl _SYSTEMD_UNIT=mysql.service PRIORITY=2 PRIORITY=3

Flags :

Flag Usage
--disk-usage Shows the current disk usage of all journal files : the sum of the disk usage of all archived and active journal files.
-f --follow Show only the most recent journal entries, and continuously print new entries as they are appended to the journal (i.e. like tail -f)
--no-pager Do not pipe output into a pager
a pager is a utility used to display text, scroll up/down, scroll 1line/1page, search, ..., such as less
-o outputOption
--output=outputOption
Controls the formatting of the journal entries that are shown. Among available values : verbose
-r --reverse reverse output : newest entries first
-S 'YYYY-MM-DD hh:mm:ss'
--since='YYYY-MM-DD hh:mm:ss'
Show entries on/newer than the specified timestamp. Missing time will be interpreted as 00:00:00, missing seconds will be interpreted as :00
-u unit --unit=unit Show messages for the specified systemd unit unit (list available units)
-U 'YYYY-MM-DD hh:mm:ss'
--until='YYYY-MM-DD hh:mm:ss'
Show entries on/older than the specified timestamp. Missing time will be interpreted as 00:00:00, missing seconds will be interpreted as :00
--vacuum-files=finalNumberOfLogfiles Remove archived journal files until there remain only finalNumberOfLogfiles logfiles
this will leave the active journal files untouched
--vacuum-size=finalLogsSize
  • Remove archived journal files until the disk space they use falls below finalLogsSize
  • finalLogsSize can be specified with a suffix such as K, M, G or T
this will leave the active journal files untouched
--vacuum-* functions appeared with systemd v2.18 (source). On prior versions, this will cause errors such as :
journalctl: unrecognized option '--vacuum-size=1G' will appear
In such situation, to clear logs, you'll have to run, with root privileges :
  1. find /var/log/journal -name '*journal' -a -ctime +180 -delete
  2. journalctl --disk-usage
--vacuum-time=timespan
  • Remove archived journal files until they contain no data older than timespan
  • timespan can be specified with a suffix such as s, m, h, days, weeks, months or years

Example :

Some commands :

How to wrap long lines ?

Lines output by journalctl are often longer than the display width, and the interesting part of messages is generally at the end of the line (i.e. not visible )

Just use the key to view long lines

man journalctl is not very helpful when it comes to wrapping long lines (search wrap and pager for details). To view long lines, you can also :
journalctl -u networking --no-pager
But this will throw everything to the screen, making it impossible to read. So you'll end up with :
journalctl -u networking --no-pager | less
i.e. asking no pager, please in the first place, then piping everything into ... a pager .

This construct could prove useful in scripts, however it is not necessary when trying to grep something which is hidden by the default pager :

journalctl -u networking --output=verbose | grep foo
will actually search and find foo, even though foo is beyond the right screen border when running :
journalctl -u networking --output=verbose

journalctl errors

Failed to determine timestamp: Cannot assign requested address
I observed this when running commands like : whereas this returns no log entry :
journalctl -u haproxy
Looks like "over-filtering" with -u + --until causes it.

join

Usage :

Join 2 files on their matching records, as does the SQL operator.

Example :

The Dalton's :

Considering 2 files :

  • names.txt
    Joe Dalton
    Jack Dalton
    William Dalton
    Averell Dalton
  • ages.txt
    Joe 23
    Jack 22
    William 21
    Averell 20

join names.txt ages.txt returns :

Joe Dalton 23
Jack Dalton 22
William Dalton 21
Averell Dalton 20