Search notes:

jq

jq is like sed for JSON data.

Installation on apt based systems

sudo apt install jq -y

Usage

jq [options…] filter [files…]
filter is a program, written in the jq language, that specifies how the input data (JSON format) is transformed for the output.
By default, jq will pretty-print its output (but see -c command line option).
The following jq filter (.num) returns the value of the num element of the JSON object passed to jq (i. e. 42): 
echo '{
  "num":  42,
  "txt": "Hello world"
}' | jq .num

Filters

A filter takes input and produces output. Some filters even produce multiple outputs.
Literals (Such as "Hello world" or 42) are filters: these disregard their input and produce their own value as output.
The output of a filter can be piped into another filter or collected into an array.
. Identity (output is equal to input) (for example echo 11 | jq '(. + 10) * 2' produces 42.
.. Recursive descent (similar to XPath // operator)
.keyName, .["keyName"] Value JSON object's element whose name is keyName. If followed by question mark (.keyName?, .["keyName"]?), then value evaluates to null if keyName does not exist (rather than outputting an error)
.[n] nth element of array. n can be negative to count from the other end (.[-1] is last element)
.[n:m] Slice. Can be applied on array or string.
.[] All elements (.[]? prevents errors)
… , … Comma operator: input is fed to the filter on the left and on the right, output values are concatenated.
… | … Pipe operator (similar to the pipe of shells)
[ elem1, elem2, … ] Array construction
{ key: val … } Object construction

Operators and functions

+, -
*
/
%
length Returns the number of Unicode codepoints of a string, the number of elements in an array or the number of key/value pairs in an object. length(null) returns 0.
utf8bytelength
keys, keys_unsorted
has(key)
in
map(x), map_values(x)
path(expr)
del(path_expression) del removes a key and its associated value from a JSON object.
getpath(PATHS), setpath(PATHS; VALUE), delpaths(PATHS)
to_entries, to_entries, from_entries, with_entries
select(boolean_expression)
arrays, objects, iterables, booleans, numbers, normals, finites, strings, nulls, values, scalars
empty
error(message)
halt, halt_error, halt_error(exit_code)
$__loc__
paths, paths(node_filter), leaf_paths
add
any, any(condition), any(generator; condition)
all, all(condition), all(generator; condition)
flatten, flatten(depth)
range(upto), range(from;upto) range(from;upto;by)
floor
sqrt
tonumber, tostring
type
infinite, nan, isinfinite, isnan, isfinite, isnormal
sort, sort_by(path_expression)
group_by(path_expression)
min, max, min_by(path_exp), max_by(path_exp)
unique, unique_by(path_exp)
reverse
contains(element)
indices(s)
index(s), rindex(s)
inside
startswith(str), endswith(str)
combinations, combinations(n)
ltrimstr(str), rtrimstr(str)
explode
implode
split(str)
join(str)
ascii_downcase, ascii_upcase
while(cond; update)
until(cond; next)
recurse(f), recurse, recurse(f; condition), recurse_down
walk(f)
transpose
bsearch(x)
tojson, fromjson
builtins Returns a list of all built in functions in the format name/arity
repeat

Date functions

fromdateiso8601 Parses dates in ISO 8601 format, returns number of seconds since the Unix epoch
todateiso8601 Reverse of fromdateiso8601
fromdate
todate An alias for todateiso8601
now Current time in seconds since Unix epoch
Pipe empty JSON object to jq (because jq expects a JSON object) and evaluate now: 
$ echo '{}' | jq 'now'
1679756466.497302
Pipe the evaluated number of seconds to todateiso8601 so that we get an ISO 8601 formatted string: 
$ echo '{}' | jq 'now | todateiso8601'
1679756466.497302
"2023-03-25T15:01:06Z"
Get the number of seconds from an ISO 8601 formatted string: 
$ echo '"2022-02-18T14:57:13Z"' | jq 'fromdateiso8601'
1645196233
There are also interfaces to the C-library time functions:
  • strptime
  • strftime
  • strflocaltime
  • mktime
  • gmtime
  • localtime

SQL style operators

INDEX(stream; index_expression)
JOIN($idx; stream; idx_expr; join_expr)
JOIN($idx; stream; idx_expr)
JOIN($idx; idx_expr:
IN(s)
IN(source; s)

Variables

Variables start with a dollar sign ($thisIsAVariable)

Modules

jq allows to create libraries/modules in .jq files.

@ syntax

The @ syntax formats and escapes strings.

Command line options

--version
--seq Use the application/json-seq MIME type scheme for separating JSON texts in input and output: print an ASCII RS (record separator) character before each value on output and an ASCII LF (line feed) after every output.
--stream Outputarrays of path and leaf values (scalars and empty arrays or empty objects), useful for processing large inptus in conjunction with filtering ("a" becomes [[],"a"] ; [[],"a",["b"]] becomes [[0],[]], [[1],"a"] and [[1,0],"b"])
-s --slurp Read entire input string into a large array, then run filter once (Instead of running the filter for each JSON object in the input).
-R --raw-input Input is not treated as JSON; each line is passed to the filter as a string. Combine with --slurp to process entire input as one string.
-n --null-input Run filter once with null as input (for example to construct JSON data from scratch)
-c --compact-output Do not pretty-print output.
--tab Use tab rather than two spaces for indentation
--indent n Use the given number of spaces for indentation.
-C --color-output See also JQ_COLORS environment variable
-M --monochrome-output
-a --ascii-output
--unbuffered Flush the output after each JSON object is printed
-S --sort-keys Output object fields in sorted order
-r --raw-output Strings won't be quoted
-j --join-output Like -r but jq won`t print a newline after an output.
-f filename --from-file filename Read filter from filename (like awk's -f option). Apparently, such files allow # for comments
-Ldirectory Prepend directory to the search list for modules. If this option is used then no builtin search list is used.
-e -exit-status
--arg name value Pass a value to the jq program as a predefined variable ($name then has the value value)
--argjson name JSON-text
--slurpfile variable-name filename Assigns to content of a JSON document to a variable.
--rawfile variable-name filename Assign content of filename to variable-name.
--argfile variable-name filename Use --slurpfile instead.
--args Remaining arguments are positional string arguments. These are available to the jq program as $ARGS.positional[].
--jsonargs Remaining arguments are positional JSON text arguments. These are available to the jq program as $ARGS.positional[].
--run-tests [filename]

TODO

$ json='[ {"id": 7, "name": "ABC"}, {"id": 22, "name": "XYZ"}]'
$ echo "$json" | jq -r '.[] | select(.name == "XYZ") | "id = \(.id), name = \(.name)"'
id = 22, name = XYZ

$ read -r id name < <(echo "$json" | jq -r '.[] | select(.name == "XYZ") | "\(.id) \(.name)"' )
$ echo $id
22
$ echo $name
XYZ

Get value of a JSON object whose key contains of spaces or special characters

Wrong: 
$ echo '{"abc def": 42, "ghi/jkl": 99}' | jq ."abc def"
jq: error: syntax error, unexpected def, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
Better (note the single quotes): 
$ echo '{"abc def": 42, "ghi/jkl": 99}' | jq '."abc def"'
42

See also

yq has the same idea, but is for YAML.
fq is inspired by jq, but works with binary data formats.
If ~/.jq is a file, it is sourced into the main program
JMESPath is another query language for JSON.
Tools for data analysis.

Links

https://stedolan.github.io/jq/

Index