How to Use jq Style Queries in Python on JSON With Examples

jq is excellent for a one-off filter and gets hard to maintain when the query grows to conditionals, formatting and error handling. Python reads the same data as native types, and the logic becomes ordinary, testable code. Here is a translation table for the common queries, building on jq for API responses. The examples assume Python 3.13 and the standard library only.

Load and navigate

import json

with open("response.json") as handle:
    data = json.load(handle)

print(data["meta"]["total"])
print(data.get("meta", {}).get("next", "none"))
for item in data["items"][:2]:
    print(item["id"], item["name"])

42
none
101 alpha
102 beta

Chained .get() calls with a default of {} handle missing keys without exceptions, like jq's // operator. Use direct indexing, which raises, when a missing field means the data is bad and you want to know. For command line use, json.load(sys.stdin) reads a pipe: curl -s URL | python3 script.py.

Filter, map and sort

ok = [i for i in data["items"] if i["status"] == "ok"]
names = [i["name"] for i in ok]
by_size = sorted(ok, key=lambda i: i["size"], reverse=True)[:3]
total = sum(i["size"] for i in ok)
print(names, total)

['alpha', 'gamma'] 3072

These correspond to jq '[.items[] | select(.status=="ok")]', map(.name), sort_by(.size) | reverse and add. Comprehensions are readable enough to review and fast. Handle optional keys inside the condition with i.get("status") == "ok" when the data is uneven.

Group and count

from collections import Counter, defaultdict

counts = Counter(i["status"] for i in data["items"])
print(counts.most_common())

groups = defaultdict(list)
for i in data["items"]:
    groups[i["owner"]].append(i["name"])
print(dict(groups))

[('ok', 2), ('failed', 1)]
{'ann': ['alpha', 'beta'], 'bob': ['gamma']}

Grouping is where jq expressions get cryptic and Python stays plain. Counter.most_common() gives a sorted tally in one call, similar to sort | uniq -c from counting top values.

Deep paths and JSON Lines

import json
import sys

def dig(obj, path, default=None):
    for part in path.split("."):
        if isinstance(obj, dict):
            obj = obj.get(part, default)
        elif isinstance(obj, list) and part.isdigit():
            obj = obj[int(part)] if int(part) < len(obj) else default
        else:
            return default
    return obj

for line in sys.stdin:
    record = json.loads(line)
    print(dig(record, "user.address.city", "unknown"))

A tiny helper handles dotted paths. For bigger needs, the jmespath package offers a query language close to jq's, as in jmespath.search("items[?status=='ok'].name", data). JSON Lines files, one object per line, stream with constant memory when you iterate the file and parse each line, rather than loading everything at once.

Choosing between the two for a given job

The decision is rarely about capability, since both can express every query on this page. It is about who will maintain the line. A one-line filter in a shell pipeline that nobody will touch again is cheaper in jq. A transformation with a conditional, a lookup table and a formatted report is cheaper in Python, because a colleague can read it, step through it in a debugger and write a test for it. The break-even is roughly the point where a jq program needs a comment to explain itself.

Another factor is where the data goes next. If the output feeds a database insert, an HTTP call or a chart, the Python version already has the values in native types and keeps going without a serialisation round trip. If the output is text for another command in a pipeline, jq produces it directly and avoids starting an interpreter, which matters when the filter runs thousands of times in a loop.

Errors, types and bad input

Real API responses are untidy. A field is sometimes a string and sometimes null, a list is sometimes a single object, and a numeric identifier arrives as text on one endpoint and as a number on another. The standard library gives you tools for each case, but you have to choose them on purpose.

Wrap the parse in a handler for json.JSONDecodeError and report the position it gives you, because a truncated download and an HTML error page look identical to a person and very different to the code. Check isinstance before indexing when a shape can vary, and prefer failing loudly on the field the script cannot do without. Silent defaults turn a schema change into wrong numbers instead of an exception, which is the worse outcome for a report.

Numbers deserve special care. Python parses a JSON integer as an arbitrary-precision int, so large identifiers survive intact, while a decimal becomes a float with the usual rounding. For money, pass parse_float=Decimal to json.load so amounts keep their exact digits. jq 1.7 preserves large integers when it only passes them through, but converts them to doubles as soon as it does arithmetic on them, so the same identifier can survive one tool and not the other.

Writing the result back out

Output is where scripts become useful to the next step. json.dump(result, sys.stdout, indent=2, sort_keys=True) produces stable, diff-friendly text, and a sorted key order means two runs on the same data compare equal in version control. Pass ensure_ascii=False when the consumer expects UTF-8 and the data contains non-ASCII names, otherwise every accented letter becomes an escape sequence.

For tabular output, the csv module handles quoting that hand-built strings get wrong, such as a field that contains a comma or a newline. Write to a temporary file and rename it over the target when another process may read the file while the script is still running, so a reader never sees half a document. These are the same habits that make shell scripts safe, and they carry over unchanged.

Performance and memory with large documents

For files under a few megabytes the difference between tools does not matter, and reading everything with json.load is the right choice. The picture changes when a response is hundreds of megabytes. json.load builds the full object graph in memory, and a Python dict of small values uses several times the memory of the text it came from, so a 500 MB file can exhaust a small container.

Three remedies are available, in order of effort. If the data is a JSON Lines file, iterate and parse one line at a time and keep running totals, which uses memory proportional to the largest record. If it is a single huge array, ijson streams items as they are parsed. If the job only needs a few fields, filtering with jq first and passing the small result to Python gets the speed of a compiled parser and the clarity of Python for the rest.

Measure before choosing. Time the real file with the time command and watch peak memory with /usr/bin/time -v, because the answer depends on the shape of the data more than on its size. A flat list of ten million small objects behaves differently from a few thousand deeply nested ones, and an optimisation tuned for the wrong shape adds code without saving anything.

Where this does not apply

For a one-time look at a response in a terminal, jq is faster to type and ships on most systems; see jq compared with gron and jless. Python's json parses numbers as floats or ints and can lose precision on 64-bit identifiers if you coerce them. Very large files need a streaming parser such as ijson. See the json module documentation and the JMESPath tutorial.