← Blog
A lavender tape dispenser with a cream roll and orange cutting edge on a pale background.
Published on
2026-11-16

JSONL vs JSON: choose a format for data pipelines

Choose JSON when the receiving system expects one complete document, such as a configuration object or an API response. Choose JSONL when it expects a sequence of independently readable values, such as records in a processing job. The difference between JSONL and JSON is how the data is separated, not whether it can contain nested objects.

This guide compares those choices and provides a Python reader and writer with explicit input limits and error behavior. The example uses six synthetic values and was checked with 16 local tests on Windows CPython 3.12.14. It does not measure throughput or test a cloud integration.

JSONL vs JSON: the practical differences

DecisionJSON documentJSONL file
Unit to consumeOne complete value, often an object or arrayA separate value on each line
Typical receiving contractAn API payload, configuration, or complete exportA record-oriented import, log reader, or processing job
Reader designParse the document; use a streaming parser if neededRead and process successive lines
Invalid inputDecide how to reject or recover the documentDecide whether a bad record stops the job or is quarantined
Schema and deliveryDefine separatelyDefine separately

Start with the consumer. Renaming data.json to data.jsonl does not convert it. An endpoint expecting an array may reject a sequence of otherwise valid objects. Likewise, a record importer may treat a JSON array on one line as one value rather than several records.

JSONL is convenient for a simple iterator, but JSON does not inherently require loading an entire dataset into memory: specialized parsers can process large JSON documents incrementally. Neither extension establishes a speed advantage for your workload. Measure parsing, transformations, output, and transfers together.

Understand the document and line boundaries

A JSON document contains one value; an array can collect several records. The JSON standard permits objects, arrays, strings, numbers, booleans, and null. Here are two values inside one array:

[
  {"id":"001","note":"first\nsecond"},
  null
]

The equivalent sequence in JSONL has no enclosing array or separating comma:

{"id":"001","note":"first\nsecond"}
null

The JSON Lines rules require UTF-8 without a byte-order mark and one valid JSON value per line. Blank lines are invalid; null is valid. LF and CRLF endings work. A final newline is recommended, although not required.

A string's escaped \n stays within its record; a literal line break splits the physical lines. Pretty-printing an object across multiple lines therefore breaks this layout. A pipeline can require objects with particular fields, but that is its own schema rule. JSONL itself permits scalars and arrays.

Try a Python reader and writer

Save the following as jsonl_records.py. It uses only Python's standard library. The demo creates temporary files, compares a JSON array with a JSONL sequence, and removes its files when it finishes. The six values deliberately include Unicode, an escaped newline, null, a nested array, a number, and a string.

"""Local JSON/JSONL comparison; standard library, no network access."""
import json
import math
from pathlib import Path
from tempfile import TemporaryDirectory


def check_limit(limit):
    if isinstance(limit, bool) or not isinstance(limit, int) or limit < 1:
        raise ValueError("max_line_bytes must be a positive integer")


def unique_object(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError("duplicate object key")
        result[key] = value
    return result


def reject_constant(text):
    raise ValueError("non-finite constant")


def finite_float(text):
    value = float(text)
    if not math.isfinite(value):
        raise ValueError("outside finite float range")
    return value


def read_jsonl(stream, max_line_bytes=65536):
    """Read a binary stream. The byte limit includes LF or CRLF, if present."""
    check_limit(max_line_bytes)
    line_number = 0
    while True:
        raw = stream.readline(max_line_bytes + 1)
        if not raw:
            return
        line_number += 1
        if len(raw) > max_line_bytes:
            raise ValueError(f"line {line_number}: byte limit exceeded")
        try:
            if raw.startswith(b"\xef\xbb\xbf"):
                raise ValueError("BOM not allowed")
            text = raw.decode("utf-8")
            if not text.strip():
                raise ValueError("blank line")
            value = json.loads(text, object_pairs_hook=unique_object,
                               parse_constant=reject_constant,
                               parse_float=finite_float)
        except (ValueError, RecursionError):
            raise ValueError(
                f"line {line_number}: invalid JSONL or reader policy violation"
            ) from None
        yield value


def write_jsonl(path, records, max_line_bytes=65536):
    """Create a new file; failures can leave a partial output file."""
    check_limit(max_line_bytes)
    with open(path, "xb") as target:
        for value in records:
            text = json.dumps(value, ensure_ascii=False, allow_nan=False,
                              separators=(",", ":"))
            raw = (text + "\n").encode("utf-8")
            if len(raw) > max_line_bytes:
                raise ValueError("output line byte limit exceeded")
            target.write(raw)


def main():
    records = [
        {"id": "001", "note": "first\nsecond", "city": "Zürich"},
        None,
        {"active": True, "count": 0},
        [1, 2],
        3.5,
        "ready",
    ]
    with TemporaryDirectory() as directory:
        root = Path(directory)
        whole = root / "records.json"
        lines = root / "records.jsonl"
        whole.write_text(json.dumps(records, ensure_ascii=False, allow_nan=False),
                         encoding="utf-8")
        write_jsonl(lines, records)
        with whole.open(encoding="utf-8") as source:
            from_json = json.load(source)
        with lines.open("rb") as source:
            from_jsonl = list(read_jsonl(source))  # Six-value test fixture only.
        assert from_json == from_jsonl == records
        physical_lines = lines.read_bytes().count(b"\n")
        assert physical_lines == len(records)
        print("6 values match in JSON and JSONL")
        print(f"JSONL physical lines: {physical_lines}")
        print("Escaped newline preserved:", from_jsonl[0]["note"] == "first\nsecond")


if __name__ == "__main__":
    main()

Run it with Python 3.12:

python jsonl_records.py

Expected output:

6 values match in JSON and JSONL
JSONL physical lines: 6
Escaped newline preserved: True

The list(...) call is only for comparing this six-value fixture. In a processing job, iterate over read_jsonl(source) and handle each value without collecting the entire file. The reader requires a binary stream. Its 65,536-byte default is an example policy, not a JSONL limit; change it to match the consumer.

Know what the example checks

The 16 tests cover round trips, line endings, missing final newlines, blank and malformed lines, invalid UTF-8, byte-order marks, numeric policies, duplicate keys, byte boundaries, lazy reading, and writer failures. An oversized-line test checks that the reader requests and receives only the limit plus one byte from an in-memory stream before rejecting it. This is a correctness check, not a memory benchmark.

The byte limit includes an existing LF or CRLF. Decoded strings and Python objects consume additional memory, and deeply nested input can hit parser limits. The writer encodes one value before checking its size, so its limit is not a preallocation ceiling. Use JSON-compatible values with string dictionary keys; this example does not validate a business schema or preserve every possible Python type.

The Python JSON documentation describes parser hooks used here. This reader rejects non-finite constants and duplicate object keys. The latter is a deliberate interoperability rule, stricter than JSON syntax. It also rejects a number such as 1e400 because it exceeds its finite-float policy, even though that number is syntactically valid JSON.

The fixture's values survive the round trip; that does not establish arbitrary decimal precision. For financial or scientific values, agree on numeric range and representation with the receiving application before choosing parser settings.

Choose an error and recovery policy

The reader stops at the first bad line and reports its number without copying the record into the error message. That keeps the failure visible and avoids casually putting source data into logs. If a job instead quarantines records, record the rejection count and reason, restrict access to rejected data, and decide when too many errors should fail the run.

The writer creates a new file and refuses to overwrite an existing one. It is not transactional: if a later value fails, earlier lines remain in the output. Keep that partial file out of the consumer's input set. A production job needs its own completion signal and a policy for cleaning up or retrying failed runs.

For restartable jobs, tie a checkpoint to a specific source version and completed side effects. A line number alone does not prove an output has been committed or protect against reading a changed file. Exactly-once processing requires coordination beyond choosing JSONL.

For a concrete log-oriented use, the LLM monitoring guide includes an offline audit-file tool. A different consumer, such as the dataset loader in the LoRA fine-tuning guide, has its own field requirements. Reusing an extension does not make those records interchangeable.

Keep local file writes separate from object uploads

A local file workflow and an object-storage workflow have different operations. Do not assume a remote object supports the same append behavior as a local file. A straightforward batch design writes a fresh local file, validates it, closes it, and uploads the completed result under a new object key.

Give consumers an explicit way to identify completed files, and record the schema version, record count, and checksum where useful. Test upload failure, retry, and publication of the completion signal separately. The example above stops at local files and makes no claim about those remote operations.

If combining files, check the boundary between them: a missing final newline can join two records. Also check that the files follow the same application schema and numeric rules. Successfully parsing every line does not establish that their meanings agree.

Run record processing with Hivenet

Compute with Hivenet provides CPU infrastructure for customer-operated processing jobs. Hivenet Object Storage provides S3-compatible storage for datasets and application files. Your reader, validation rules, scheduling, and retry behavior remain part of the workload you run.

Before choosing capacity, collect a representative file, the largest expected record, daily volume, processing deadline, and the receiving system's requirements. Use those inputs to test a CPU job and its storage connection. A six-record local example can check behavior; it cannot size a production instance.

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.