Vector Tile Reference

Streaming GeoParquet to Tippecanoe stdin as NDJSON

End the tippecanoe command with a lone - and feed it one GeoJSON feature per line on standard input, and no multi-gigabyte .geojson file is ever written — decode and encode overlap, and peak memory is bounded by one batch instead of the whole dataset.

When to use this

Stream over stdin whenever writing an intermediate file is the wrong trade. GeoJSON stores coordinates as ASCII, so the temporary file is routinely 4–10× the size of the source Parquet; on a disk-constrained CI runner or a container with an ephemeral volume, that intermediate is often the thing that fails the build. Piping removes it entirely.

Concretely, prefer the stdin pipe when:

  • The dataset is large enough that a temporary .geojson would strain disk or exceed a runner’s scratch quota.
  • You are in CI, where fewer intermediate artifacts means fewer cleanup steps and less flaky-disk exposure.
  • You want decode and encode to run concurrently — Tippecanoe starts building tiles while the reader is still on the early row groups.

This page is about the stdin interface itself — the line protocol, the trailing -, flushing, and parallelism. For a full, batteries-included Python converter that wraps this interface, see converting large GeoParquet files to vector tiles.

Specification detail

Tippecanoe reads from standard input when the input path is a single -. What it expects on that stream is newline-delimited GeoJSON — one complete Feature object per line, no enclosing array. This format goes by several names; they are the same thing:

Name What it is Tippecanoe reads it
NDJSON Newline-delimited JSON, one object per line yes, on -
GeoJSONSeq GeoJSON text sequence (one Feature per line) yes
GeoJSON FeatureCollection One object with a features: [...] array yes, but as a single blob

The distinction is load-bearing for --read-parallel. That flag tells Tippecanoe to split the input at newline boundaries and hand slices to worker threads — which only works when each line is an independent feature. A single FeatureCollection has no per-feature newlines to split on, so --read-parallel degrades to a single reader. Emit GeoJSONSeq, not a collection.

Token Effect
- (trailing arg) Read features from stdin instead of a file
--read-parallel Split newline-delimited stdin across threads
-L{"layer":...,"file":"-"} Named-layer form when combining multiple stdin sources

Each emitted line is a compact object: {"type":"Feature","geometry":{...},"properties":{...}}\n. Use compact separators — (",", ":") in Python’s json.dumps — to keep the pipe payload small.

Backpressure is handled for you by the operating system. The pipe between producer and Tippecanoe has a fixed kernel buffer (64 KB on Linux by default); when Tippecanoe reads slower than the producer writes, the producer’s write() simply blocks until space frees up. That is the mechanism that bounds memory — you do not need your own queue or rate limiter, only to avoid buffering the whole dataset in userspace before writing. To stream more than one named layer from separate sources in a single Tippecanoe run, use the -L layer form, where each entry names a layer and a file; "file":"-" in one entry designates the stdin stream.

Production command

Three interchangeable producers feed the same tippecanoe … - consumer. Pick by what is already in your stack.

A — pyarrow generator, streaming batches to stdout. The key details are the trailing -, writing one line per feature, and flushing each batch so Tippecanoe consumes incrementally instead of waiting on a full buffer:

python
import sys, json
import pyarrow.parquet as pq
from shapely import wkb
from shapely.geometry import mapping

SOURCE = "data/roads_europe.geoparquet"
COLS = ["id", "class", "name", "geometry"]   # projection: only what tiles need

def stream_ndjson(path: str, out=sys.stdout) -> None:
    pf = pq.ParquetFile(path)
    for batch in pf.iter_batches(batch_size=100_000, columns=COLS):
        geoms = batch.column("geometry").to_pylist()
        attrs = {c: batch.column(c).to_pylist() for c in COLS if c != "geometry"}
        lines = []
        for i, raw in enumerate(geoms):
            if raw is None:
                continue
            feature = {
                "type": "Feature",
                "geometry": mapping(wkb.loads(raw)),
                "properties": {c: attrs[c][i] for c in attrs},
            }
            lines.append(json.dumps(feature, separators=(",", ":")))
        out.write("\n".join(lines) + "\n")
        out.flush()                # hand this batch to the reader now

if __name__ == "__main__":
    stream_ndjson(SOURCE)
bash
python stream_ndjson.py \
  | tippecanoe \
      --output=dist/roads.pmtiles \
      --layer=roads \
      --minimum-zoom=4 \
      --maximum-zoom=14 \
      --drop-densest-as-needed \
      --read-parallel \
      --force \
      -

B — DuckDB COPY … TO '/dev/stdout'. No Python at all; DuckDB’s spatial extension serializes GeoJSON and the shell pipes it in. ARRAY false is what makes it emit one object per line rather than a JSON array:

bash
duckdb -c "
INSTALL spatial; LOAD spatial;
COPY (
  SELECT id, class, name, geometry
  FROM read_parquet('data/roads_europe.geoparquet')
) TO '/dev/stdout' (FORMAT JSON, ARRAY false);
" \
  | tippecanoe --output=dist/roads.pmtiles --layer=roads \
      --minimum-zoom=4 --maximum-zoom=14 --read-parallel --force -

C — ogr2ogr to /vsistdout/. The GDAL path, useful when GeoParquet is one of several formats you handle uniformly. The GeoJSONSeq driver writes newline-delimited features:

bash
ogr2ogr -f GeoJSONSeq /vsistdout/ data/roads_europe.geoparquet \
  | tippecanoe --output=dist/roads.pmtiles --layer=roads \
      --minimum-zoom=4 --maximum-zoom=14 --read-parallel --force -

Interaction effects

With column projection and bbox filtering. What you stream is entirely determined by the upstream read. Applying column projection pushdown shrinks each line to the properties the style needs, and bbox spatial filtering reduces how many lines are emitted at all. Both act before serialization, so the pipe carries the minimum payload — the COLS list and any WHERE in the producer are where those cuts live.

With --read-parallel and production flags. --read-parallel only helps if the producer emits true GeoJSONSeq and keeps the pipe fed; a slow serializer starves the workers regardless. Pair it with the essential production flags — --drop-densest-as-needed, --force, --quiet — which behave identically on stdin input as on a file. Spatial ordering upstream (the H3/S2 sort) additionally lets the parallel readers process spatially contiguous slices, improving tile-assembly locality.

With subprocess orchestration. In Python you can skip the shell and drive Tippecanoe with subprocess.Popen(cmd, stdin=PIPE), writing to proc.stdin directly. The mechanics — spawning the process, streaming batches, closing stdin, and handling a non-zero exit — are covered end to end in the large-file conversion walkthrough.

Performance impact

Streaming removes the intermediate write and the subsequent full re-read — the single largest cost of the file-based path. It also overlaps decode and encode: while Python or DuckDB reads and serializes, Tippecanoe is already encoding earlier features, so the two stages run concurrently rather than back to back.

Approach Extra disk written Peak RAM Relative wall-clock
GeoParquet → .geojson file → Tippecanoe 4–10× source size one batch (writer) 2–4× (write + re-read)
GeoParquet → stdin NDJSON → Tippecanoe none one batch 1× (baseline)

Peak memory is bounded by batch_size × average serialized feature size, independent of total file size — the property that lets a multi-gigabyte file convert on a 4 GB worker. Throughput is gated by whichever stage is slower; for attribute-heavy polygons the JSON serializer often is, which is why compact separators and column projection matter.

Which producer to reach for follows from where the bottleneck sits. DuckDB’s serializer is written in C++ and typically out-runs a pure-Python loop, so producer B is fastest when no per-feature Python logic is required. The pyarrow generator (producer A) is the right choice when you need custom transforms in the loop — pre-simplifying geometries, deriving a property, or dropping features on a condition — because those hooks live naturally in Python. ogr2ogr (producer C) trades some speed for format uniformity, and is convenient when the same pipeline must also accept FlatGeobuf or GeoPackage inputs through one code path.

Common mistakes

Emitting a FeatureCollection array instead of NDJSON. If the producer writes {"type":"FeatureCollection","features":[ ... ]}, Tippecanoe still reads it, but --read-parallel finds no per-feature newlines and falls back to a single reader — the build runs but does not parallelise. Symptom: adding --read-parallel changes nothing. Fix: emit one Feature per line (GeoJSONSeq); in DuckDB set ARRAY false, in GDAL use -f GeoJSONSeq.

Buffering all of stdout, or never flushing. Collecting every line into a list and writing once at the end reintroduces the whole-dataset memory cost the pipe was meant to avoid, and Tippecanoe sits idle until the end. Symptom: RAM climbs with dataset size and the encoder starts late. Fix: write and flush() per batch, as in producer A, so the reader consumes lines as they are produced.

Forgetting the trailing -. Without it, tippecanoe --output=out.pmtiles --read-parallel has no input source and exits with tippecanoe: no input files, while your producer dies with BrokenPipeError: [Errno 32] Broken pipe because nothing is reading stdin. Symptom: an immediate broken-pipe traceback on the producer side. Fix: make - the final argument to tippecanoe. If features are otherwise valid but the pipe breaks mid-stream, the cause is usually an invalid geometry crashing the encoder — guard the WKB decode and skip null or corrupt geometries.

Parent: GeoParquet Input Processing