Column Projection Pushdown for GeoParquet

Name the geometry column plus only the attribute fields your style actually renders, and the Parquet reader will skip the compressed column chunks for everything else — never decompressing them, never allocating memory for them, and never carrying them into the stream that feeds Tippecanoe.

When to use this

Reach for column projection whenever the source table is wide relative to what the tiles consume. Building footprints, parcel exports, and Overture-style theme dumps routinely carry 30 to 120 columns — source provenance, edit timestamps, multilingual name variants, raw tag maps — while the finished map style filters and paints on four or five of them. Every unnamed column is pure overhead: it is read from disk, decompressed, decoded to Arrow, and then discarded.

Projection pays off most when:

• The table has more than a dozen columns and the style references a small subset.
• One or more columns are heavy: WKT duplicates of the geometry, JSON blobs, long free-text descriptions, or wide struct/list tag maps.
• You run inside a memory-limited CI worker where peak RAM, not wall-clock time, is the binding constraint.

If the table is already narrow (geometry plus three or four scalar columns), projection changes little — the reader is close to reading everything anyway. It is the wide-table case where the wins are dramatic.

Specification detail

Parquet is columnar: each row group stores every column as an independent, separately compressed column chunk, and the file footer records the byte offset and length of each chunk. A reader that is told which columns it needs seeks directly to those chunks and never touches the bytes of the others. This is projection pushdown — the projection is “pushed down” into the reader instead of applied after a full materialisation. The GeoParquet layout inherits this directly; the geometry column is just another column chunk, typically WKB-encoded.

Two rules govern correctness. First, the geometry column must appear in the list — the reader has no special knowledge that a map pipeline needs it. Its name comes from the primary_column field of the geo metadata (usually geometry, but geom, wkb_geometry, or shape show up in QGIS and PostGIS exports). Second, projection must be applied at the reader, not after: read_table() followed by df[["a", "b"]] has already paid the full read cost before you slice.

Projection also reaches into nested columns. Overture and similar datasets store a names or sources field as a Parquet struct; requesting the top-level column reads the whole struct, but pyarrow accepts dotted leaf paths like columns=["names.primary", "geometry"] so only the one sub-field’s chunk is decoded. The same applies across a multi-file dataset: pyarrow.dataset.Dataset.to_table(columns=[...]) pushes the projection into every fragment, so a partitioned Overture-style export reads only the named columns from each file rather than one file’s worth of everything.

Production command

The script below reads a wide GeoParquet table, projects down to the geometry plus four style-relevant fields, and reports exactly how many bytes the projection saved. It streams row groups so peak memory stays bounded by one batch rather than the whole file.

import pyarrow.parquet as pq

SOURCE = "data/overture_buildings.geoparquet"
GEOM_COL = "geometry"                       # from geo metadata primary_column
STYLE_COLS = ["id", "class", "height", "name"]  # only what the style renders

def projected_columns(path: str) -> list[str]:
    schema = pq.read_schema(path)
    keep = [c for c in STYLE_COLS if c in schema.names]
    if GEOM_COL not in schema.names:
        raise KeyError(f"geometry column {GEOM_COL!r} absent; schema={schema.names}")
    return keep + [GEOM_COL]                 # never forget the geometry

def bytes_for_columns(path: str, columns: list[str]) -> int:
    """Sum compressed column-chunk sizes for the projected columns only."""
    md = pq.ParquetFile(path).metadata
    wanted = set(columns)
    total = 0
    for rg in range(md.num_row_groups):
        row_group = md.row_group(rg)
        for c in range(row_group.num_columns):
            col = row_group.column(c)
            if col.path_in_schema.split(".")[0] in wanted:
                total += col.total_compressed_size
    return total

if __name__ == "__main__":
    cols = projected_columns(SOURCE)
    full = pq.ParquetFile(SOURCE).metadata.serialized_size  # footer only; see note
    projected = bytes_for_columns(SOURCE, cols)
    all_cols = bytes_for_columns(SOURCE, pq.read_schema(SOURCE).names)
    print(f"columns kept : {cols}")
    print(f"projected read: {projected/1e6:8.1f} MB")
    print(f"full read     : {all_cols/1e6:8.1f} MB")
    print(f"I/O avoided   : {100*(1 - projected/all_cols):5.1f}%")

    # Stream only the projected columns into downstream processing
    pf = pq.ParquetFile(SOURCE)
    for batch in pf.iter_batches(batch_size=200_000, columns=cols):
        handle(batch)   # decode WKB, emit features, etc.

The DuckDB equivalent expresses the same projection as SQL. Naming the columns explicitly — never SELECT * — is what lets the Parquet reader skip chunks:

INSTALL spatial; LOAD spatial;
-- Only these five columns are read from the file
COPY (
  SELECT id, class, height, name, geometry
  FROM read_parquet('data/overture_buildings.geoparquet')
) TO 'projected.geoparquet' (FORMAT PARQUET);

To confirm DuckDB is actually pushing the projection down rather than reading everything and discarding, prefix the query with EXPLAIN ANALYZE and look for the PARQUET_SCAN node — its Projection list should name exactly your five columns, and the reported bytes scanned should drop accordingly.

Interaction effects

Projection is the first of three cuts you can stack before Tippecanoe ever runs, and each is orthogonal:

With bbox filtering. Column projection removes unwanted columns; bbox spatial filtering removes unwanted rows. Combine them in one DuckDB query — a narrow SELECT list plus a WHERE on the bbox column — and the reader skips both column chunks and whole row groups in a single pass. The two pushdowns compose: projected bytes are only read for the row groups that survive the spatial predicate.

With NDJSON streaming. When you stream GeoParquet to Tippecanoe’s stdin as NDJSON, the projected column set is exactly the set of properties each GeoJSON feature carries. Projecting at the reader means the serializer loop never sees the discarded fields, so the NDJSON payload is smaller and JSON encoding is faster — the savings propagate all the way down the pipe.

With attribute filtering at encode time. Tippecanoe’s own -y / --include attribute filters drop columns inside the encoder. Reader-side projection and encoder-side -y are belt-and-suspenders: projection saves the read and serialization cost that -y cannot, because by the time -y acts the data has already been read from disk and parsed. Use projection to bound I/O and RAM; use -y as the final guarantee that no stray attribute reaches the tile.

Performance impact

On wide tables the effect is close to linear in the fraction of bytes each kept column occupies. A representative Overture buildings extract with 47 columns, where geometry and four scalars account for roughly 15% of compressed bytes, reads about 85% fewer bytes under projection.

The RAM effect is frequently larger than the I/O effect, because the discarded columns are often the heaviest: dropping a wide struct tag map or a WKT twin of the geometry removes the objects that would otherwise dominate the decoded Arrow batch. Lower peak RAM is what lets a job that previously OOM-killed a 4 GB worker complete on the same worker.

One caveat sets the ceiling on the win: the geometry column is usually the single largest chunk in a spatial table, and you can never project it away. Once the read is down to geometry plus a few scalars, further trimming yields little — geometry dominates the remaining bytes. That is why projection and the row-wise cut are complementary rather than redundant: projection removes attribute weight, and bbox filtering removes geometry rows, which is the only lever left once the column set is already minimal.

Common mistakes

SELECT * (or the default columns=None) silently defeats pushdown. There is no error — the job simply reads every column. Symptom: your “optimised” run reads the same bytes and uses the same RAM as before. Fix: name columns explicitly and verify with EXPLAIN ANALYZE in DuckDB, or by comparing total_compressed_size sums as in the script above. If a wrapper builds the column list dynamically, assert len(cols) < len(schema.names) so a regression that reintroduces every column fails loudly.

Forgetting the geometry column. Projecting ["id", "class", "height", "name"] reads cleanly and fast — and produces features with no shapes, so Tippecanoe emits an empty or point-only layer. The failure is downstream and easy to misattribute. Guard it: raise if primary_column is absent from the projection, exactly as projected_columns() does above. Remember the geometry column may not be literally named geometry.

Projecting after materialising. table = pq.read_table(path); table = table.select(cols) looks like projection but is not — read_table with no columns argument has already read and decoded every column before select() narrows it. The pushdown must be an argument to the read call (read_table(path, columns=cols) or iter_batches(columns=cols)), not a post-hoc slice. Same trap in pandas: pd.read_parquet(path)[cols] reads the whole file first; pass pd.read_parquet(path, columns=cols) instead.

Parent: GeoParquet Input Processing

Related

• Bbox Spatial Filtering of GeoParquet Inputs — the row-wise counterpart: skip whole row groups outside your region while projection skips columns.
• Streaming GeoParquet to Tippecanoe stdin as NDJSON — the projected column set becomes the feature property set piped straight into the encoder.
• Dropping Unused Attributes to Reduce Tile Size — encoder-side -y / --exclude filters that complement reader-side projection.