Incremental and Partitioned Tile Builds

A whole-planet rebuild for a change in one city is the default behaviour of every naive tile pipeline, and it is why nightly builds turn into weekend builds. Partitioning by region makes the build parallel; hashing each partition’s input makes it incremental. Together they turn a four-hour serial rebuild into twenty minutes of work on the parts that actually changed.

Prerequisites

Core Concept: Partition Boundaries and the Overlap Problem

The obvious partition is a bounding box per region, and the obvious bug follows immediately: a feature crossing a boundary appears in two partitions, or in neither.

Both failure modes are visible. A feature in neither leaves a gap along the boundary. A feature in both is drawn twice, which for a translucent fill is visible as a darker seam and for a label is a duplicate.

The fix is to make the partition assignment a property of the feature, not of the tile: each feature belongs to exactly one partition, chosen deterministically — by the partition containing its centroid, or by an explicit region column in the source. Tiles at the boundary then contain features from several partitions, which is exactly what the merge is for.

Step-by-Step Implementation

Step 1 — Choose a partition scheme

Quadkeys at a fixed low zoom make good partitions: they are disjoint by construction, they align with the tile grid so no tile straddles two partitions at higher zooms, and their count is predictable.

def quadkey(x: int, y: int, z: int) -> str:
    parts = []
    for i in range(z, 0, -1):
        digit = 0
        mask = 1 << (i - 1)
        if x & mask: digit += 1
        if y & mask: digit += 2
        parts.append(str(digit))
    return "".join(parts)

# z4 gives 256 partitions; z5 gives 1024. Pick so each holds
# a workable amount of data — a few hundred MB of source is a good target.

Step 2 — Extract and hash each partition

for QK in $(cat partitions.txt); do
  ogr2ogr -f GeoJSONSeq "extract/$QK.geojsonl" source.gpkg \
    -spat $(bbox_of "$QK") -t_srs EPSG:4326
  sha256sum "extract/$QK.geojsonl" | cut -d' ' -f1 > "extract/$QK.sha"
done

The hash covers the extracted data, not the source — that way a change anywhere in the source only invalidates the partitions whose extract actually differs.

Step 3 — Rebuild only what changed

for QK in $(cat partitions.txt); do
  NEW=$(cat "extract/$QK.sha")
  OLD=$(cat "build/$QK.sha" 2>/dev/null || echo none)
  if [ "$NEW" = "$OLD" ] && [ -f "build/$QK.mbtiles" ]; then
    continue                      # unchanged: keep the existing archive
  fi
  tippecanoe -o "build/$QK.mbtiles" -l roads -Z5 -z14 --force \
    --drop-densest-as-needed --simplification=8 "extract/$QK.geojsonl"
  cp "extract/$QK.sha" "build/$QK.sha"
done

Step 4 — Merge every partition, always

The merge runs over all partitions regardless of which were rebuilt, because the published tileset must contain all of them:

tile-join -o dist/roads.mbtiles -f \
  --name "Roads v43" --attribution "© OpenStreetMap contributors" \
  build/*.mbtiles

Step 5 — Verify the whole tileset, not the changed parts

sqlite3 dist/roads.mbtiles \
  "SELECT COUNT(*) FROM tiles WHERE length(tile_data) > 500000;"
sqlite3 dist/roads.mbtiles \
  "SELECT COUNT(*), MIN(zoom_level), MAX(zoom_level) FROM tiles;"

Verifying only the rebuilt partitions is the mistake that makes incremental builds untrustworthy. The gate must see what readers will.

Choosing the Partition Zoom

The partition zoom is the one parameter that has to be chosen rather than derived, and it trades two costs against each other.

Too few partitions and incrementality is coarse: a change to one street invalidates a partition covering a whole country, and the rebuild is nearly as expensive as a full one. Parallelism is also limited, since a build cannot use more workers than it has partitions with data in them.

Too many partitions and the merge dominates. tile-join reads and writes every tile in every input, so its cost grows with partition count even when the total tile count is unchanged, and past a few hundred partitions the serial merge is longer than the parallel build it enabled.

The practical procedure is to pick the zoom that puts a typical change inside a handful of partitions, then check that the count is within an order of magnitude of the worker count. At zoom 3 a global dataset has 64 partitions, at zoom 4 it has 256, and at zoom 5 it has 1,024 — of which, for any land-based dataset, well under half contain data at all.

One thing that is not a consideration: partition size in bytes. Geographic data is so non-uniformly distributed that sizing by area produces a twenty-fold spread in build time between the densest and sparsest partitions. If the spread matters — and it does once the slowest partition sets the wall clock — size by feature count instead, which costs one extra pass over the source and evens the schedule out considerably.

Optimization Knobs

Integration With Adjacent Pipeline Stages

Upstream, the source must support a spatial predicate efficiently. For GeoParquet this is bbox filtering with row-group skipping; for PostGIS it is a GiST index and a ST_Intersects clause.

Sideways, partitioning by region and composing by layer combine: each layer can be built as a set of regional shards, merged within the layer and then across layers.

Downstream, nothing changes. The merged output is an ordinary tileset published under a version prefix, and a reader cannot tell how it was produced.

Troubleshooting

Seams along partition boundaries

Symptom: A hairline discontinuity following a straight line that does not correspond to any real feature.

Cause: Features were clipped at the partition boundary rather than assigned whole, so each half was simplified independently.

Fix: Assign by centroid or by an explicit region column, and re-extract. No merge setting repairs clipped geometry.

Features appear twice near a boundary

Symptom: Doubled labels, or a translucent fill that is visibly darker along a line.

Cause: Overlapping extract predicates — usually a bounding box with an inclusive test on both edges.

Fix: Make the predicate half-open, or assign explicitly rather than by spatial predicate.

An incremental build produces a tileset missing a region

Symptom: A whole area is blank in the published tileset.

Cause: A partition archive was deleted or failed to build, and the merge silently produced output without it.

Fix: Assert the partition count before merging — the count of archives must equal the count of partitions, every run.

Proving the Partitioned Build Equals the Monolithic One

Everything downstream of this topic assumes that a partitioned build produces the same tileset a single invocation would. That assumption is worth verifying once, deliberately, because every subsequent optimisation compounds on it and a subtle divergence is very hard to notice later.

The test is direct. Build a modest region both ways, then compare the tiles rather than the archives:

# Monolithic
tippecanoe -o /tmp/whole.mbtiles -l roads -Z5 -z14 --force \
  --drop-densest-as-needed --simplification=8 region.geojsonl

# Partitioned, then merged
# ... build/*.mbtiles produced per partition ...
tile-join -o /tmp/merged.mbtiles -f build/*.mbtiles

# Compare tile bodies, not file bytes: row order and metadata legitimately differ
for DB in /tmp/whole.mbtiles /tmp/merged.mbtiles; do
  sqlite3 "$DB" \
    "SELECT zoom_level, tile_column, tile_row, hex(md5(tile_data))
     FROM tiles ORDER BY zoom_level, tile_column, tile_row;" > "$DB.hashes"
done
diff /tmp/whole.mbtiles.hashes /tmp/merged.mbtiles.hashes && echo "identical"

Comparing file bytes will always differ — insertion order, metadata and SQLite page layout are not stable — so the comparison has to be per-tile content.

A difference is informative rather than alarming, and it will be one of three things. Features clipped at a partition boundary rather than assigned whole, which shows up as differing tiles along partition edges only. Simplification differences on features that span partitions, which is the same cause seen at a different zoom. Or genuinely non-deterministic ordering, which points at a shell glob or an unsorted input.

Once the comparison passes, record which flag set and which partition scheme it was verified against, because both are inputs to the equivalence. Re-run it when either changes — which is the same trigger that invalidates every cached partition anyway, so the two checks belong together.

When Partitioning Is the Wrong Answer

Three situations where the machinery costs more than it returns, and it is worth naming them because partitioning has a way of being adopted reflexively.

The build already fits the window. A forty-minute nightly build with a six-hour window does not need to be twenty minutes. The partitioning adds a merge step, a cache directory, a partition-count assertion and a class of boundary bugs, in exchange for time nobody was waiting for.

Every run changes everything. A full re-import, a regenerated derivative, a source whose every row carries a new timestamp — all of these invalidate every partition on every run. The build is then parallel, which is useful, and incremental in name only.

The source cannot be queried by region efficiently. If extracting a partition requires a full scan of the source, the extraction phase costs N full scans instead of one, and the parallel build is slower than the monolithic one it replaced. Fix the source’s spatial indexing first; the partitioning is worth nothing until that is true.

In-Depth Guides

Parallelizing Tippecanoe Builds by Bounding Box — the parallel half: worker sizing, extraction predicates and the merge tail.

Rebuilding Only Changed Regions in a Tile Pipeline — the incremental half: hashing, cache keys and the correctness checks partial rebuilds need.

FAQ

How many partitions is right?

Enough that a typical change touches a few of them, and few enough that the merge stays manageable. A few hundred is a good working range for a country-scale tileset; thousands make the merge the dominant cost.

Does partitioning change the output tiles?

It should not. A correctly partitioned build produces the same tiles as a monolithic one, and that equivalence is worth verifying once by building both ways and comparing tile hashes.

Can partitions have different zoom ranges?

Yes, and it is occasionally useful — a dense metropolitan partition built deeper than a rural one. The merged range is the union, and the style should declare the deepest.

What invalidates every partition at once?

A change to the flag set, the Tippecanoe version, or the partition scheme itself. All three belong in the hash, so that the invalidation is automatic rather than something someone has to remember.

Related

• Resolving SQLite Locks in Large MBTiles Generation — why each worker must write its own file.
• Layer Composition and tile-join — the same decomposition applied by layer instead of region.
• CI/CD Tile Build Automation — the hashing and idempotency this topic depends on.
• Bbox Spatial Filtering of GeoParquet Inputs — making the per-partition extract cheap.
• Automated Generation Pipelines with Tippecanoe — the parent section.