Vector Tile Reference

Parallelizing Tippecanoe Builds by Bounding Box

Tippecanoe uses several threads internally but one process cannot outrun one machine’s disk on a planet-scale input. Splitting the input by bounding box and running N independent builds turns the expensive stage into an embarrassingly parallel one; the merge afterwards is serial and becomes the floor on total wall clock.

When to Use This

When a single build no longer fits the time budget, or no longer fits the machine. The threshold in practice is around an hour of build time or a source that exceeds available RAM — below that, the merge tail and the extra machinery cost more than they save.

It is also the answer to SQLite lock contention. Two workers cannot safely write one MBTiles file; giving each worker its own output file removes the problem by construction rather than by tuning.

Specification Detail

Element Choice Consequence
Partition key Quadkey at a fixed zoom Aligns with the tile grid; no tile straddles two partitions
Feature assignment Centroid, or an explicit region column Each feature tiled once, whole
Worker count Cores minus one, bounded by disk Tippecanoe is I/O heavy; more workers than the disk supports slows everything
Output One .mbtiles per partition No shared handle, no lock contention
Merge Single tile-join over all partitions Serial, and the wall-clock floor
N independent builds and one serial mergeThe source is split by bounding box into disjoint partitions, each built by its own worker into its own archive, and all archives merged by a single tile-join pass.PARALLEL SHAPESplitquadkey bboxesdisjointN workersone archive eachno shared handlescales with cores anddisktile-joinsingle writerthe serial floorVerifywhole tileset
Everything before the merge scales with cores. The merge does not, which is why partition count matters more than worker count past a point.

Production Command

bash
#!/usr/bin/env bash
set -euo pipefail

PARTITION_ZOOM=4
WORKERS=$(( $(nproc) - 1 ))
mkdir -p extract build

# 1. Emit the bounding box of every partition at the partition zoom
python3 - "$PARTITION_ZOOM" > partitions.tsv <<'PY'
import math, sys
z = int(sys.argv[1])
def lon(x): return x / 2**z * 360.0 - 180.0
def lat(y): return math.degrees(math.atan(math.sinh(math.pi * (1 - 2 * y / 2**z))))
for x in range(2**z):
    for y in range(2**z):
        print(f"{z}-{x}-{y}\t{lon(x)}\t{lat(y+1)}\t{lon(x+1)}\t{lat(y)}")
PY

# 2. Extract and build one partition (skipping empties)
build_one() {
  IFS=$'\t' read -r NAME W S E N <<< "$1"
  ogr2ogr -f GeoJSONSeq "extract/$NAME.geojsonl" source.gpkg roads \
    -spat "$W" "$S" "$E" "$N" -t_srs EPSG:4326 -skipfailures
  [ -s "extract/$NAME.geojsonl" ] || { rm -f "extract/$NAME.geojsonl"; return 0; }
  tippecanoe -o "build/$NAME.mbtiles" -l roads -Z5 -z14 --force --quiet \
    --drop-densest-as-needed --simplification=8 "extract/$NAME.geojsonl"
}
export -f build_one

# 3. Run the workers
parallel -j "$WORKERS" build_one :::: partitions.tsv

# 4. Merge — one process, always over every archive that exists
tile-join -o dist/roads.mbtiles -f \
  --name "Roads v43" --attribution "© OpenStreetMap contributors" \
  build/*.mbtiles

# 5. Gate the merged result
sqlite3 dist/roads.mbtiles \
  "SELECT COUNT(*) FROM tiles WHERE length(tile_data) > 500000" | grep -qx 0

The -s test that skips empty extracts matters more than it looks: at partition zoom 4 over a country dataset, most of the 256 partitions are ocean and produce nothing. Building them anyway wastes worker slots and adds empty archives to the merge.

Sizing the Partitions

The wall clock is set by the slowest partition, not the average, and geographic data is wildly non-uniform. A partition covering a dense metropolitan area can take twenty times as long as one covering farmland, which means a naive uniform grid leaves most workers idle waiting for one.

Two corrections help. Choose the partition zoom so that the densest partition is a comfortable unit of work rather than sizing for the average. And schedule longest-first: extract all partitions, sort by input size descending, and feed that order to the worker pool, so the long jobs start immediately and the short ones fill the tail.

Why the slowest partition sets the scheduleBuild times for four partitions of the same size in area: dense metropolitan, suburban, rural and ocean, showing a twenty-fold spread.LOAD SKEWminutes to build one partitionDense metropolitan41 minSuburban12 minRural3 minOcean (skipped)12 s
Sizing partitions by area gives this spread. Sizing by feature count evens it out and is worth the extra pass over the source.
Where a parallel build's forty minutes goFour phases of a 64-partition parallel build: extraction, parallel tile generation, the serial merge and verification.WALL CLOCK40 min, 64 partitions5 minExtract 64partitions21 minParallel generation19 mintile-join mergeVerify
The merge does not parallelise, so past a few dozen partitions it becomes the dominant cost — which is where the optimum partition count comes from.

Interaction Effects

With SQLite locking. One writer per file is the entire fix — no PRAGMA makes two writers safe. The lock contention guide covers why WAL mode helps readers and not this.

With disk throughput. Tippecanoe writes a large temporary index per build. N concurrent builds means N such indexes, and on a single spinning disk or a throttled cloud volume the workers starve each other. Watch the disk queue depth before adding workers.

With incremental rebuilds. Partitioning is the prerequisite for rebuilding only what changed. Adding hashing to the script above is a dozen lines and converts a parallel build into an incremental one.

With feature assignment. A -spat extract selects features intersecting the box, so a feature crossing a boundary appears in both partitions. For most basemap layers the duplicate is harmless because the merge concatenates identical geometry; for translucent fills and labels it is visible, and an explicit centroid-based assignment is required.

Performance Impact

Measured on a 16-core machine with NVMe storage, building a national road network:

Configuration Build phase Merge Total
Single build 3 h 34 — 3 h 34
4 partitions, 4 workers 58 min 14 min 1 h 12
64 partitions, 15 workers 21 min 19 min 40 min
256 partitions, 15 workers 14 min 27 min 41 min

The last two rows show the crossover: past a point, finer partitioning shortens the parallel phase and lengthens the merge by more. The optimum here is a few dozen partitions, and it moves with the machine.

Common Mistakes

More workers than the disk supports. Adding workers past the disk’s throughput makes every build slower, and the symptom is high iowait with low CPU.

Uniform partitions over non-uniform data. One partition takes an hour and fifteen workers idle for fifty minutes of it.

Merging with a glob whose order varies. Changes layer order inside tiles between runs, which changes the bytes without changing the content and defeats deduplication.

Verifying only the partitions. Each partition can pass the size budget while the merged tile fails, exactly as in layer composition.

FAQ

Does partitioning change the output?

It should not, for correctly assigned features. Verifying that once — building both ways and comparing tile content hashes — is worth the afternoon it costs, because everything afterwards depends on the assumption.

Why not just give Tippecanoe more threads?

It already uses several, and the bottleneck at scale is the single temporary index and the single output file. Separate processes with separate files sidestep both.

Should partitions align with administrative boundaries?

Only if the source is naturally organised that way. Quadkeys align with the tile grid, which guarantees no tile straddles two partitions — a property administrative boundaries do not have.

Can the merge be parallelised?

Not usefully with tile-join, which is single-writer. Merging in a tree — pairs, then pairs of pairs — is possible and rarely pays, because each level re-reads and re-writes everything.