Rebuilding Only Changed Regions in a Tile Pipeline

Hash each partition’s extracted input, compare it with the hash recorded beside the previous build of that partition, and skip the build when they match. That single comparison converts a parallel pipeline into an incremental one, and on a typical daily change it removes 95% of the work.

When to Use This

When most runs change a small fraction of the data. Daily OpenStreetMap diffs, a regional editing team, a dataset where corrections arrive continuously but locally. If every run changes everything — a full re-import, a regenerated derivative — incrementality buys nothing and adds machinery.

Specification Detail: What Belongs in the Cache Key

The hash must cover everything that can change the output tiles, or a stale partition will be published as current.

Leave any of these out and there is a change that produces identical hashes and different correct output — which is the one failure mode an incremental build must not have.

import hashlib
import json
import subprocess
from pathlib import Path

def partition_key(extract: Path, flags: list[str], layer: str, scheme: str) -> str:
    h = hashlib.sha256()
    h.update(extract.read_bytes())
    version = subprocess.run(["tippecanoe", "--version"],
                             capture_output=True, text=True).stderr.strip()
    h.update(json.dumps({
        "flags": flags, "layer": layer, "scheme": scheme, "tippecanoe": version,
    }, sort_keys=True).encode())
    return h.hexdigest()

Production Command

import shutil
import subprocess
from pathlib import Path

FLAGS = ["-Z5", "-z14", "--drop-densest-as-needed", "--simplification=8"]
LAYER = "roads"
SCHEME = "quadkey-z4-centroid"


def build_partition(name: str, extract: Path, build_dir: Path) -> str:
    """Build the partition if its key changed. Returns 'built' or 'cached'."""
    archive = build_dir / f"{name}.mbtiles"
    keyfile = build_dir / f"{name}.key"
    key = partition_key(extract, FLAGS, LAYER, SCHEME)

    if archive.exists() and keyfile.exists() and keyfile.read_text().strip() == key:
        return "cached"

    tmp = archive.with_suffix(".mbtiles.tmp")
    subprocess.run(
        ["tippecanoe", "-o", str(tmp), "-l", LAYER, "--force", "--quiet",
         *FLAGS, str(extract)],
        check=True,
    )
    # Publish the archive and its key atomically, in that order: an interrupted
    # run must never leave a key claiming an archive that does not match it.
    shutil.move(tmp, archive)
    keyfile.write_text(key + "\n")
    return "built"


results = {name: build_partition(name, Path(f"extract/{name}.geojsonl"), Path("build"))
           for name in open("partitions.txt").read().split()}
built = sum(1 for v in results.values() if v == "built")
print(f"{built} rebuilt, {len(results) - built} cached")

The ordering in that function is the part worth copying. Writing the key after moving the archive means an interrupted run leaves a stale-but-honest state: the key is missing, so the next run rebuilds. Writing it first would leave a key asserting that an archive matches when it does not.

The Checks a Partial Rebuild Needs

An incremental build is trusted only as far as it is verified, and the verification must cover the published tileset rather than the rebuilt parts.

Partition count. The number of archives entering the merge must equal the number of partitions expected. A partition whose build failed and whose archive was deleted produces a merged tileset with a hole and no error.

Merged size budget. Run the size gate on the merged output every run, not only when something was rebuilt. A change in one partition can push a boundary tile over the limit.

Total feature count within tolerance. Compare against the previous published build. A large swing means the extract changed shape — often a source schema change — rather than the data genuinely moving.

A periodic full rebuild. Once a week or once a month, rebuild every partition from scratch and compare the merged output’s tile hashes against the incremental one. Any divergence is a bug in the cache key, and this is the only check that will ever find it.

Interaction Effects

With extraction. Hash the extract rather than the source, so a change in one region does not invalidate its neighbours. It also means the extraction step runs every time — cheap, and the thing that makes the comparison meaningful.

With CI caching. The build directory is the cache. In a hosted CI runner it must be restored and saved as an artefact, and the runner’s cache size limits usually mean caching the extracts and keys rather than the archives.

With attribute joins. If the pipeline also does a daily attribute join, the geometry side becomes almost entirely cached and the join dominates — which is exactly the intended end state.

Performance Impact

The first row is the honest cost of incrementality: even a no-op run pays for extraction and the merge. Skipping the merge when no partition changed brings that down to a few minutes, and is worth the extra conditional.

Common Mistakes

Hashing the source instead of the extract. Any change anywhere invalidates every partition, and the pipeline is parallel but not incremental.

Leaving the flag set out of the key. Changing --simplification produces a tileset that is half old settings and half new, with a visible discontinuity at partition boundaries.

Writing the cache key before the archive. An interrupted run then claims a partition is current when it is not, and the stale tiles are published on the next merge.

Never doing a full rebuild. A cache-key bug is invisible until something forces the comparison. Schedule it.

FAQ

How much does the extraction cost if nothing changed?

It has to run to produce the bytes being hashed, so it is the floor on a no-op run. On most sources it is a few minutes; if it is not, a change feed from the source is the next optimisation.

Can I hash the source query result instead of a file?

Yes, and it is tidier — hash the streamed extract as it is written rather than reading the file back. The property that matters is that the hash covers exactly what Tippecanoe will consume.

What if a partition legitimately produces no tiles?

Record that. An empty partition with a recorded key is cached like any other; a missing archive with no key is indistinguishable from a failed build, which is why the partition-count check matters.

Does this work with PMTiles?

The per-partition archives must be MBTiles, since tile-join reads those. Convert the merged output at the end, as in the conversion guide.

Related

• Incremental and Partitioned Tile Builds — the parent topic and the partitioning half.
• Parallelizing Tippecanoe Builds by Bounding Box — the pipeline this adds hashing to.
• CI/CD Tile Build Automation — the idempotency argument in its general form.
• Joining CSV Attributes to Tiles with tile-join — the other half of a two-cadence pipeline.