PyQGIS Engineering & Automation

Making Batch Output Atomic with Write-Then-Rename

Publish batch output so readers never see a partial file: a temporary path in the destination directory, verification before the swap, releasing handles,…

TL;DR: write the output to a temporary path on the same filesystem, verify it, release every handle, then os.replace() it onto the live path — the rename is atomic, so a concurrent reader sees either the previous output or the new one and never something in between. This page is part of the scheduled batch processing and pipeline orchestration guide.

Complete Runnable Code

python
"""Publish a generated dataset without ever exposing a partial file."""
import os
import tempfile
from contextlib import contextmanager
from qgis.core import QgsVectorLayer


@contextmanager
def atomic_output(final_path: str, suffix: str = ""):
    """Yield a temporary path beside `final_path`; move it into place on success.

    The temporary file is created in the same directory so the final rename
    stays within one filesystem, which is what makes it atomic. On any
    exception the temporary file is removed and `final_path` is untouched.
    """
    directory = os.path.dirname(os.path.abspath(final_path)) or "."
    os.makedirs(directory, exist_ok=True)
    handle, temp_path = tempfile.mkstemp(dir=directory, suffix=suffix or
                                         os.path.splitext(final_path)[1])
    os.close(handle)
    os.unlink(temp_path)          # we want the name, not the empty file

    try:
        yield temp_path
        os.replace(temp_path, final_path)      # atomic within one filesystem
    except Exception:
        if os.path.exists(temp_path):
            os.unlink(temp_path)
        raise


def verify_vector(path: str, min_rows: int = 1) -> None:
    """Open the output and assert it is usable before it is published."""
    layer = QgsVectorLayer(path, "verify", "ogr")
    try:
        if not layer.isValid():
            raise RuntimeError("output at %s does not open" % path)
        count = layer.featureCount()
        if count < min_rows:
            raise RuntimeError("output has %d rows, expected at least %d" % (count, min_rows))
    finally:
        del layer                  # release the provider handle before any rename


def publish_layer(write_fn, final_path: str, min_rows: int = 1) -> None:
    """Run `write_fn(temp_path)`, verify the result, then publish it atomically."""
    with atomic_output(final_path) as temp_path:
        write_fn(temp_path)
        verify_vector(temp_path, min_rows)
What a reader sees during a safe publish A timeline of a publish showing that a concurrent reader sees the previous output throughout the write and verification, and the new output only after the rename — never a partial file. one publish, from a reader’s point of view writing reader sees the old file verifying reader sees the old file rename one instant no in-between after reader sees the new file

Architecture Breakdown

Why the rename is the whole trick

os.replace() maps to a filesystem rename, which POSIX and NTFS both guarantee to be atomic within a single filesystem: at any instant the path refers to the old inode or the new one. There is no window in which it refers to a partially written file, and no window in which it refers to nothing at all.

That guarantee is what lets a consumer read the output at any moment without coordination. No lock, no “is it ready” flag file, no retry loop — the reader either gets yesterday’s dataset or today’s, and both are complete.

The same-filesystem requirement

A rename across filesystems is not a rename; it is a copy followed by a delete, and the copy is observable. Creating the temporary file in the destination directory, as the context manager does, is what keeps the operation on one filesystem.

What breaks the atomicity guarantee A grid of four mistakes — writing in place, renaming across filesystems, leaving the file open, and skipping verification — showing what each allows a reader to observe. what a reader can see writing in place a half-written dataset rename across filesystems a copy in progress handle still open a rename that fails on Windows no verification a complete but wrong file

This is the mistake most likely to survive testing, because a developer machine usually has one big filesystem while a server has a separate volume for data. The code works locally and quietly loses its guarantee in production.

Releasing handles before renaming

On Windows an open handle blocks a rename outright; on Linux the rename succeeds but the writer keeps writing to the old inode, which is worse because it fails silently. Either way, every QgsVectorLayer or GDAL dataset pointing at the temporary file must be released first.

del layer is the blunt instrument; in a longer function, scoping the verification inside its own helper — as above — makes the release automatic when the helper returns.

Verifying Before You Publish

The rename guarantees a reader never sees half a file. Whether the whole file is worth reading is a separate question, and one the job has to answer for itself.

What to verify before renaming A decision tree over output types: check row counts for a vector layer, check dimensions and statistics for a raster, and check parse validity for a text or JSON artefact. What kind of output is this? vector opens + row count against a floor raster dimensions + stats not all nodata text / JSON parses + non-empty schema if you have one

Atomicity guarantees a reader never sees a partial file. It says nothing about whether the file is correct, and a complete but empty output is a worse outcome than no output at all, because downstream systems will happily consume it.

The check should be cheap and specific to the output type. For a vector layer, that it opens and has at least some plausible number of rows; for a raster, that the dimensions are right and the statistics are not entirely nodata; for a JSON artefact, that it parses and has the keys the consumer expects. Comparing against the previous run’s numbers is better still: an output that suddenly has three per cent of yesterday’s rows is almost certainly a truncated input rather than a genuine change.

Publishing a Multi-File Output

Formats with sidecars — shapefile above all — cannot be published with a single rename, because there is no one file to rename. Two approaches work.

The first is to stop using them: a GeoPackage is a single file and the pattern applies directly, which is one of the better arguments for it in an automated pipeline. The second, when the format is fixed by a consumer, is to publish a directory rather than files:

python
import os
import shutil


def publish_directory(build_fn, final_dir: str) -> None:
    """Build into a temporary directory and swap it into place."""
    parent = os.path.dirname(os.path.abspath(final_dir)) or "."
    staging = os.path.join(parent, ".staging-" + os.path.basename(final_dir))
    previous = final_dir + ".previous"

    shutil.rmtree(staging, ignore_errors=True)
    os.makedirs(staging)
    build_fn(staging)

    if os.path.exists(final_dir):
        os.replace(final_dir, previous)        # keep the old one, briefly
    os.replace(staging, final_dir)
    shutil.rmtree(previous, ignore_errors=True)

A directory swap is two renames rather than one, so there is a brief instant where the path does not exist. That is a weaker guarantee than the single-file case, and worth stating plainly rather than pretending otherwise — but it is still far better than writing files into a live directory one at a time.

Production Best Practices

  • Create the temporary file in the destination directory, so the rename stays on one filesystem.
  • Verify before renaming, and make the check specific to the output type.
  • Release every handle before the rename, including layers you only opened to verify.
  • Clean up the temporary file on failure, or a failing job slowly fills the volume.
  • Prefer single-file formats for anything published this way.
  • Compare against the previous run where you can; an unexpected size change is the cheapest data-quality signal available.

Frequently Asked Questions

Is os.replace() really atomic on Windows?

For a single file, yes — it maps to MoveFileEx with the replace flag, which is atomic with respect to other readers. What differs from POSIX is that Windows will not perform the rename at all while another process holds an open handle to either file, so the operation fails loudly rather than corrupting anything.

That failure is a feature: it tells you about a handle you forgot to release, which on Linux would have been a silent bug.

What about readers that keep the file open across runs?

A reader holding an open handle continues to see the old inode after the rename, because on POSIX the file exists as long as someone has it open. That is usually what you want — a long-running query completes against a consistent snapshot — but it means disk space is not reclaimed until every reader closes.

For a service that holds files open indefinitely, the pattern needs a reload signal as well; the rename alone will not reach it.

Should I keep the previous output?

Keeping one previous version costs a little disk and buys a fast rollback, which is worth it for anything a person consumes. Renaming the old file to .previous before the swap gives you that in one extra line.

Keeping many versions is a different job — that is archival, and it belongs in a directory of timestamped outputs rather than in the publication step.

Does this work for a database table?

The equivalent is a transaction: write to a staging table, verify it, then swap in a single transaction that renames or replaces the live table. PostGIS handles this cleanly, and it gives the same guarantee — readers see the old table or the new one.

What does not work is deleting and re-inserting rows in the live table, which exposes every intermediate state to anyone querying at the time.

How do I make the temporary file obvious in a directory listing?

Prefix it with a dot or a known marker such as .staging-, so a person looking at the directory during a run can tell what it is, and so any cleanup script can recognise orphans safely. A temporary file named like the real output is a trap for the next person who tidies up.

What if verification fails?

Do not publish, do not delete the previous output, and exit non-zero with the reason. The previous output stays live, which is the right outcome: yesterday’s correct data is more useful than today’s wrong data, and the alert says which. Keeping the failed temporary file around for inspection is often worth it too, provided something eventually cleans it up.