PyQGIS Engineering & Automation

Streaming Cloud Optimized GeoTIFFs over HTTP

Read remote rasters without downloading them: GDAL /vsicurl/ range requests, the configuration options that decide performance, how to verify a file is…

TL;DR: open the remote file through GDAL’s /vsicurl/ virtual filesystem, set GDAL_DISABLE_READDIR_ON_OPEN so it does not probe the directory on every open, and read windowed extents — a correctly built COG then costs a handful of HTTP range requests instead of a download. This page is part of the vector and raster data access patterns guide.

Complete Runnable Code

python
"""Read a window from a remote COG without downloading the whole file."""
from qgis.core import QgsRasterLayer, QgsRectangle

try:
    from osgeo import gdal
except ImportError:
    gdal = None


def configure_remote_access() -> None:
    """GDAL settings that make remote raster reads fast. Call once at start-up."""
    if gdal is None:
        return
    gdal.SetConfigOption("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
    gdal.SetConfigOption("CPL_VSIL_CURL_ALLOWED_EXTENSIONS", ".tif,.tiff,.vrt")
    gdal.SetConfigOption("GDAL_HTTP_MULTIPLEX", "YES")
    gdal.SetConfigOption("VSI_CACHE", "TRUE")
    gdal.SetConfigOption("VSI_CACHE_SIZE", "25000000")     # 25 MB per file


def remote_raster(url: str, name: str = "remote") -> QgsRasterLayer:
    """Open a remote GeoTIFF as a layer. Nothing but the header is fetched."""
    configure_remote_access()
    layer = QgsRasterLayer("/vsicurl/%s" % url, name)
    if not layer.isValid():
        raise RuntimeError("could not open %s: %s" % (url, layer.error().summary()))
    return layer


def is_cloud_optimized(url: str) -> dict:
    """Report whether the remote file has the internal structure a COG needs."""
    if gdal is None:
        raise RuntimeError("GDAL Python bindings are not available")
    configure_remote_access()
    dataset = gdal.Open("/vsicurl/%s" % url)
    if dataset is None:
        raise RuntimeError("could not open %s" % url)

    band = dataset.GetRasterBand(1)
    block_x, block_y = band.GetBlockSize()
    return {"tiled": block_x != dataset.RasterXSize,   # a strip is the whole width
            "block_size": (block_x, block_y),
            "overview_count": band.GetOverviewCount()}


def read_window(url: str, extent: QgsRectangle, width: int, height: int):
    """Read one window at a chosen output size, letting GDAL pick an overview."""
    layer = remote_raster(url)
    provider = layer.dataProvider()
    block = provider.block(1, extent, width, height)
    return block
What makes a remote read cheap Three bands showing the COG read path: your window request, GDAL translating it into HTTP range requests against the overviews and tiles it needs, and the server returning only those byte ranges. Your code a window + resolution what you asked for GDAL / VSI reads the header once tile index range requests only the tiles needed picks an overview by resolution The server HTTP 206 responses partial content no full download ever If the server does not support range requests, GDAL has to fetch the whole file — which is the difference between a second and several minutes.

Architecture Breakdown

/vsicurl/ turns a URL into a file

GDAL’s virtual filesystem layer implements seek and read over HTTP using range requests. Prefixing a URL with /vsicurl/ gives every GDAL-based reader — including QGIS’s raster provider — random access to a remote file without any download step.

The prefix is all that is required, but the defaults are tuned for local files and produce surprisingly poor remote behaviour until they are adjusted.

The settings that decide remote read performance A grid of four GDAL configuration options for remote rasters, showing what each controls and a sensible starting value. controls a starting value GDAL_DISABLE_READDIR_ON_OPEN sibling file probing EMPTY_DIR CPL_VSIL_CURL_ALLOWED_EXTENSIONS which files are opened remotely .tif GDAL_HTTP_MULTIPLEX parallel range requests YES VSI_CACHE_SIZE bytes cached per file 25000000

GDAL_DISABLE_READDIR_ON_OPEN is the one that matters most. By default GDAL lists the containing directory when opening a file, looking for sidecars — which over HTTP means an extra request, and against object storage can mean listing a bucket with thousands of objects.

What makes a GeoTIFF cloud optimized

Two structural properties, both of which the file must have been written with. It must be internally tiled rather than striped, so a spatial window maps to a small set of contiguous byte ranges. And it must carry overviews, so a request for a coarse view reads a small pyramid level instead of decimating full-resolution data.

A plain GeoTIFF served over HTTP is still readable through /vsicurl/, and it will be slow: a striped file means every window touches rows spanning the full width, and no overviews means every zoomed-out view reads everything.

Windowed reads are the whole point

Asking for the full extent at full resolution downloads the file the long way round. The value comes from asking for the extent and resolution you actually intend to use — provider.block() with an extent and an output size lets GDAL choose the appropriate overview and fetch only the tiles that intersect.

That is the same discipline as local raster access, described in reading raster pixel values, with a much larger penalty for ignoring it.

Diagnosing a Slow Remote Raster

Slow remote access has three common causes, and is_cloud_optimized() distinguishes the first from the others in one call.

Why the remote raster is slow A decision tree over three causes of slow remote raster reads: the file is not a real COG, the server does not support range requests, and GDAL is probing for sibling files on every open. The remote raster takes minutes no internal tiles not a real COG reprocess it whole file fetched no range support check the server slow to open directory probing set READDIR_ON_OPEN

If tiled is False or overview_count is zero, the file is not a COG whatever it is called, and no amount of client configuration will fix it — it has to be rewritten, which gdal_translate with -of COG does in one command.

If the structure is right but reads are still slow, check whether the server honours range requests: a curl -I showing Accept-Ranges: bytes is the quick test, and its absence means every read fetches the whole file. Some content delivery configurations strip this, which is a frustrating way to lose the entire benefit.

If opening is slow but reading is fast, it is directory probing, and the configuration option above resolves it.

Using It in a Pipeline

python
def sample_remote_at_points(url: str, points: list, band: int = 1) -> list:
    """Sample a remote raster at scattered points with one layer open."""
    layer = remote_raster(url)
    provider = layer.dataProvider()
    results = []
    for point in points:                     # one identify per point, one open in total
        value = provider.identify(point, 1).results().get(band)
        results.append(value)
    return results

Opening the layer once and reusing it matters far more remotely than locally: each open re-reads the header and the tile index, which is several round trips before any pixel is fetched.

What Changes About How You Write Code

Reading remotely does not change the API — the same provider, the same block calls — but it changes which habits are expensive. Three in particular flip from harmless to costly.

Opening a layer repeatedly is close to free locally and expensive remotely, because each open costs several round trips before any pixel arrives. Reading at full resolution when a coarse view would do is wasteful locally and dramatic remotely, since the overview pyramid exists precisely so you do not have to. And iterating points one at a time, each with its own identify call, turns into one request per point unless the tiles happen to be cached.

The remedy in every case is to batch and to be explicit about resolution: open once, ask for the extent and size you actually intend to display or analyse, and let the VSI cache absorb the repeats. Code written that way is also faster on local files — it is simply that remote access makes the difference impossible to ignore.

Production Best Practices

  • Set GDAL_DISABLE_READDIR_ON_OPEN before opening anything remote.
  • Verify the file is genuinely a COG rather than trusting the name.
  • Read windows, never the full extent at full resolution.
  • Open the layer once and reuse it for every sample.
  • Enable the VSI cache so repeated reads of the same tiles do not re-request them.
  • Check Accept-Ranges on the server when performance is inexplicably bad.

Frequently Asked Questions

Do I need the GDAL Python bindings?

Not for reading — QgsRasterLayer("/vsicurl/...") works with the QGIS provider alone, and the configuration options can be set through environment variables instead. The bindings are useful for the diagnostic function above, which inspects block size and overview count directly.

Does this work with authenticated endpoints?

Yes, through the relevant virtual filesystem: /vsis3/ for S3, /vsigs/ for Google Cloud Storage, /vsiaz/ for Azure, each configured with credentials through GDAL config options. For a plugin, those credentials belong in the QGIS authentication database rather than in code, as covered in storing credentials with QgsAuthManager.

How large a VSI cache should I set?

Enough to hold the tiles a typical operation touches more than once — 25 to 100 megabytes covers most interactive use. The cache is per file and per process, so a job opening many files needs the total in mind rather than the individual number.

Is a COG slower than a local file?

For a single small window, noticeably: the round trips dominate. For a large file you only need part of, dramatically faster, because a local copy means downloading everything first. The crossover is early, which is why COGs are worth the conversion for anything above a few hundred megabytes.

Should I use a VRT over several COGs?

A VRT is a good way to present a tiled collection as one layer, and GDAL will fetch only from the constituent files a window touches. Keep the VRT itself local or nearby, since it is read in full on every open, and be aware that a VRT over thousands of files makes opening slow even when reading is fast.

Can I write a COG from PyQGIS?

Yes — gdal.Translate with the COG driver, or gdal:translate through Processing with the appropriate creation options. Write it once at the end of a pipeline rather than trying to append to one: the format’s value comes from a carefully laid out header and overview pyramid, which is built at write time.