PyQGIS Engineering & Automation

Profiling PyQGIS Memory with tracemalloc and RSS

Find out where PyQGIS memory actually goes: process resident set size as the only complete number, tracemalloc for the Python half, the GDAL block cache…

TL;DR: measure process resident set size first, because a PyQGIS leak is usually native and invisible to a Python profiler — then use tracemalloc to attribute the Python half and the GDAL cache report to bound the raster half. This page is part of the memory management and garbage collection for GIS objects guide.

Complete Runnable Code

python
"""A small memory probe for PyQGIS batch loops."""
import gc
import os
import tracemalloc
from contextlib import contextmanager

try:
    from osgeo import gdal
except ImportError:                      # GDAL is optional for the Python-only half
    gdal = None


def rss_bytes() -> int:
    """Resident set size of this process, in bytes. Linux; falls back to 0 elsewhere."""
    try:
        with open("/proc/self/statm", encoding="ascii") as handle:
            pages = int(handle.read().split()[1])
        return pages * os.sysconf("SC_PAGE_SIZE")
    except (OSError, IndexError, ValueError):
        return 0


@contextmanager
def memory_probe(label: str, top: int = 5):
    """Report RSS delta, Python allocation delta and the top Python allocation sites."""
    gc.collect()
    tracemalloc.start()
    before_rss = rss_bytes()
    before_snapshot = tracemalloc.take_snapshot()
    try:
        yield
    finally:
        gc.collect()
        after_snapshot = tracemalloc.take_snapshot()
        after_rss = rss_bytes()

        print("%s: RSS %+.1f MB" % (label, (after_rss - before_rss) / 1e6))
        stats = after_snapshot.compare_to(before_snapshot, "lineno")
        python_delta = sum(s.size_diff for s in stats)
        print("%s: Python %+.1f MB across %d site(s)"
              % (label, python_delta / 1e6, len(stats)))
        for stat in stats[:top]:
            print("    %+.1f MB  %s" % (stat.size_diff / 1e6, stat))
        if gdal is not None:
            print("%s: GDAL cache %.1f MB of %.1f MB"
                  % (label, gdal.GetCacheUsed() / 1e6, gdal.GetCacheMax() / 1e6))
        tracemalloc.stop()

Wrap any suspicious loop in it and the three numbers together tell you which half of the process is growing:

python
with memory_probe("parcel pass"):
    for feature in layer.getFeatures():
        process(feature)
What each measurement can actually see Three bands showing measurement coverage: tracemalloc sees Python allocations only, resident set size sees the whole process including native heaps, and the GDAL cache report sees the block cache specifically. tracemalloc Python objects by allocation site blind to C++ the big half RSS everything resident Python and native no attribution just a number GDAL report block cache a bounded slice per process settable ceiling A PyQGIS leak is usually native, which is why tracemalloc alone so often reports that nothing is wrong.

Architecture Breakdown

Resident set size is the only complete number

RSS is what the operating system says the process is holding. It includes the Python heap, every C++ allocation QGIS made, GDAL’s block cache and the memory-mapped regions of open files. It is the number that decides whether the job survives, and it is the only one that sees all of them.

Its weakness is the mirror image: it offers no attribution at all. A rising RSS tells you there is a problem and nothing about where, which is why it is a first measurement rather than a diagnosis.

tracemalloc sees Python and nothing else

tracemalloc traces allocations made through Python’s allocator, attributing each to the line that made it. For a leak caused by accumulating QgsFeature wrappers in a list, it points straight at the line. For a leak in the C++ geometry those wrappers point to, it reports a few hundred bytes per feature and misses the megabytes.

Where the memory went in one raster loop A bar chart attributing peak memory in a raster processing loop: Python objects, the GDAL block cache, geometry held in a result list, and the rest of the process. Python objects 42 MB — what tracemalloc sees GDAL block cache 780 MB retained geometry 310 MB base process 190 MB Indicative attribution for a loop over a large GeoTIFF. Only the first bar is visible to a Python profiler, which is why the first measurement to take is process RSS.

That gap is the single most important thing to understand about profiling PyQGIS. The first bar is what a Python profiler can see; the other three are the ones that actually exhaust the machine.

The GDAL cache is a bounded, separate pool

GDAL keeps decompressed raster blocks in a global cache that defaults to a percentage of system RAM — often several gigabytes. It is not a leak: the cache is doing its job, and it will release memory under pressure. But it makes RSS readings confusing, and in a container with a hard memory limit it can trigger the out-of-memory killer before it ever notices pressure.

Setting gdal.SetCacheMax() explicitly at the start of a batch job turns an unpredictable number into a known one, which makes every other measurement easier to interpret.

Reading the Three Numbers Together

Taken separately the three measurements are ambiguous; taken together they narrow the cause to one of three, and each has a different fix.

Which tool to reach for next A decision tree driven by what the process resident size is doing: flat means look at Python, growing with rasters means the block cache, and growing with vectors means retained wrappers. What is process RSS doing? flat tracemalloc a Python-side leak grows on rasters GDAL cache report cap it grows on vectors count wrappers retained features

A useful habit is to record all three at the end of every batch run rather than only when investigating. A single reading tells you almost nothing, because absolute memory use varies with platform, allocator and what the machine was doing beforehand. A series of readings across identical runs tells you whether the number is stable, which is the only question that matters.

Flat RSS with growing Python allocations is an ordinary Python leak, and tracemalloc will name the line. Growing RSS with flat Python allocations is the PyQGIS-specific case: something native is being retained, most often because a Python wrapper is keeping a C++ object alive, or because the GDAL cache is filling.

The distinction between those two native causes is what the third number settles. If the cache report accounts for the growth, cap the cache. If it does not, you are retaining wrappers — a list of features, a dictionary of geometries, or a signal connection holding a layer.

Turning a Probe into a Regression Test

Once a leak is fixed, a bounded assertion stops it coming back:

python
def test_batch_does_not_grow(tmp_path):
    """The same work repeated must not grow the process without bound."""
    baseline = None
    for iteration in range(5):
        run_one_batch(tmp_path)
        gc.collect()
        current = rss_bytes()
        if iteration == 1:               # ignore the first, which warms caches
            baseline = current
        elif baseline is not None:
            assert current - baseline < 50 * 1024 * 1024, "RSS grew by more than 50 MB"

Skipping the first iteration matters: caches fill, providers initialise, and the first pass is always more expensive than the steady state. Asserting against the second iteration measures growth rather than start-up.

Production Best Practices

  • Measure RSS first. It is the only number that sees everything.
  • Cap the GDAL cache in any batch job, so the largest pool is a known size.
  • Call gc.collect() before each measurement, or you are measuring collection timing rather than retention.
  • Attribute with tracemalloc only after RSS says the Python side is involved.
  • Watch for retained wrappers — a list of features is a list of C++ geometries.
  • Test for growth across iterations, not absolute size; absolute numbers vary by platform and make brittle assertions.

Frequently Asked Questions

Why does tracemalloc say nothing is leaking?

Because the leak is on the C++ side. A QgsFeature wrapper is a small Python object pointing at a much larger native allocation, so a list of a million features looks like a few tens of megabytes to tracemalloc and like a gigabyte to the operating system.

When RSS grows and tracemalloc does not, look for what those small Python objects are keeping alive rather than for the objects themselves.

Does gc.collect() actually free anything?

It collects Python objects in reference cycles, which is worth doing before a measurement so you are not counting garbage that was about to go anyway. What it cannot do is free a C++ object whose owner still holds it — a layer in the project registry stays until it is removed, however many times you collect.

In tight loops that never yield to the event loop, an occasional explicit collect is also worthwhile, because the automatic threshold may not trigger for thousands of objects.

How do I measure RSS on Windows or macOS?

psutil.Process().memory_info().rss works everywhere and is worth the dependency for a diagnostic tool. The /proc reading above avoids the dependency on Linux, which suits code you want to ship inside a plugin without adding a requirement.

Is a growing RSS always a leak?

No. Allocators do not return freed memory to the operating system immediately, and fragmentation means a process that has peaked at two gigabytes may hold that much resident afterwards even though it is mostly free. What matters is whether the peak grows across repeated identical work.

That is exactly what the iteration test above measures, and it is a far more reliable signal than a single reading.

Should I profile inside QGIS Desktop or a standalone script?

Standalone, wherever possible. The desktop holds caches, layers and a rendering pipeline that vary between runs, which makes small differences impossible to see. A standalone script doing exactly the work you care about gives you a quiet baseline.

What about memory used by a background task?

Threads share the process address space, so a task’s allocations show up in the same RSS. That makes attribution harder, not easier: a growing number with several tasks running says nothing about which one is responsible. Profile one task at a time, in isolation, before profiling them together.