Asynchronous Task Execution with QgsTask

Learn how to implement thread-safe background processing in QGIS plugins using QgsTask — covering subclassing, data pre-fetching, cancellation, progress…

Long-running spatial operations routinely block the main event loop in desktop GIS applications, causing interface unresponsiveness, cursor spinners, and eventual operating system warnings. Within the QGIS ecosystem, QgsTask provides a standardized, thread-safe mechanism to offload heavy computation to background workers while preserving UI responsiveness. This topic is a cornerstone of Plugin Development & UI Integration, enabling developers to build professional-grade tools that scale with enterprise workloads without sacrificing user experience.

Unlike raw QThread implementations, QgsTask integrates directly with the QGIS task manager, handles thread pooling automatically, and enforces strict separation between background execution and main-thread UI updates. This article provides a production-tested workflow, complete code patterns, and troubleshooting strategies for implementing robust background processing in QGIS plugins and standalone scripts.

Prerequisites

Before implementing asynchronous workflows, ensure your development environment meets these baseline requirements:

  • QGIS 3.20+: Earlier versions contain known task manager memory leaks and incomplete signal routing.
  • Python 3.9+: The interpreter bundled with all currently supported QGIS releases. Required for modern type hints, stable exception handling, and concurrent.futures compatibility if bridging external libraries.
  • Qt signals and slots familiarity: Understanding how pyqtSignal routes across thread boundaries is mandatory — see Signal and Slot Event Handling in QGIS for a deep dive into event loop mechanics.
  • Basic plugin structure: Knowledge of initGui(), run(), and resource cleanup patterns from Plugin Lifecycle and Resource Management.

Developers should also review the official QgsTask API documentation to understand available methods, inheritance constraints, and thread-safety guarantees before writing production code.

Architecture and Internal Mechanics

The QGIS task system sits between user code and Qt’s thread pool. Understanding how requests flow through this stack is essential for avoiding the most common threading mistakes.

QgsTask Execution Lifecycle Diagram showing how a QgsTask flows from the main thread through the QGIS task manager and Qt thread pool, executes run() in a worker thread, then delivers finished() back to the main thread. State transitions — Running, Success, Failure, and Terminated — are labelled on the arrows. Main Thread Worker Thread (Qt Thread Pool) QgsTaskManager (task queue, thread pool allocation, state machine) Pre-fetch data layer.getFeatures() addTask(task) taskManager().addTask() finished(result) update UI / add layers run() → bool I/O, geometry ops, math State machine resolves outcome run()=True → taskCompleted run()=False → taskTerminated isCanceled() → taskTerminated poll isCanceled() in loop Thread boundary: main thread ↔ worker thread enforced by QgsTaskManager

The QGIS task architecture operates on a producer-consumer model. When you submit a task to QgsApplication.taskManager(), the manager assigns it to a worker thread from a pre-allocated pool. The critical rule governing this architecture is strict thread isolation:

  1. run() executes in a background thread. You may perform I/O, heavy Python math, and read pre-fetched data structures here. Never call QgsProject.instance(), iface, or any Qt widget from this method.
  2. finished() executes in the main thread. This is the only safe location to update UI elements, add layers to QgsProject, modify the map canvas, or trigger layer reloads.
  3. Signals emitted from run() are queued automatically if connected with Qt.QueuedConnection (the default for cross-thread connections), but direct UI manipulation remains prohibited regardless.

Attempting to bypass this boundary causes QObject::moveToThread warnings, segmentation faults, or silent data corruption. The correct pattern is to pre-fetch all required data on the main thread before submitting the task, pass it into the task constructor, and only write results back to the project in finished().

For workflows requiring complex parameter validation, standardized output handling, and automatic batch execution, consider Building Custom Processing Algorithms instead, as the Processing Framework abstracts much of the threading complexity while providing built-in progress tracking and history management.

Step-by-Step Implementation

1. Subclass QgsTask

The most reliable approach for custom geoprocessing is subclassing QgsTask. This gives you explicit control over execution flow, cancellation handling, and progress reporting.

The example below processes a list of geometries that were fetched on the main thread before the task was submitted. Fetching them inside run() via QgsProject or a QgsVectorLayer would violate thread safety. Careful memory ownership here aligns with the broader memory management and garbage collection rules that apply to all PyQGIS objects.

python
from qgis.core import QgsTask, QgsMessageLog, Qgis, QgsGeometry
from qgis.PyQt.QtCore import pyqtSignal
from typing import Optional


class HeavyVectorProcessor(QgsTask):
    """Background task that processes pre-fetched geometries without blocking the UI.

    All geometry data must be pre-fetched and cloned on the main thread
    before this task is submitted to the task manager.
    """

    # Custom signal for thread-safe progress reporting — Qt marshals
    # arguments across the thread boundary automatically.
    progress_updated = pyqtSignal(int, str)

    def __init__(self, geometries: list[QgsGeometry], buffer_distance: float) -> None:
        """
        Parameters
        ----------
        geometries : list[QgsGeometry]
            Pre-fetched and cloned on the main thread before task submission.
        buffer_distance : float
            Buffer distance in layer CRS units.
        """
        super().__init__(
            f"Processing {len(geometries)} geometries",
            QgsTask.CanCancel,
        )
        # Clone each geometry so this task owns its own copies independently
        # of any QgsVectorLayer that might be edited while the task runs.
        self.geometries: list[QgsGeometry] = [g.clone() for g in geometries]
        self.buffer_distance = buffer_distance
        self.processed_count: int = 0
        self.result_geometries: list[QgsGeometry] = []
        self.error_msg: Optional[str] = None

    def run(self) -> bool:
        """Execute in a background thread. Return True on success, False on failure.

        Never access QgsProject, iface, or any Qt widget from this method.
        All data exchange with the main thread goes through instance variables
        read in finished(), or via queued signals.
        """
        try:
            total = len(self.geometries)
            if total == 0:
                self.error_msg = "No geometries to process."
                return False

            for i, geom in enumerate(self.geometries):
                # Check cancellation at the start of each iteration
                if self.isCanceled():
                    return False

                # Geometry operations are safe in a worker thread — they do
                # not touch the Qt object tree or any QGIS project state.
                buffered = geom.buffer(self.buffer_distance, 8)
                self.result_geometries.append(buffered)
                self.processed_count += 1

                progress_pct = int((i + 1) / total * 100)
                self.setProgress(progress_pct)
                self.progress_updated.emit(progress_pct, f"Processing {i + 1}/{total}")

            return True

        except Exception as exc:
            self.error_msg = str(exc)
            QgsMessageLog.logMessage(
                f"Task failed: {exc}",
                "HeavyVectorProcessor",
                level=Qgis.Critical,
            )
            return False

    def finished(self, result: bool) -> None:
        """Execute on the main thread after run() completes.

        Safe to call iface, QgsProject.instance().addMapLayer(), or any
        Qt widget method from here.
        """
        if result:
            QgsMessageLog.logMessage(
                f"Completed — processed {self.processed_count} geometries.",
                "HeavyVectorProcessor",
                level=Qgis.Success,
            )
            # Add results to canvas, refresh symbology, or open a results dialog
        elif self.isCanceled():
            QgsMessageLog.logMessage(
                "Task cancelled by user.",
                "HeavyVectorProcessor",
                level=Qgis.Warning,
            )
        else:
            QgsMessageLog.logMessage(
                f"Task failed: {self.error_msg}",
                "HeavyVectorProcessor",
                level=Qgis.Critical,
            )

2. Pre-Fetch Data and Register with the Task Manager

Fetch all required data on the main thread before task submission. Once submitted, the task manager handles thread allocation and lifecycle. Note how vector and raster data access patterns — specifically the use of layer.getFeatures() and geometry cloning — are applied here to guarantee thread safety.

python
from qgis.core import (
    QgsApplication,
    QgsProject,
    QgsMessageLog,
    Qgis,
)
from qgis.PyQt.QtCore import Qt


def execute_background_task(layer_id: str, distance: float) -> None:
    """Create and submit a HeavyVectorProcessor task.

    Parameters
    ----------
    layer_id : str
        The map layer ID as returned by QgsMapLayer.id().
    distance : float
        Buffer distance in layer CRS units.
    """
    # --- Main thread: read all data before task submission ---
    layer = QgsProject.instance().mapLayer(layer_id)
    if not layer or not layer.isValid():
        QgsMessageLog.logMessage(
            "Layer not found or invalid.",
            "HeavyVectorProcessor",
            level=Qgis.Warning,
        )
        return

    # Iterate features and clone geometries on the main thread.
    # Do NOT pass the QgsVectorLayer itself — the task must not
    # call layer methods from the worker thread.
    geometries = [f.geometry() for f in layer.getFeatures()]

    # --- Create and wire the task ---
    task = HeavyVectorProcessor(geometries, distance)

    # Connect signals before addTask() — the task may start immediately.
    task.progress_updated.connect(
        lambda pct, msg: update_progress_bar(pct, msg),
        Qt.QueuedConnection,
    )
    task.taskCompleted.connect(lambda: handle_success(task))
    task.taskTerminated.connect(lambda: handle_failure(task))

    QgsApplication.taskManager().addTask(task)
    QgsMessageLog.logMessage(
        f"Submitted task for {len(geometries)} geometries.",
        "HeavyVectorProcessor",
    )

3. Handle Results and UI Updates

When the task finishes, finished() runs on the main thread. This is where you safely interact with Qt widgets. If your plugin requires complex modal dialogs or dynamic form generation to display results, follow established patterns for designing Qt dialogs and form widgets to ensure proper parent-child ownership and memory management.

python
from qgis.core import QgsVectorLayer, QgsProject, QgsWkbTypes, QgsFeature
from qgis.PyQt.QtWidgets import QMessageBox


def update_progress_bar(percent: int, message: str) -> None:
    """Main-thread slot for per-iteration progress updates.

    Throttle to ~10-20 Hz if the dataset is very large; emitting
    signals at thousands of Hz will saturate the event queue.
    """
    # iface.mainWindow().statusBar().showMessage(f"{message} ({percent}%)")
    pass


def handle_success(task: HeavyVectorProcessor) -> None:
    """Build a memory layer from task results and add it to the project."""
    result_layer = QgsVectorLayer(
        "Polygon?crs=EPSG:4326",
        "Buffered Geometries",
        "memory",
    )
    result_layer.startEditing()
    for geom in task.result_geometries:
        feat = QgsFeature()
        feat.setGeometry(geom)
        result_layer.addFeature(feat)
    result_layer.commitChanges()
    QgsProject.instance().addMapLayer(result_layer)


def handle_failure(task: HeavyVectorProcessor) -> None:
    """Notify the user when a task ends without producing results."""
    if task.isCanceled():
        return  # User-initiated; no error dialog needed
    QMessageBox.critical(
        None,
        "Processing error",
        f"Task failed: {task.error_msg}",
    )

Advanced Patterns

Chunked Processing and Task Dependencies

For enterprise-scale workloads, split massive datasets into manageable batches and submit multiple QgsTask instances with different feature ranges. The task manager’s thread pool will execute them in parallel, maximizing CPU utilisation without saturating memory.

Use QgsTask.addSubTask() to chain tasks into dependency graphs. A sub-task starts only after its parent completes, making it straightforward to build ETL pipelines where a reprojection step must finish before a spatial join begins.

python
from qgis.core import QgsTask, QgsApplication
from typing import Optional


class ReprojectionTask(QgsTask):
    """Reproject geometries to a target CRS in a worker thread."""

    def __init__(self, geometries: list, target_crs_auth_id: str) -> None:
        super().__init__("Reprojecting features", QgsTask.CanCancel)
        self.geometries = [g.clone() for g in geometries]
        self.target_crs_auth_id = target_crs_auth_id
        self.reprojected: list = []

    def run(self) -> bool:
        from qgis.core import QgsCoordinateTransform, QgsCoordinateReferenceSystem, QgsProject
        # Note: QgsCoordinateTransform is safe to construct in a worker thread
        # because it reads CRS definitions from files, not from QgsProject state.
        target_crs = QgsCoordinateReferenceSystem(self.target_crs_auth_id)
        for geom in self.geometries:
            if self.isCanceled():
                return False
            self.reprojected.append(geom.clone())
        return True

    def finished(self, result: bool) -> None:
        if result:
            # Coordinate transformations and CRS handling are covered in
            # /pyqgis-core-architecture-data-handling/coordinate-transformations-and-crs-handling/
            pass


class SpatialJoinTask(QgsTask):
    """Perform a spatial join once reprojection completes."""

    def __init__(self, reproject_task: ReprojectionTask) -> None:
        super().__init__("Spatial join", QgsTask.CanCancel)
        # Store a reference so finished() can read reprojected geometries
        self.reproject_task = reproject_task

    def run(self) -> bool:
        geometries = self.reproject_task.reprojected
        # Heavy spatial join logic here …
        return True

    def finished(self, result: bool) -> None:
        pass


def run_pipeline(geometries: list) -> None:
    """Submit a two-stage reprojection → join pipeline."""
    reproject = ReprojectionTask(geometries, "EPSG:32632")
    join = SpatialJoinTask(reproject)

    # addSubTask ensures join only starts after reproject completes
    reproject.addSubTask(join, [], QgsTask.ParentDependsOnSubTask)
    QgsApplication.taskManager().addTask(reproject)

Quick Functions with QgsTask.fromFunction

For lightweight, one-off operations, skip subclassing and wrap a callable directly. The trade-off is coarser progress control and no custom signals — suitable for short operations under a few seconds.

python
from qgis.core import QgsApplication, QgsTask
from typing import Optional


def _compute_area_sum(task: QgsTask, geometries: list) -> Optional[dict]:
    """Compute the total area of a geometry list in a worker thread.

    Parameters
    ----------
    task : QgsTask
        The hosting task object — use task.isCanceled() to check for cancellation.
    geometries : list[QgsGeometry]
        Pre-cloned geometries passed in from the main thread.
    """
    total = 0.0
    for i, geom in enumerate(geometries):
        if task.isCanceled():
            return None
        total += geom.area()
        task.setProgress(int((i + 1) / len(geometries) * 100))
    return {"total_area": total, "count": len(geometries)}


def _on_area_complete(exception: Optional[Exception], result: Optional[dict] = None) -> None:
    """Receive the result dict (or exception) on the main thread."""
    if exception:
        raise exception
    if result:
        print(f"Total area: {result['total_area']:.2f}, features: {result['count']}")


# geometries must be pre-fetched and cloned on the main thread
task = QgsTask.fromFunction(
    "Compute area sum",
    _compute_area_sum,
    on_finished=_on_area_complete,
    geometries=pre_fetched_geometries,
)
QgsApplication.taskManager().addTask(task)

Thread-Safe Communication Patterns

Directly calling QgsProject.instance().addMapLayer() or iface.mapCanvas().refresh() from run() will trigger Qt thread-safety violations. Use these proven patterns instead:

  • Custom signals: Define pyqtSignal in your task class and connect them in the main thread before submission. Qt automatically marshals arguments across thread boundaries using Qt.QueuedConnection.
  • State objects: Store intermediate results in instance variables during run(). Access them only in finished() where thread ownership is guaranteed.
  • Message logging: Use QgsMessageLog.logMessage() for background diagnostics. It is thread-safe and routes to the QGIS Log Messages panel without blocking.

Avoid sharing mutable Python objects (lists, dicts) between the background and main threads without explicit locking. If you must pass complex structures, serialize them to JSON or use thread-safe queues from the queue module.

Pitfalls and Debugging

  • QObject::moveToThread warningQgsProject, iface, or a widget was accessed inside run(). Move all project/layer reads to before task submission and access results in finished() only.
  • Task disappears from manager — An unhandled exception in run() causes silent termination. Wrap all logic in try/except, log errors, and return False explicitly.
  • Progress bar jumps or freezes — Emitting signals too frequently saturates the event queue. Throttle progress_updated emissions to at most one call per feature batch or time-gate them to 50 ms intervals.
  • Memory grows over time — Unclosed file handles, database cursors, or undisconnected signals accumulate across task submissions. Disconnect signals in finished() and use context managers for external resources.
  • Stale layer data in results — If a layer is edited after geometries are fetched but before the task completes, the task operates on stale copies. Clone data immediately at fetch time, before submitting, and refuse submission if the layer is in edit mode.
  • addSubTask not starting — Sub-tasks submitted before the parent addTask() call will not be registered correctly. Always call addSubTask() before taskManager().addTask().
  • fromFunction silently swallows exceptions — Unless you inspect the exception argument in on_finished, errors disappear. Always check and re-raise.
Symptom Likely Cause Resolution
QObject::moveToThread warning Project or widget accessed in run() Pre-fetch on main thread; write only in finished()
Task absent from manager Unhandled exception in run() Wrap in try/except, log, return False
Progress bar freezes Signal emission too frequent Throttle to ~10–20 Hz or batch progress updates
Memory leak in long session Signal not disconnected Disconnect in finished(), use context managers
Stale geometry in output Layer edited after fetch Clone at fetch time; reject if layer is in edit mode

Conclusion

QgsTask transforms brittle, UI-blocking scripts into resilient, production-ready QGIS plugins. By respecting thread boundaries — pre-fetching data on the main thread, processing it in run(), and updating the UI in finished() — developers can build tools that handle enterprise workloads without compromising user experience. Start with simple background operations, validate thread safety rigorously, and scale to complex dependency graphs using addSubTask() as your pipelines mature.

For the specific problem of large-scale raster and vector pipelines that must not freeze the QGIS interface, see running heavy geoprocessing in the background without freezing the UI, which covers memory profiling, chunking strategies, and fallback mechanisms for legacy systems.


Related