Asynchronous Task Execution with QgsTask
Learn how to implement thread-safe background processing in QGIS plugins using QgsTask — covering subclassing, data pre-fetching, cancellation, progress…
Long-running spatial operations routinely block the main event loop in desktop GIS applications, causing interface unresponsiveness, cursor spinners, and eventual operating system warnings. Within the QGIS ecosystem, QgsTask provides a standardized, thread-safe mechanism to offload heavy computation to background workers while preserving UI responsiveness. This topic is a cornerstone of Plugin Development & UI Integration, enabling developers to build professional-grade tools that scale with enterprise workloads without sacrificing user experience.
Unlike raw QThread implementations, QgsTask integrates directly with the QGIS task manager, handles thread pooling automatically, and enforces strict separation between background execution and main-thread UI updates. This article provides a production-tested workflow, complete code patterns, and troubleshooting strategies for implementing robust background processing in QGIS plugins and standalone scripts.
Prerequisites
Before implementing asynchronous workflows, ensure your development environment meets these baseline requirements:
- QGIS 3.20+: Earlier versions contain known task manager memory leaks and incomplete signal routing.
- Python 3.9+: The interpreter bundled with all currently supported QGIS releases. Required for modern type hints, stable exception handling, and
concurrent.futurescompatibility if bridging external libraries. - Qt signals and slots familiarity: Understanding how
pyqtSignalroutes across thread boundaries is mandatory — see Signal and Slot Event Handling in QGIS for a deep dive into event loop mechanics. - Basic plugin structure: Knowledge of
initGui(),run(), and resource cleanup patterns from Plugin Lifecycle and Resource Management.
Developers should also review the official QgsTask API documentation to understand available methods, inheritance constraints, and thread-safety guarantees before writing production code.
Architecture and Internal Mechanics
The QGIS task system sits between user code and Qt’s thread pool. Understanding how requests flow through this stack is essential for avoiding the most common threading mistakes.
The QGIS task architecture operates on a producer-consumer model. When you submit a task to QgsApplication.taskManager(), the manager assigns it to a worker thread from a pre-allocated pool. The critical rule governing this architecture is strict thread isolation:
run()executes in a background thread. You may perform I/O, heavy Python math, and read pre-fetched data structures here. Never callQgsProject.instance(),iface, or any Qt widget from this method.finished()executes in the main thread. This is the only safe location to update UI elements, add layers to QgsProject, modify the map canvas, or trigger layer reloads.- Signals emitted from
run()are queued automatically if connected withQt.QueuedConnection(the default for cross-thread connections), but direct UI manipulation remains prohibited regardless.
Attempting to bypass this boundary causes QObject::moveToThread warnings, segmentation faults, or silent data corruption. The correct pattern is to pre-fetch all required data on the main thread before submitting the task, pass it into the task constructor, and only write results back to the project in finished().
For workflows requiring complex parameter validation, standardized output handling, and automatic batch execution, consider Building Custom Processing Algorithms instead, as the Processing Framework abstracts much of the threading complexity while providing built-in progress tracking and history management.
Step-by-Step Implementation
1. Subclass QgsTask
The most reliable approach for custom geoprocessing is subclassing QgsTask. This gives you explicit control over execution flow, cancellation handling, and progress reporting.
The example below processes a list of geometries that were fetched on the main thread before the task was submitted. Fetching them inside run() via QgsProject or a QgsVectorLayer would violate thread safety. Careful memory ownership here aligns with the broader memory management and garbage collection rules that apply to all PyQGIS objects.
from qgis.core import QgsTask, QgsMessageLog, Qgis, QgsGeometry
from qgis.PyQt.QtCore import pyqtSignal
from typing import Optional
class HeavyVectorProcessor(QgsTask):
"""Background task that processes pre-fetched geometries without blocking the UI.
All geometry data must be pre-fetched and cloned on the main thread
before this task is submitted to the task manager.
"""
# Custom signal for thread-safe progress reporting — Qt marshals
# arguments across the thread boundary automatically.
progress_updated = pyqtSignal(int, str)
def __init__(self, geometries: list[QgsGeometry], buffer_distance: float) -> None:
"""
Parameters
----------
geometries : list[QgsGeometry]
Pre-fetched and cloned on the main thread before task submission.
buffer_distance : float
Buffer distance in layer CRS units.
"""
super().__init__(
f"Processing {len(geometries)} geometries",
QgsTask.CanCancel,
)
# Clone each geometry so this task owns its own copies independently
# of any QgsVectorLayer that might be edited while the task runs.
self.geometries: list[QgsGeometry] = [g.clone() for g in geometries]
self.buffer_distance = buffer_distance
self.processed_count: int = 0
self.result_geometries: list[QgsGeometry] = []
self.error_msg: Optional[str] = None
def run(self) -> bool:
"""Execute in a background thread. Return True on success, False on failure.
Never access QgsProject, iface, or any Qt widget from this method.
All data exchange with the main thread goes through instance variables
read in finished(), or via queued signals.
"""
try:
total = len(self.geometries)
if total == 0:
self.error_msg = "No geometries to process."
return False
for i, geom in enumerate(self.geometries):
# Check cancellation at the start of each iteration
if self.isCanceled():
return False
# Geometry operations are safe in a worker thread — they do
# not touch the Qt object tree or any QGIS project state.
buffered = geom.buffer(self.buffer_distance, 8)
self.result_geometries.append(buffered)
self.processed_count += 1
progress_pct = int((i + 1) / total * 100)
self.setProgress(progress_pct)
self.progress_updated.emit(progress_pct, f"Processing {i + 1}/{total}")
return True
except Exception as exc:
self.error_msg = str(exc)
QgsMessageLog.logMessage(
f"Task failed: {exc}",
"HeavyVectorProcessor",
level=Qgis.Critical,
)
return False
def finished(self, result: bool) -> None:
"""Execute on the main thread after run() completes.
Safe to call iface, QgsProject.instance().addMapLayer(), or any
Qt widget method from here.
"""
if result:
QgsMessageLog.logMessage(
f"Completed — processed {self.processed_count} geometries.",
"HeavyVectorProcessor",
level=Qgis.Success,
)
# Add results to canvas, refresh symbology, or open a results dialog
elif self.isCanceled():
QgsMessageLog.logMessage(
"Task cancelled by user.",
"HeavyVectorProcessor",
level=Qgis.Warning,
)
else:
QgsMessageLog.logMessage(
f"Task failed: {self.error_msg}",
"HeavyVectorProcessor",
level=Qgis.Critical,
)
2. Pre-Fetch Data and Register with the Task Manager
Fetch all required data on the main thread before task submission. Once submitted, the task manager handles thread allocation and lifecycle. Note how vector and raster data access patterns — specifically the use of layer.getFeatures() and geometry cloning — are applied here to guarantee thread safety.
from qgis.core import (
QgsApplication,
QgsProject,
QgsMessageLog,
Qgis,
)
from qgis.PyQt.QtCore import Qt
def execute_background_task(layer_id: str, distance: float) -> None:
"""Create and submit a HeavyVectorProcessor task.
Parameters
----------
layer_id : str
The map layer ID as returned by QgsMapLayer.id().
distance : float
Buffer distance in layer CRS units.
"""
# --- Main thread: read all data before task submission ---
layer = QgsProject.instance().mapLayer(layer_id)
if not layer or not layer.isValid():
QgsMessageLog.logMessage(
"Layer not found or invalid.",
"HeavyVectorProcessor",
level=Qgis.Warning,
)
return
# Iterate features and clone geometries on the main thread.
# Do NOT pass the QgsVectorLayer itself — the task must not
# call layer methods from the worker thread.
geometries = [f.geometry() for f in layer.getFeatures()]
# --- Create and wire the task ---
task = HeavyVectorProcessor(geometries, distance)
# Connect signals before addTask() — the task may start immediately.
task.progress_updated.connect(
lambda pct, msg: update_progress_bar(pct, msg),
Qt.QueuedConnection,
)
task.taskCompleted.connect(lambda: handle_success(task))
task.taskTerminated.connect(lambda: handle_failure(task))
QgsApplication.taskManager().addTask(task)
QgsMessageLog.logMessage(
f"Submitted task for {len(geometries)} geometries.",
"HeavyVectorProcessor",
)
3. Handle Results and UI Updates
When the task finishes, finished() runs on the main thread. This is where you safely interact with Qt widgets. If your plugin requires complex modal dialogs or dynamic form generation to display results, follow established patterns for designing Qt dialogs and form widgets to ensure proper parent-child ownership and memory management.
from qgis.core import QgsVectorLayer, QgsProject, QgsWkbTypes, QgsFeature
from qgis.PyQt.QtWidgets import QMessageBox
def update_progress_bar(percent: int, message: str) -> None:
"""Main-thread slot for per-iteration progress updates.
Throttle to ~10-20 Hz if the dataset is very large; emitting
signals at thousands of Hz will saturate the event queue.
"""
# iface.mainWindow().statusBar().showMessage(f"{message} ({percent}%)")
pass
def handle_success(task: HeavyVectorProcessor) -> None:
"""Build a memory layer from task results and add it to the project."""
result_layer = QgsVectorLayer(
"Polygon?crs=EPSG:4326",
"Buffered Geometries",
"memory",
)
result_layer.startEditing()
for geom in task.result_geometries:
feat = QgsFeature()
feat.setGeometry(geom)
result_layer.addFeature(feat)
result_layer.commitChanges()
QgsProject.instance().addMapLayer(result_layer)
def handle_failure(task: HeavyVectorProcessor) -> None:
"""Notify the user when a task ends without producing results."""
if task.isCanceled():
return # User-initiated; no error dialog needed
QMessageBox.critical(
None,
"Processing error",
f"Task failed: {task.error_msg}",
)
Advanced Patterns
Chunked Processing and Task Dependencies
For enterprise-scale workloads, split massive datasets into manageable batches and submit multiple QgsTask instances with different feature ranges. The task manager’s thread pool will execute them in parallel, maximizing CPU utilisation without saturating memory.
Use QgsTask.addSubTask() to chain tasks into dependency graphs. A sub-task starts only after its parent completes, making it straightforward to build ETL pipelines where a reprojection step must finish before a spatial join begins.
from qgis.core import QgsTask, QgsApplication
from typing import Optional
class ReprojectionTask(QgsTask):
"""Reproject geometries to a target CRS in a worker thread."""
def __init__(self, geometries: list, target_crs_auth_id: str) -> None:
super().__init__("Reprojecting features", QgsTask.CanCancel)
self.geometries = [g.clone() for g in geometries]
self.target_crs_auth_id = target_crs_auth_id
self.reprojected: list = []
def run(self) -> bool:
from qgis.core import QgsCoordinateTransform, QgsCoordinateReferenceSystem, QgsProject
# Note: QgsCoordinateTransform is safe to construct in a worker thread
# because it reads CRS definitions from files, not from QgsProject state.
target_crs = QgsCoordinateReferenceSystem(self.target_crs_auth_id)
for geom in self.geometries:
if self.isCanceled():
return False
self.reprojected.append(geom.clone())
return True
def finished(self, result: bool) -> None:
if result:
# Coordinate transformations and CRS handling are covered in
# /pyqgis-core-architecture-data-handling/coordinate-transformations-and-crs-handling/
pass
class SpatialJoinTask(QgsTask):
"""Perform a spatial join once reprojection completes."""
def __init__(self, reproject_task: ReprojectionTask) -> None:
super().__init__("Spatial join", QgsTask.CanCancel)
# Store a reference so finished() can read reprojected geometries
self.reproject_task = reproject_task
def run(self) -> bool:
geometries = self.reproject_task.reprojected
# Heavy spatial join logic here …
return True
def finished(self, result: bool) -> None:
pass
def run_pipeline(geometries: list) -> None:
"""Submit a two-stage reprojection → join pipeline."""
reproject = ReprojectionTask(geometries, "EPSG:32632")
join = SpatialJoinTask(reproject)
# addSubTask ensures join only starts after reproject completes
reproject.addSubTask(join, [], QgsTask.ParentDependsOnSubTask)
QgsApplication.taskManager().addTask(reproject)
Quick Functions with QgsTask.fromFunction
For lightweight, one-off operations, skip subclassing and wrap a callable directly. The trade-off is coarser progress control and no custom signals — suitable for short operations under a few seconds.
from qgis.core import QgsApplication, QgsTask
from typing import Optional
def _compute_area_sum(task: QgsTask, geometries: list) -> Optional[dict]:
"""Compute the total area of a geometry list in a worker thread.
Parameters
----------
task : QgsTask
The hosting task object — use task.isCanceled() to check for cancellation.
geometries : list[QgsGeometry]
Pre-cloned geometries passed in from the main thread.
"""
total = 0.0
for i, geom in enumerate(geometries):
if task.isCanceled():
return None
total += geom.area()
task.setProgress(int((i + 1) / len(geometries) * 100))
return {"total_area": total, "count": len(geometries)}
def _on_area_complete(exception: Optional[Exception], result: Optional[dict] = None) -> None:
"""Receive the result dict (or exception) on the main thread."""
if exception:
raise exception
if result:
print(f"Total area: {result['total_area']:.2f}, features: {result['count']}")
# geometries must be pre-fetched and cloned on the main thread
task = QgsTask.fromFunction(
"Compute area sum",
_compute_area_sum,
on_finished=_on_area_complete,
geometries=pre_fetched_geometries,
)
QgsApplication.taskManager().addTask(task)
Thread-Safe Communication Patterns
Directly calling QgsProject.instance().addMapLayer() or iface.mapCanvas().refresh() from run() will trigger Qt thread-safety violations. Use these proven patterns instead:
- Custom signals: Define
pyqtSignalin your task class and connect them in the main thread before submission. Qt automatically marshals arguments across thread boundaries usingQt.QueuedConnection. - State objects: Store intermediate results in instance variables during
run(). Access them only infinished()where thread ownership is guaranteed. - Message logging: Use
QgsMessageLog.logMessage()for background diagnostics. It is thread-safe and routes to the QGIS Log Messages panel without blocking.
Avoid sharing mutable Python objects (lists, dicts) between the background and main threads without explicit locking. If you must pass complex structures, serialize them to JSON or use thread-safe queues from the queue module.
Pitfalls and Debugging
QObject::moveToThreadwarning —QgsProject,iface, or a widget was accessed insiderun(). Move all project/layer reads to before task submission and access results infinished()only.- Task disappears from manager — An unhandled exception in
run()causes silent termination. Wrap all logic intry/except, log errors, and returnFalseexplicitly. - Progress bar jumps or freezes — Emitting signals too frequently saturates the event queue. Throttle
progress_updatedemissions to at most one call per feature batch or time-gate them to 50 ms intervals. - Memory grows over time — Unclosed file handles, database cursors, or undisconnected signals accumulate across task submissions. Disconnect signals in
finished()and use context managers for external resources. - Stale layer data in results — If a layer is edited after geometries are fetched but before the task completes, the task operates on stale copies. Clone data immediately at fetch time, before submitting, and refuse submission if the layer is in edit mode.
addSubTasknot starting — Sub-tasks submitted before the parentaddTask()call will not be registered correctly. Always calladdSubTask()beforetaskManager().addTask().fromFunctionsilently swallows exceptions — Unless you inspect theexceptionargument inon_finished, errors disappear. Always check and re-raise.
| Symptom | Likely Cause | Resolution |
|---|---|---|
QObject::moveToThread warning | Project or widget accessed in run() | Pre-fetch on main thread; write only in finished() |
| Task absent from manager | Unhandled exception in run() | Wrap in try/except, log, return False |
| Progress bar freezes | Signal emission too frequent | Throttle to ~10–20 Hz or batch progress updates |
| Memory leak in long session | Signal not disconnected | Disconnect in finished(), use context managers |
| Stale geometry in output | Layer edited after fetch | Clone at fetch time; reject if layer is in edit mode |
Conclusion
QgsTask transforms brittle, UI-blocking scripts into resilient, production-ready QGIS plugins. By respecting thread boundaries — pre-fetching data on the main thread, processing it in run(), and updating the UI in finished() — developers can build tools that handle enterprise workloads without compromising user experience. Start with simple background operations, validate thread safety rigorously, and scale to complex dependency graphs using addSubTask() as your pipelines mature.
For the specific problem of large-scale raster and vector pipelines that must not freeze the QGIS interface, see running heavy geoprocessing in the background without freezing the UI, which covers memory profiling, chunking strategies, and fallback mechanisms for legacy systems.
Related
- Plugin Development & UI Integration — parent guide covering the full architecture of QGIS plugin development
- Building Custom Processing Algorithms — use the Processing Framework when you need standardised progress tracking, history, and batch execution
- Signal and Slot Event Handling in QGIS — understand
Qt.QueuedConnectionand cross-thread signal dispatch - Plugin Lifecycle and Resource Management — proper
unload()teardown to prevent signal leaks across task submissions - Memory Management and Garbage Collection for GIS Objects — SIP ownership rules that govern geometry cloning and task result objects