PyQGIS Engineering & Automation

Attribute Data and the Expression Engine in PyQGIS

Field schemas, QgsExpression and the expression context in PyQGIS: how to add fields, evaluate rules per feature, push filters into the provider, register…

Every non-trivial GIS automation eventually stops being about geometry and starts being about attributes: reclassifying a land-use code, deriving a density from two columns, selecting the subset of features a downstream process is allowed to touch. QGIS answers all three with the same machinery — a field schema on the layer and an expression engine that evaluates rules against it. This page, part of the PyQGIS Core Architecture & Data Handling guide, covers QgsField and schema changes, QgsExpression and the context that gives it meaning, the edit buffer that stands between a write and the disk, and the patterns that keep attribute work fast enough to run over millions of rows.

By the end you will know which of the four available write paths to use for a given job, why an expression that works in the field calculator returns NULL in your script, and how to make a per-feature rule run an order of magnitude faster without changing what it computes.

Prerequisites Checklist

  • QGIS 3.28 LTR or newer. QgsExpressionContextUtils and the scope helpers used below are stable from 3.10 onward, but the examples assume the current LTR API.
  • Python 3.9+ for the type hints in every sample.
  • A layer you can write to. GeoPackage is the easiest to experiment with; shapefile imposes a ten-character field-name limit that will confuse early experiments, as described in vector and raster data access patterns.
  • Familiarity with the edit buffer. Attribute writes through the layer are buffered, and the buffer is the source of most surprises in this area.
  • A test dataset with NULLs in it. Expression semantics around NULL are the single largest source of wrong results, and a dataset without any will hide every one of those bugs.

How the Expression Engine Works

A QGIS expression is not evaluated against a feature directly. It is evaluated against a QgsExpressionContext, which is an ordered stack of scopes — global, project, layer, and feature — each contributing variables and functions. When the evaluator meets a bare name like population, it walks that stack from the most specific scope outward until something resolves it. A field reference resolves in the feature scope; @project_folder resolves in the project scope; a custom function you registered resolves globally.

This design is what lets the same expression string work in the field calculator, in a symbology rule, in a Processing parameter and in your own code. It is also why an expression that works in the GUI returns nothing in a script: the GUI assembles a full context for you, and a bare QgsExpression("population > 1000") evaluated with no context has no feature scope, so population resolves to NULL and the comparison yields NULL rather than False.

What an expression needs before it can be evaluated Three bands showing the pieces of an expression evaluation: the expression text is parsed into a tree, an expression context supplies scopes for the project, the layer and the current feature, and the evaluator walks the tree resolving field and variable references against those scopes. Expression QgsExpression parsed once prepare(context) resolves field indexes Context global + project scope variables layer scope fields, @layer_name feature scope the current row Result evaluate(feature) a Python value hasEvalError() never raises An expression with no context still parses and still evaluates — it simply cannot see any field, so every reference returns NULL.

Two calls turn this from a subtlety into a rule you can apply mechanically. setFields() or a layer scope tells the expression which field names exist; prepare(context) resolves each field name to a column index once, so the per-feature evaluation is an array lookup rather than a string comparison. Skipping prepare() is legal and produces correct results — it just does the same lookup work on every single row.

Step-by-Step Implementation

Step 1 — Inspect the schema before touching it

Field names are case-sensitive in some providers and folded in others, and a name that exists in your test file may not exist in production data. Resolve the index once, and treat -1 as a hard error rather than a value to pass on.

python
from qgis.core import QgsVectorLayer


def field_index(layer: QgsVectorLayer, name: str) -> int:
    """Return the index of `name` on `layer`, raising if it is absent.

    QGIS returns -1 for an unknown field rather than raising, and -1 is a valid
    argument to several attribute APIs, where it silently does nothing.
    """
    idx = layer.fields().indexFromName(name)
    if idx < 0:
        available = ", ".join(f.name() for f in layer.fields())
        raise KeyError("no field %r on %r (have: %s)" % (name, layer.name(), available))
    return idx

Step 2 — Add a field through the provider

New columns are a schema change, so they go through the data provider rather than the edit buffer. The provider write is immediate and not undoable; call updateFields() afterwards so the layer’s cached schema matches what is now on disk.

python
from qgis.core import QgsField, QgsVectorLayer
from qgis.PyQt.QtCore import QVariant


def add_double_field(layer: QgsVectorLayer, name: str) -> int:
    """Add a double-precision field to `layer` and return its index."""
    if layer.fields().indexFromName(name) >= 0:
        return layer.fields().indexFromName(name)   # already present, nothing to do

    provider = layer.dataProvider()
    if not provider.addAttributes([QgsField(name, QVariant.Double)]):
        raise RuntimeError("provider refused to add %r" % name)
    layer.updateFields()
    return field_index(layer, name)

Step 3 — Build and prepare an expression

Construct the expression once, hand it a context that carries the layer scope, and prepare it. QgsExpression never raises on a bad expression: it records an error you have to ask for.

python
from qgis.core import (QgsExpression, QgsExpressionContext,
                       QgsExpressionContextUtils, QgsVectorLayer)


def prepared_expression(layer: QgsVectorLayer, text: str):
    """Return a parsed, prepared expression and the context to evaluate it with."""
    expression = QgsExpression(text)
    if expression.hasParserError():
        raise ValueError("cannot parse %r: %s" % (text, expression.parserErrorString()))

    context = QgsExpressionContext()
    context.appendScopes(QgsExpressionContextUtils.globalProjectLayerScopes(layer))
    expression.prepare(context)
    return expression, context

Step 4 — Evaluate per feature and write the result

The write goes through changeAttributeValue() inside an edit session, so it lands on the undo stack and can be rolled back as a unit. Check the evaluation error on every row: an expression that divides by a NULL column produces an error for that feature only, and silently writing the resulting NULL is how bad data spreads.

python
from qgis.core import QgsVectorLayer


def calculate_field(layer: QgsVectorLayer, target: str, text: str) -> int:
    """Evaluate `text` for every feature and write it to `target`. Returns rows written."""
    target_idx = add_double_field(layer, target)
    expression, context = prepared_expression(layer, text)

    written = 0
    layer.startEditing()
    try:
        for feature in layer.getFeatures():
            context.setFeature(feature)
            value = expression.evaluate(context)
            if expression.hasEvalError():
                raise RuntimeError("feature %s: %s" % (feature.id(), expression.evalErrorString()))
            if layer.changeAttributeValue(feature.id(), target_idx, value):
                written += 1
        if not layer.commitChanges():
            raise RuntimeError("commit failed: %s" % "; ".join(layer.commitErrors()))
    except Exception:
        layer.rollBack()
        raise
    return written

The try block matters more than it looks. Without the rollback, an exception halfway through leaves the layer in an open edit session with partial changes buffered — and the next piece of code to call startEditing() inherits them.

Where each way of changing an attribute actually writes A grid of four write paths — changeAttributeValue, dataProvider.changeAttributeValues, the field calculator algorithm and a direct provider write — showing whether each is undoable, whether it needs an edit session, and when it reaches disk. undoable? needs editing? reaches disk layer.changeAttributeValue yes yes on commit provider.changeAttributeValues no no immediately native:fieldcalculator n/a no writes a new layer layer.dataProvider().addAttributes no no immediately

Advanced Patterns

Push the rule into the provider when it only selects

If the expression decides which rows to process rather than computing a value, it does not need to run in Python at all. QgsFeatureRequest.setFilterExpression() hands the same expression text to the provider, which compiles what it can into its native query language — a SQL WHERE clause on PostGIS, a SQLite predicate on GeoPackage — and returns only matching features.

python
from qgis.core import QgsFeatureRequest, QgsVectorLayer


def active_large_parcels(layer: QgsVectorLayer):
    """Yield parcels the provider itself selected, never materialising the rest."""
    request = (QgsFeatureRequest()
               .setFilterExpression("status = 'ACTIVE' AND area_m2 > 1000")
               .setSubsetOfAttributes(["parcel_id", "area_m2"], layer.fields()))
    yield from layer.getFeatures(request)

What the provider cannot compile it evaluates itself, feature by feature, after reading the row — so the call is always correct and sometimes free. Expressions using QGIS-specific functions ($area, @atlas_feature) always fall into the second category, which is a good reason to prefer a stored area_m2 column over $area in any expression that runs at scale.

Register a custom function once, use it everywhere

The @qgsfunction decorator adds a function to the global expression scope. Once registered it is available to the field calculator, symbology, labelling and your own code alike, which makes it a good home for a domain rule that would otherwise be copied into a dozen expression strings.

python
from qgis.core import qgsfunction


@qgsfunction(args="auto", group="Custom", referenced_columns=[])
def band_for_density(density, feature, parent):
    """Return the planning density band for a value in dwellings per hectare."""
    if density is None:
        return None
    if density < 20:
        return "low"
    return "medium" if density < 60 else "high"

Declaring referenced_columns=[] matters for performance: it tells QGIS the function does not read fields directly, so the engine does not have to fetch every attribute to evaluate it.

Virtual fields for values that must stay derived

A virtual field stores the expression, not the value, and recomputes it whenever the column is read. That is exactly right for a value which must never drift from its inputs — a ratio of two columns that other processes keep updating — and exactly wrong for anything expensive, because the cost is paid on every read, including every repaint of the attribute table.

Choosing where to evaluate an attribute rule A decision tree on where an attribute rule belongs: a provider-side filter expression when it only selects rows, a prepared QgsExpression when the rule must run per feature in Python, and a virtual field when the value should be visible in the attribute table. What is this attribute rule for? selecting rows setFilterExpression runs in the provider computing values prepared QgsExpression evaluate per feature showing a column virtual field recomputed on read

Bulk writes that skip the buffer

For a one-way batch job with no undo requirement, provider.changeAttributeValues() takes a dictionary of feature id to attribute-map and writes it in one call. It bypasses the edit buffer entirely, which makes it substantially faster over large datasets and completely unrecoverable if it goes wrong. Use it in pipelines that can re-run from source, not in plugins where a user expects Ctrl+Z to work.

Cost of evaluating one expression a million times A bar chart comparing three evaluation strategies over a million features: constructing the expression inside the loop, constructing it once outside the loop, and constructing it once and calling prepare with the context before iterating. built per feature 96 s built once 14 s built once + prepared 6 s Indicative totals for a two-field arithmetic expression over a million features. Parsing dominates the first case; unresolved field lookups dominate the second.

Pitfalls and Debugging

  • NULL is not False. In QGIS expression semantics, comparing anything with NULL yields NULL, and a filter that evaluates to NULL excludes the row. A rule like status != 'CLOSED' therefore silently drops every feature whose status is unset. Write status IS NULL OR status != 'CLOSED' when unset should count.

  • hasEvalError() is per evaluation, not per expression. It reflects the most recent evaluate() call. Checking it once after the loop tells you only about the last feature.

  • Changing a field on a layer being iterated. Writing attributes inside a getFeatures() loop on the same layer is undefined behaviour on some providers. Collect the changes into a dictionary and apply them after the loop, or iterate a separate request.

  • updateFields() forgotten after a schema change. The provider has the new column but the layer’s cached fields() does not, so indexFromName() returns -1 for a field that visibly exists in the file.

  • Shapefile field names silently truncated. The DBF format allows ten characters. A field called population_density becomes populatio, and every expression referencing the full name resolves to NULL with no error at all.

  • A commit that returns False. commitChanges() reports failure through its return value and fills commitErrors(). Ignoring it produces a script that logs success and changes nothing — most often because a constraint or a read-only provider rejected the write.

Frequently Asked Questions

Why does my expression return NULL for every feature?

Because the context has no feature scope, or no layer scope, so the field name never resolves. An expression is evaluated against a context, not against a layer, and a default-constructed QgsExpressionContext is empty.

Build the context with QgsExpressionContextUtils.globalProjectLayerScopes(layer) and call context.setFeature(feature) before each evaluation. If the field name is spelled differently from what the provider actually stores — a truncated shapefile name, or a case difference — the reference will still resolve to NULL, so check layer.fields().names() when the context looks right.

Do I have to call prepare()?

No, but it is close to free and it is worth an order of magnitude on large layers. prepare() resolves every field reference in the parsed tree to a column index once. Without it, each evaluation looks each name up by string.

The one requirement is that the context you prepare with must have the same fields as the contexts you later evaluate with. Preparing against one layer and evaluating against another with a different schema produces wrong values rather than an error.

What is the difference between a virtual field and a calculated field?

A calculated field is an ordinary column: the expression runs once, the values are written to disk, and they stay as they are until something recalculates them. A virtual field stores the expression instead, and recomputes the value every time the column is read.

Use a calculated field for anything expensive or anything that must be reproducible later; use a virtual field when the value must never fall out of step with the columns it derives from. Virtual fields are also not written to most output formats, so a job that exports its result needs a real column.

Can expressions be used from a background thread?

Evaluation itself is fine on a worker thread as long as the context does not reach back into main-thread objects. The trap is the layer scope: it holds a reference to the layer, and touching a QgsVectorLayer from a worker is not safe.

For threaded work, build the context and prepare the expression on the main thread, then evaluate against features you pre-fetched. The asynchronous task execution guide covers the wider rule this is one instance of.

How do I test an expression without a layer?

Construct a QgsFeature with an explicit QgsFields schema, set attributes on it, and evaluate against a context whose feature scope you set yourself. No project, no provider and no file are required, which makes expression rules straightforward to unit-test with pytest-qgis.

Conclusion

Attribute work in PyQGIS is governed by two ideas that repay learning properly: expressions are evaluated against a context rather than a feature, and writes go either through the buffered, undoable layer API or the immediate, unrecoverable provider API. Get the context right and expressions behave identically in your script and in the field calculator. Choose the write path deliberately and you get either an undo stack or throughput, rather than discovering which one you have after a failed run.

Related guides

Guide

Filtering Features with QgsFeatureRequest Expressions

Push attribute conditions down to the data provider with QgsFeatureRequest.setFilterExpression: what compiles to SQL, how to combine it with attribute…

Read Guide

How to Add a Field and Populate It with an Expression in PyQGIS

Add a column to a QGIS vector layer from Python and fill it from an expression: the provider schema change, updateFields, a prepared QgsExpression, and a…

Read Guide

Registering a Custom Expression Function with qgsfunction

Add your own function to the QGIS expression engine with the qgsfunction decorator: the feature and parent arguments, declaring referenced columns, reporting…

Read