PyQGIS Engineering & Automation

How to Add a Field and Populate It with an Expression in PyQGIS

Add a column to a QGIS vector layer from Python and fill it from an expression: the provider schema change, updateFields, a prepared QgsExpression, and a…

TL;DR: add the column through the data provider, call updateFields(), then evaluate a prepared QgsExpression per feature inside a single edit session and commit once — the provider handles the schema, the edit buffer handles the values, and mixing the two up is what makes this fail. This page is part of the attribute data and the expression engine guide.

Complete Runnable Code

python
"""Add a numeric field to a layer and fill it from a QGIS expression."""
from qgis.core import (QgsExpression, QgsExpressionContext, QgsExpressionContextUtils,
                       QgsField, QgsVectorLayer)
from qgis.PyQt.QtCore import QVariant


def add_calculated_field(layer: QgsVectorLayer, name: str, expression_text: str,
                         field_type: QVariant.Type = QVariant.Double) -> int:
    """Add `name` to `layer` and populate it by evaluating `expression_text`.

    Returns the number of features written. Raises on a parse error, on a
    refused schema change, or on a failed commit — never leaves the layer in an
    open edit session.
    """
    expression = QgsExpression(expression_text)
    if expression.hasParserError():
        raise ValueError("cannot parse %r: %s" % (expression_text,
                                                  expression.parserErrorString()))

    # 1. schema change: goes through the provider, is immediate, is not undoable
    if layer.fields().indexFromName(name) < 0:
        if not layer.dataProvider().addAttributes([QgsField(name, field_type)]):
            raise RuntimeError("provider refused to add field %r" % name)
        layer.updateFields()                      # refresh the layer's cached schema

    index = layer.fields().indexFromName(name)
    if index < 0:
        raise RuntimeError("field %r still missing after updateFields()" % name)

    # 2. one context, prepared once — not once per feature
    context = QgsExpressionContext()
    context.appendScopes(QgsExpressionContextUtils.globalProjectLayerScopes(layer))
    expression.prepare(context)

    written = 0
    layer.startEditing()
    try:
        for feature in layer.getFeatures():
            context.setFeature(feature)
            value = expression.evaluate(context)
            if expression.hasEvalError():
                raise RuntimeError("feature %s: %s" % (feature.id(),
                                                       expression.evalErrorString()))
            if layer.changeAttributeValue(feature.id(), index, value):
                written += 1
        if not layer.commitChanges():
            raise RuntimeError("commit failed: %s" % "; ".join(layer.commitErrors()))
    except Exception:
        layer.rollBack()
        raise
    return written

Call it with any valid QGIS expression — "$area / 10000", "population / area_km2", "upper(\"name\")" — and the field type that matches what the expression returns.

The four calls that add and fill a column A four-step pipeline: add the attribute through the provider, refresh the layer field cache, evaluate the prepared expression per feature, and commit the buffered writes in one transaction. addAttributes() provider, immediate updateFields() refresh the cache evaluate() per feature commitChanges() one transaction schema then then

Architecture Breakdown

dataProvider().addAttributes() — the schema half

A new column is a change to the file on disk, not a change to a row, so it does not go through the edit buffer at all. addAttributes() writes it immediately and returns a boolean; there is no undo, and no commit is required or possible.

The call that is easy to forget is updateFields(). Without it the provider has the new column but the layer’s cached fields() does not, so indexFromName() returns -1 for a field that visibly exists in the file — and -1 is accepted by changeAttributeValue(), where it silently does nothing.

Picking the field type for the value you are about to write A grid of four QVariant field types with the Python value each accepts, a typical use, and the failure produced by choosing the wrong one. accepts typical use wrong choice gives QVariant.Double float areas, ratios silent truncation QVariant.Int int counts, codes overflow on large ids QVariant.String str labels, classes numbers sort as text QVariant.Bool bool flags unsupported on shapefile

QgsExpression.prepare() — resolving names once

Preparing an expression walks its parsed tree and binds every field reference to a column index using the fields visible in the context. After that, each evaluation is an array lookup rather than a name lookup, which on a large layer is the difference between a job that takes seconds and one that takes minutes.

The context you prepare with must carry the same fields as the contexts you later evaluate with. Preparing against one layer and evaluating against another with a different schema does not raise — it reads the wrong columns.

changeAttributeValue() — the buffered half

This writes into the edit session’s change buffer and pushes an entry onto the undo stack. Nothing reaches disk until commitChanges(), which is what makes the whole operation atomic: an exception halfway through leaves the file untouched, provided you roll back.

commitChanges() reports failure through its return value rather than an exception. Ignoring it produces a script that logs success and changes nothing, usually because a constraint or a read-only provider rejected the write.

Handling a Failure on One Feature

Not every evaluation error deserves the same response, and deciding which you have is the difference between a run that stops usefully and one that either dies on the first gap in the data or writes nonsense for a thousand rows.

What to do when an expression fails on one feature A decision tree over per-feature evaluation errors: write NULL and continue when the input is genuinely unknown, abort the whole run when the error indicates a broken expression, and record the identifier when the failure is data-specific. What does the eval error mean? NULL input write NULL expected for gaps bad expression abort and roll back every row will fail one odd row log the feature id fix the data

The example above treats any evaluation error as fatal, which is the right default: an expression that fails on one row usually fails on the next thousand. When the input genuinely contains gaps — a ratio whose denominator is sometimes NULL — the correct behaviour is to write NULL for that feature and carry on, which QGIS expression semantics already do without raising. Reserve the hard failure for errors that indicate the expression itself is wrong.

Why the Order of Those Calls Matters

The sequence in the example is not a style choice. A schema change and a value change travel through two different halves of the QGIS layer API, and each has its own rules about when it takes effect.

The provider write happens immediately and cannot be undone, which is why it comes first: if the column cannot be created — a read-only file, an unsupported type, a name the format rejects — you want to know before any values have been computed. Doing it inside the edit session instead produces the worst outcome available, where the column is created, the commit then fails, and the layer is left with a new empty field nobody asked for.

The value writes happen in the buffer and take effect together. That is what makes the operation recoverable: an exception on feature nine hundred rolls back the previous eight hundred and ninety-nine, and the file on disk is exactly as it was. Skipping the rollback breaks that guarantee in a particularly unhelpful way, because the layer stays in an open edit session and the next piece of code to call startEditing() inherits the partial changes as though they were its own.

Running It Headlessly

Nothing here requires a GUI. Inside a standalone script, bootstrap the application first and open the layer directly:

python
from qgis.core import QgsApplication, QgsVectorLayer

QgsApplication.setPrefixPath("/usr", True)
app = QgsApplication([], False)
app.initQgis()
try:
    layer = QgsVectorLayer("/data/parcels.gpkg|layername=parcels", "parcels", "ogr")
    if not layer.isValid():
        raise SystemExit("could not open the layer")
    print(add_calculated_field(layer, "area_ha", "$area / 10000"))
finally:
    app.exitQgis()

The standalone scripts and headless execution guide covers the bootstrap in full, including the environment variables a container needs.

Production Best Practices

  • Check indexFromName() twice — before adding, to avoid a duplicate, and after updateFields(), to confirm the provider actually accepted the change.
  • Prepare the expression once, outside the loop. It is one line and worth an order of magnitude.
  • Always roll back on the error path. A layer left in an open edit session poisons whatever runs next.
  • Prefer a real column over $area in expressions that run at scale. Geometric functions are recomputed per evaluation and cannot be pushed down to the provider.
  • Mind the shapefile field-name limit. Ten characters, silently truncated, and every expression referencing the full name then resolves to NULL.
  • Write the expression text into the layer or project metadata when the value must be reproducible later. A column with no record of how it was derived is a number nobody can defend.

Frequently Asked Questions

Why is my new field full of NULLs?

Either the expression referenced a field name that does not resolve — check layer.fields().names() against exactly what you typed — or the context had no feature scope when it was evaluated. Both produce NULL rather than an error, which is why the eval-error check in the example matters.

Can I do this with a Processing algorithm instead?

Yes: native:fieldcalculator takes the same expression and writes a new layer. That is the better choice inside a pipeline, because it leaves the input untouched. Use the in-place approach above when you specifically want to modify an existing dataset and keep its identifiers.

Does this work on a shapefile?

It does, with two caveats: field names are truncated to ten characters, and adding a field rewrites the whole DBF, which is slow on large files. GeoPackage has neither limitation and is the better target for anything you control.

How do I update an existing field instead of adding one?

Skip the provider call entirely and resolve the index of the field that is already there. The rest of the function is unchanged — the edit buffer does not care whether the column is new. What does change is the risk profile: overwriting a populated column destroys whatever was in it, and the undo stack only protects you until the commit.

For anything irreversible, run it on a copy first and compare. native:fieldcalculator writing to a new layer is the safer shape when the values matter.

Can I write to several fields in one pass?

Yes, and you should when more than one derived value comes from the same iteration. Prepare one expression per target field, evaluate them all inside the same feature loop, and use changeAttributeValues() — plural — to apply a whole attribute map for a feature in a single call. One pass over the data instead of three is the entire saving, and on a large layer it is a large one.