SQLAlchemy / Chapter 4 of 4

Performance, transactions, and migrations

Diagnose SQLAlchemy query performance, handle transaction conflicts, and test database behavior. Includes Alembic migrations and SQL checks.

4 topicsUpdated
24 / 27

Performance, SQL visibility, caching, and production practices#

Query count, index design, returned row size, and transaction length dominate performance. Measure emitted SQL and database query plans before changing pool settings or optimizing Python syntax.

PythonProjection, eager load, stream, and statement cache
from sqlalchemy import bindparam, lambda_stmt
from sqlalchemy.orm import load_only, selectinload

stmt = (
    select(User)
    .options(load_only(User.id, User.name), selectinload(User.addresses))
    .execution_options(yield_per=500)
)
for user in session.scalars(stmt):
    export(user)

by_email = lambda_stmt(
    lambda: select(User).where(User.email == bindparam("email"))
)
user = session.scalar(by_email, {"email": "ada@example.com"})
SQLSQL idea
SELECT id, name FROM user_account;
SELECT * FROM address WHERE user_id IN (...);
SELECT * FROM user_account WHERE email = :email;
SymptomFirst actionLikely fix
Repeated selects in loopCount emitted queries.selectinload, joinedload, purpose-built aggregate.
One slow queryRun EXPLAIN (ANALYZE, BUFFERS).Index/predicate/join/data model correction.
Huge memory exportMeasure cardinality.Projection, yield_per, streaming, batches.
Too many connectionsCount workers × pools.Smaller pools/pooler/no engine-per-request.
Deadlock/timeoutsInspect transaction duration/order.Short transactions, consistent lock order, targeted retry.
Repeated compilationProfile after SQL and DB plan.Keep default statement cache; use lambda_stmt for hot dynamic shapes.
PythonSQL logging and plan
import logging
logging.basicConfig()
logging.getLogger("sqlalchemy.engine").setLevel(logging.INFO)
# Or create_engine(url, echo="debug") in development.

plan = session.execute(
    text("EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM user_account WHERE email=:e"),
    {"e": "ada@example.com"},
).scalars().all()
SQLPostgreSQL SQL
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM user_account WHERE email = 'ada@example.com';
Cache data, never Sessions.
A Session contains mutable transaction and identity-map state. Cache explicit data/query results with a clear invalidation strategy, not a Session object.
25 / 27

Constraints, locks, optimistic versions, isolation, retries, and exception taxonomy#

Transactions make a group of writes atomic, but concurrent transactions can still race. Pick the strongest simple database guarantee: constraint, atomic set-based update, row lock, version counter, or serializable retry.

PythonAtomic conditional update
from sqlalchemy import update

with SessionLocal.begin() as session:
    result = session.execute(
        update(Product)
        .where(Product.id == product_id, Product.quantity >= requested)
        .values(quantity=Product.quantity - requested)
    )
    if result.rowcount != 1:
        raise ValueError("out of stock or missing product")
SQLAtomic SQL
UPDATE product
SET quantity = quantity - :requested
WHERE id = :product_id AND quantity >= :requested;
-- One row means the rule held at write time.
ProblemBest simple toolReason
Duplicate email/bookingUnique constraintWorks across all processes and code paths.
Decrement only if enough stockConditional UPDATEOne atomic statement, no read-modify-write gap.
Exclusive editwith_for_update()Lock row until transaction end.
Detect stale form updateversion_id_colCompare-and-swap behavior.
Serializable conflictRetry entire short transactionDatabase may require it; use a new Session.
Optional partial operationbegin_nested()Inner SAVEPOINT can roll back alone.
PythonExpected database error handling
from sqlalchemy.exc import IntegrityError, OperationalError

try:
    with SessionLocal.begin() as session:
        session.add(User(name="Ada", email="taken@example.com"))
except IntegrityError as exc:
    # Context manager rolled back already.
    raise ValueError("email is already registered") from exc

# OperationalError can include timeout/disconnect/deadlock.
# Retry only documented transient failures and retry the WHOLE
# idempotent transaction with a fresh Session.
SQLFailed SQL transaction
INSERT INTO user_account (name, email)
VALUES ('Ada', 'taken@example.com');
-- UNIQUE violation
ROLLBACK;
Set database-side timeouts.
An application request timeout may not stop a database query that is already running. Configure statement and lock timeouts appropriate to your database and workload.
26 / 27

Testing, test transactions, Alembic migrations, and SQL compilation checks#

Test database behavior you depend on: unique constraints, query result shape, loader count, locks, JSON operators, and migration on representative data. Alembic is SQLAlchemy’s separate migration project.

PythonSmall pytest-style fixture
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import Session

@pytest.fixture
def session():
    engine = create_engine("sqlite+pysqlite:///:memory:")
    Base.metadata.create_all(engine)
    with Session(engine) as session:
        yield session
        session.rollback()
    engine.dispose()

def test_email_unique(session):
    session.add(User(name="Ada", email="ada@example.com"))
    session.commit()
    session.add(User(name="Other", email="ada@example.com"))
    with pytest.raises(IntegrityError):
        session.commit()
ShellAlembic essentials
pip install alembic
alembic init migrations
# Set target_metadata = Base.metadata in env.py
alembic revision --autogenerate -m "create user account"
alembic upgrade head
alembic downgrade -1
alembic current
alembic history
PythonMigration draft
def upgrade() -> None:
    op.add_column(
        "user_account",
        sa.Column("nickname", sa.String(), nullable=True),
    )
    op.create_index("ix_user_account_name", "user_account", ["name"])

def downgrade() -> None:
    op.drop_index("ix_user_account_name", table_name="user_account")
    op.drop_column("user_account", "nickname")
SQLSchema SQL
ALTER TABLE user_account ADD COLUMN nickname VARCHAR;
CREATE INDEX ix_user_account_name ON user_account (name);
LevelVerify
MappingColumns, relationship direction, default, constraint names.
QueryCorrect rows and no loader/N+1 regression.
TransactionRollback, conflict, retry and locking behavior.
MigrationUpgrade empty + populated databases; data migration logic.
APIOne session/request, validation, commit, eager serialization.
Autogenerate is a draft.
Review every Alembic migration: renames can look like drop/add, data changes need manual code, and some indexes/constraints/defaults cannot be inferred reliably.
27 / 27

Full feature catalog and plain-English glossary#

SQLAlchemy feature groups and terminology. Follow the official references for method arguments and database-specific options.

ORM queryingselect, aliases, joins, loader options, bundles, results, lambda statements.
ORM persistenceUnit of work, cascades, merge, flush events, bulk DML, RETURNING, versioning.
Mapping stylesAnnotated declarative, dataclass, imperative, automap, deferred reflection.
Mapping propertiesColumn, relationship, composite, synonym, column/query expression, deferred, dynamic/write-only.
RelationshipsAll cardinalities, secondary, association object/proxy, self-referential, custom joins, cascade.
InheritanceSingle/joined/concrete, polymorphic selection and loading, abstract mixins.
Core expressionsOperators, binds, literals, functions, CASE, casts, labels, custom operators.
Complex SQLSubqueries, scalar/correlated SELECT, CTE/recursive, set ops, EXISTS, LATERAL, tablesample, windows.
Core DMLInsert/update/delete, values, executemany, returning, from-select, dialect upserts.
SchemaMetadata, tables, columns, constraints, indexes, DDL, identity/sequence/computed/defaults.
Type systemGeneric/standard/vendor types, decorators, user-defined types, variants, mutability.
Engine/poolsURL, Engine, Connection, autobegin, QueuePool, NullPool, StaticPool, pool events.
AsyncIOAsync engine/connection/session/results/scoped session, run_sync, AsyncAttrs.
DialectsPostgreSQL, MySQL/MariaDB, SQLite, SQL Server, Oracle, third-party dialects.
ExtensionsAssociation proxy, automap, compiler, declarative extensions, hybrid, indexable, mutable, ordering list, sharding.
EventsPool, connection, DDL, dialect, engine, mapper, session, attribute, instrumentation.
Inspection/compilationRuntime inspect, SQL compiler customization, execution options, cache controls, plugins.

Glossary

WordMeaning
DialectDatabase-specific translator, e.g. PostgreSQL or SQLite.
DBAPIPython driver used under SQLAlchemy, e.g. psycopg/asyncpg.
Identity mapWithin one Session, one database row becomes one Python object instance.
Unit of workSQLAlchemy tracks changes, orders dependent SQL, then flushes them.
FlushSend pending changes inside current transaction; not permanent yet.
CommitMake transaction permanent/visible to other transactions.
RollbackUndo uncommitted work and reset failed transaction state.
Lazy loadFetch relationship when attribute is accessed, possibly a new query.
Eager loadFetch related data as part of original query plan.
Dialect-specificIntentional use of a backend feature other databases may not have.

Official source material: SQLAlchemy 2.0 documentation · ORM Quick Start · Core · ORM extensions · Alembic.