Performance, transactions, and migrations
Diagnose SQLAlchemy query performance, handle transaction conflicts, and test database behavior. Includes Alembic migrations and SQL checks.
Performance, SQL visibility, caching, and production practices#
Query count, index design, returned row size, and transaction length dominate performance. Measure emitted SQL and database query plans before changing pool settings or optimizing Python syntax.
from sqlalchemy import bindparam, lambda_stmt
from sqlalchemy.orm import load_only, selectinload
stmt = (
select(User)
.options(load_only(User.id, User.name), selectinload(User.addresses))
.execution_options(yield_per=500)
)
for user in session.scalars(stmt):
export(user)
by_email = lambda_stmt(
lambda: select(User).where(User.email == bindparam("email"))
)
user = session.scalar(by_email, {"email": "ada@example.com"})SELECT id, name FROM user_account;
SELECT * FROM address WHERE user_id IN (...);
SELECT * FROM user_account WHERE email = :email;| Symptom | First action | Likely fix |
|---|---|---|
| Repeated selects in loop | Count emitted queries. | selectinload, joinedload, purpose-built aggregate. |
| One slow query | Run EXPLAIN (ANALYZE, BUFFERS). | Index/predicate/join/data model correction. |
| Huge memory export | Measure cardinality. | Projection, yield_per, streaming, batches. |
| Too many connections | Count workers × pools. | Smaller pools/pooler/no engine-per-request. |
| Deadlock/timeouts | Inspect transaction duration/order. | Short transactions, consistent lock order, targeted retry. |
| Repeated compilation | Profile after SQL and DB plan. | Keep default statement cache; use lambda_stmt for hot dynamic shapes. |
import logging
logging.basicConfig()
logging.getLogger("sqlalchemy.engine").setLevel(logging.INFO)
# Or create_engine(url, echo="debug") in development.
plan = session.execute(
text("EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM user_account WHERE email=:e"),
{"e": "ada@example.com"},
).scalars().all()EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM user_account WHERE email = 'ada@example.com';Constraints, locks, optimistic versions, isolation, retries, and exception taxonomy#
Transactions make a group of writes atomic, but concurrent transactions can still race. Pick the strongest simple database guarantee: constraint, atomic set-based update, row lock, version counter, or serializable retry.
from sqlalchemy import update
with SessionLocal.begin() as session:
result = session.execute(
update(Product)
.where(Product.id == product_id, Product.quantity >= requested)
.values(quantity=Product.quantity - requested)
)
if result.rowcount != 1:
raise ValueError("out of stock or missing product")UPDATE product
SET quantity = quantity - :requested
WHERE id = :product_id AND quantity >= :requested;
-- One row means the rule held at write time.| Problem | Best simple tool | Reason |
|---|---|---|
| Duplicate email/booking | Unique constraint | Works across all processes and code paths. |
| Decrement only if enough stock | Conditional UPDATE | One atomic statement, no read-modify-write gap. |
| Exclusive edit | with_for_update() | Lock row until transaction end. |
| Detect stale form update | version_id_col | Compare-and-swap behavior. |
| Serializable conflict | Retry entire short transaction | Database may require it; use a new Session. |
| Optional partial operation | begin_nested() | Inner SAVEPOINT can roll back alone. |
from sqlalchemy.exc import IntegrityError, OperationalError
try:
with SessionLocal.begin() as session:
session.add(User(name="Ada", email="taken@example.com"))
except IntegrityError as exc:
# Context manager rolled back already.
raise ValueError("email is already registered") from exc
# OperationalError can include timeout/disconnect/deadlock.
# Retry only documented transient failures and retry the WHOLE
# idempotent transaction with a fresh Session.INSERT INTO user_account (name, email)
VALUES ('Ada', 'taken@example.com');
-- UNIQUE violation
ROLLBACK;Testing, test transactions, Alembic migrations, and SQL compilation checks#
Test database behavior you depend on: unique constraints, query result shape, loader count, locks, JSON operators, and migration on representative data. Alembic is SQLAlchemy’s separate migration project.
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
@pytest.fixture
def session():
engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(engine)
with Session(engine) as session:
yield session
session.rollback()
engine.dispose()
def test_email_unique(session):
session.add(User(name="Ada", email="ada@example.com"))
session.commit()
session.add(User(name="Other", email="ada@example.com"))
with pytest.raises(IntegrityError):
session.commit()pip install alembic
alembic init migrations
# Set target_metadata = Base.metadata in env.py
alembic revision --autogenerate -m "create user account"
alembic upgrade head
alembic downgrade -1
alembic current
alembic historydef upgrade() -> None:
op.add_column(
"user_account",
sa.Column("nickname", sa.String(), nullable=True),
)
op.create_index("ix_user_account_name", "user_account", ["name"])
def downgrade() -> None:
op.drop_index("ix_user_account_name", table_name="user_account")
op.drop_column("user_account", "nickname")ALTER TABLE user_account ADD COLUMN nickname VARCHAR;
CREATE INDEX ix_user_account_name ON user_account (name);| Level | Verify |
|---|---|
| Mapping | Columns, relationship direction, default, constraint names. |
| Query | Correct rows and no loader/N+1 regression. |
| Transaction | Rollback, conflict, retry and locking behavior. |
| Migration | Upgrade empty + populated databases; data migration logic. |
| API | One session/request, validation, commit, eager serialization. |
Full feature catalog and plain-English glossary#
SQLAlchemy feature groups and terminology. Follow the official references for method arguments and database-specific options.
select, aliases, joins, loader options, bundles, results, lambda statements.Glossary
| Word | Meaning |
|---|---|
| Dialect | Database-specific translator, e.g. PostgreSQL or SQLite. |
| DBAPI | Python driver used under SQLAlchemy, e.g. psycopg/asyncpg. |
| Identity map | Within one Session, one database row becomes one Python object instance. |
| Unit of work | SQLAlchemy tracks changes, orders dependent SQL, then flushes them. |
| Flush | Send pending changes inside current transaction; not permanent yet. |
| Commit | Make transaction permanent/visible to other transactions. |
| Rollback | Undo uncommitted work and reset failed transaction state. |
| Lazy load | Fetch relationship when attribute is accessed, possibly a new query. |
| Eager load | Fetch related data as part of original query plan. |
| Dialect-specific | Intentional use of a backend feature other databases may not have. |
Official source material: SQLAlchemy 2.0 documentation · ORM Quick Start · Core · ORM extensions · Alembic.
Help improve CodeRef
Allow Google Analytics to measure page visits and use of the guides? It uses cookies after you accept. Your choice won’t affect access to the site. Privacy notice