Polars / Chapter 2 of 7

Select, filter, and transform

Use Polars select, with_columns, and filter to transform data. Python examples cover nulls, duplicates, strings, dates, and time zones.

9 topicsUpdated 22 Oct 2026

04 / 50

Select columns and build expressions#

An expression describes a computation; a context decides how it applies to the table.

A context is the method receiving expressions. select creates a table with only the requested results. alias names a result; it does not modify the original column. Arithmetic combines values row by row, while sum() produces one total. A literal is a fixed value: use pl.lit to make intent explicit. Strings passed in many expression positions mean column names, not text constants.

PythonLine totals and a one-row report

import polars as pl

df = pl.DataFrame(
    {"order": [1001, 1002, 1003], "qty": [2, 1, 3], "price": [1250, 3000, 500]}
)
lines = df.select("order", (pl.col("qty") * pl.col("price")).alias("cents"))
print("lines:", lines.to_dicts())
print(
    "summary:",
    lines.select(
        pl.len().alias("rows"), pl.col("cents").sum().alias("total")
    ).to_dicts(),
)
print("original columns:", df.columns)

TextExpected output

lines: [{'order': 1001, 'cents': 2500}, {'order': 1002, 'cents': 3000}, {'order': 1003, 'cents': 1500}]
summary: [{'rows': 3, 'total': 7000}]
original columns: ['order', 'qty', 'price']

Shape and naming

The first select keeps three rows but changes the columns. The second reduces three rows to one report row. pl.len counts rows; an expression count counts non-null values. An alias belongs to its expression result. Avoid two results with the same name in one select.

Official reference: Select columns and build expressions.

05 / 50

Add columns and make conditional decisions#

with_columns retains existing columns and adds or replaces named results.

when(...).then(...).otherwise(...) picks a value per row. This is a column expression, not Python's scalar if. Expressions in one with_columns call see the input schema: chain another call when a calculation depends on a column you just added. Polars may compute the branches independently before choosing results; every branch must be valid for all relevant inputs.

PythonInvoice totals and quantity checks

import polars as pl

df = pl.DataFrame(
    {"order": [1, 2, 3], "qty": [2, 0, 5], "price_cents": [1000, 800, 600]}
)
result = df.with_columns(
    (pl.col("qty") * pl.col("price_cents")).alias("total_cents")
).with_columns(
    pl.when(pl.col("qty") > 0)
    .then(pl.lit("valid"))
    .otherwise(pl.lit("check"))
    .alias("status"),
    pl.when(pl.col("qty") > 0)
    .then(pl.col("total_cents") / pl.col("qty").replace(0, None))
    .otherwise(None)
    .alias("per_item_cents"),
)
print(result.to_dicts())

TextExpected output

[{'order': 1, 'qty': 2, 'price_cents': 1000, 'total_cents': 2000, 'status': 'valid', 'per_item_cents': 1000.0}, {'order': 2, 'qty': 0, 'price_cents': 800, 'total_cents': 0, 'status': 'check', 'per_item_cents': None}, {'order': 3, 'qty': 5, 'price_cents': 600, 'total_cents': 3000, 'status': 'valid', 'per_item_cents': 600.0}]

What changes

Two chained with_columns calls let the second calculation use total_cents. pl.lit creates status text rather than column references. Replacing zero denominators with null avoids division by zero. Use rename, drop, and with_row_index to change names, remove columns, or track row positions.

Official reference: Add columns and make conditional decisions.

06 / 50

Filter rows and sort results#

Filtering keeps rows where the condition is true; sorting changes their order.

Use &, |, and ~ for element-by-element AND, OR, and NOT. Put each comparison in parentheses. Python and and or expect one Boolean, not a column. A predicate is a condition selecting rows. False and null predicates are both discarded. Sorting without enough tie-breaking columns does not define which equal row comes first.

PythonPick the largest paid web orders

import polars as pl

df = pl.DataFrame(
    {
        "id": [1, 2, 3, 4],
        "status": ["paid", "paid", "open", None],
        "channel": ["web", "store", "web", "web"],
        "cents": [500, 900, 1000, 700],
    }
)
chosen = df.filter(
    (pl.col("status") == "paid") & pl.col("channel").is_in(["web", "store"])
)
print("paid:", chosen.sort(["cents", "id"], descending=[True, False]).to_dicts())
print("unknown status:", df.filter(pl.col("status").is_null())["id"].to_list())
print("range:", df.filter(pl.col("cents").is_between(600, 950))["id"].to_list())

TextExpected output

paid: [{'id': 2, 'status': 'paid', 'channel': 'store', 'cents': 900}, {'id': 1, 'status': 'paid', 'channel': 'web', 'cents': 500}]
unknown status: [4]
range: [2, 4]

Use a deterministic final order

group_by and joins need not preserve input order by default. Sort the final output when row order matters to a report, export, or test. head and slice select row positions, while filter selects values.

Official reference: Filter rows and sort results.

07 / 50

Types, casting, and exact money#

A cast converts values to another type; failed conversions should become visible decisions.

Type	Meaning and use
`String`	Text, IDs, postcodes; retains leading zeros.
`Int64 / UInt64`	Whole numbers; unsigned values cannot represent negatives. Smaller widths save memory but have smaller ranges.
`Float64`	Approximate fractional numbers, also NaN and infinity.
`Decimal(precision=18, scale=2)`	Up to 18 decimal digits in total, with two after the decimal point.
`Boolean`	True/false plus possible null.
`Date / Datetime / Duration / Time`	Calendar day / timestamp / elapsed interval / time of day.
`List / Array / Struct`	Variable-length sequence / fixed-size sequence / named fields in one cell.

cast(..., strict=True) raises on invalid conversions. strict=False produces null for values that cannot be converted; count and retain those failures. Decimal precision is total digits; scale is fractional digits. Construct exact money from strings or Python Decimal, not binary floats. Casting or arithmetic can change scale or truncate fractional digits; define and test your rounding and overflow policy explicitly. Integer minor units are often simpler for two-decimal currencies; other currencies require their own scale.

PythonParse quantities and sum exact decimal amounts

from decimal import Decimal
import polars as pl

raw = pl.DataFrame(
    {"qty_text": ["3", "bad", None], "amount_text": ["0.10", "0.20", "12.50"]}
)
parsed = raw.with_columns(
    pl.col("qty_text").cast(pl.Int64, strict=False).alias("qty"),
    pl.col("amount_text").cast(pl.Decimal(precision=18, scale=2)).alias("amount"),
)
print("quantities:", parsed["qty"].to_list())
print(
    "invalid:",
    parsed.filter(pl.col("qty_text").is_not_null() & pl.col("qty").is_null())[
        "qty_text"
    ].to_list(),
)
print("amount type:", parsed.schema["amount"])
print("exact total:", parsed["amount"].sum())
assert parsed["amount"].sum() == Decimal("12.80")
try:
    raw.select(pl.col("qty_text").cast(pl.Int64))
except pl.exceptions.InvalidOperationError:
    print("strict cast: rejected bad quantity")

TextExpected output

quantities: [3, None, None]
invalid: ['bad']
amount type: Decimal(precision=18, scale=2)
exact total: 12.80
strict cast: rejected bad quantity

Type-safe arithmetic is not a business check

A successfully parsed integer can still be a negative quantity or an impossible year. Check allowed ranges separately. Rounding a Float64 does not turn it into exact decimal storage. Decimal support and precision rules should be verified for the specific operations in a financial pipeline.

Official reference: Types, casting, and exact money.

08 / 50

Null, NaN, and missing-value policies#

Missing and invalid numeric values need different detection and replacement operations.

null is missing and can occur in any type. Python None creates it. NaN is a floating-point value meaning “not a number”; it is not counted by null_count. fill_null leaves NaN unchanged; fill_nan leaves null unchanged. Infinity is another floating-point value, not either of these. Aggregations such as mean skip null but can be contaminated by NaN. Do not use == None to find missing values.

PythonNormalize sensor measurements without confusing zero and missing

import math
import polars as pl

df = pl.DataFrame({"reading": [10.0, None, float("nan"), 0.0]})
print("null count:", df["reading"].null_count())
print("NaN count:", df.select(pl.col("reading").is_nan().sum()).item())
print("mean is NaN:", math.isnan(df["reading"].mean()))
clean = df.with_columns(pl.col("reading").fill_nan(None))
print("mean after normalization:", clean["reading"].mean())
print(
    "fill both:",
    clean.with_columns(pl.col("reading").fill_null(0))["reading"].to_list(),
)
print("observed only:", clean.drop_nulls("reading")["reading"].to_list())

TextExpected output

null count: 1
NaN count: 1
mean is NaN: True
mean after normalization: 5.0
fill both: [10.0, 0.0, 0.0, 0.0]
observed only: [10.0, 0.0]

over("customer") applies the fill within each customer. Sort first to define which observation comes before another.

PythonForward-fill within each customer, after sorting

import polars as pl

df = pl.DataFrame(
    {
        "customer": ["A", "A", "B", "B"],
        "day": [1, 2, 1, 2],
        "balance": [10, None, None, 20],
    }
)
result = df.sort(["customer", "day"]).with_columns(
    pl.col("balance").forward_fill().over("customer").alias("filled")
)
print(result.to_dicts())

TextExpected output

[{'customer': 'A', 'day': 1, 'balance': 10, 'filled': 10}, {'customer': 'A', 'day': 2, 'balance': None, 'filled': 10}, {'customer': 'B', 'day': 1, 'balance': None, 'filled': None}, {'customer': 'B', 'day': 2, 'balance': 20, 'filled': 20}]

Missing-value policy

Forward-fill assumes the last observation remains valid; the first missing balance for B stays missing. Use pl.coalesce for fallback columns. Interpolation estimates values between observations and should not be treated as measured data.

Official reference: Null, NaN, and missing-value policies.

09 / 50

Duplicates and the latest customer record#

Decide which fields define identity and which record should win.

A duplicate key is a repeated identifier. It is different from an entirely duplicated row. unique(subset=...) keeps one row per chosen key. keep="any" allows any winner; do not use it when the winner matters. For the latest record, sort by update time and a stable tie-breaker before keeping the last. Missing IDs need a policy too; unrelated records with null IDs should not accidentally collapse into one customer.

PythonKeep a deterministic latest record per customer

import polars as pl

df = pl.DataFrame(
    {
        "customer": ["A", "A", "B", "A"],
        "updated": ["2026-09-01", "2026-09-03", "2026-09-02", "2026-09-03"],
        "event_id": [1, 2, 3, 4],
        "email": ["old@a.test", "new@a.test", "b@b.test", "final@a.test"],
    }
)
df = df.with_columns(pl.col("updated").str.to_date())
print(
    "duplicate keys:",
    df.group_by("customer").len().filter(pl.col("len") > 1).sort("customer").to_dicts(),
)
latest = (
    df.sort(["customer", "updated", "event_id"])
    .unique(subset="customer", keep="last", maintain_order=True)
    .sort("customer")
)
print("latest:", latest.select("customer", "event_id", "email").to_dicts())
assert latest["customer"].n_unique() == latest.height

TextExpected output

duplicate keys: [{'customer': 'A', 'len': 3}]
latest: [{'customer': 'A', 'event_id': 4, 'email': 'final@a.test'}, {'customer': 'B', 'event_id': 3, 'email': 'b@b.test'}]

A tie needs an explicit rule

Both A updates have the same date, so event_id 4 wins. An input row index can break ties only if the input order itself is stable and is the intended policy. maintain_order makes the output order predictable for this operation and may constrain execution; it does not replace the winner rule.

Official reference: Duplicates and the latest customer record.

10 / 50

Clean strings and messy exports#

Text operations act on a whole column without a Python loop.

The .str namespace groups text operations. Whitespace and case can prevent IDs or emails from matching. A regular expression describes a text pattern; Polars uses a Rust regex engine, so Python-only features such as look-around are not supported. Use literal=True when searching for exact text with punctuation.

PythonNormalize emails and parse a controlled money format

import polars as pl

df = pl.DataFrame(
    {
        "email": ["  ADA@EXAMPLE.COM ", "no-email", None],
        "money": [" USD 1,250.50 ", "USD 20.00", "unknown"],
        "reference": ["INV-0024", "INV-0025", "other"],
    }
)
result = df.with_columns(
    pl.col("email").str.strip_chars().str.to_lowercase().alias("email"),
    pl.col("money")
    .str.strip_chars()
    .str.replace_all("USD ", "", literal=True)
    .str.replace_all(",", "", literal=True)
    .cast(pl.Decimal(18, 2), strict=False)
    .alias("amount"),
    pl.col("reference").str.extract(r"^INV-(\d+)$", 1).alias("invoice_number"),
)
print(result.to_dicts())
print(
    "invalid email rows:",
    result.filter(
        ~pl.col("email").str.contains("@", literal=True).fill_null(False)
    ).height,
)
print("money failures:", result.filter(pl.col("amount").is_null())["money"].to_list())

TextExpected output

[{'email': 'ada@example.com', 'money': ' USD 1,250.50 ', 'reference': 'INV-0024', 'amount': Decimal('1250.50'), 'invoice_number': '0024'}, {'email': 'no-email', 'money': 'USD 20.00', 'reference': 'INV-0025', 'amount': Decimal('20.00'), 'invoice_number': '0025'}, {'email': None, 'money': 'unknown', 'reference': 'other', 'amount': None, 'invoice_number': None}]
invalid email rows: 2
money failures: ['unknown']

Input formats

This parser uses comma thousands separators and a decimal point. Values such as 1.250,50 need another format. strip_chars trims edges. A contains-@ flag is not email validation. Use len_chars for characters and len_bytes for encoded bytes.

Official reference: Clean strings and messy exports.

11 / 50

Parse dates and calculate intervals#

A Date is a calendar day; a Datetime includes time; a Duration measures elapsed time.

Parsing interprets text according to a format. str.to_date uses calendar dates; str.to_datetime uses timestamps. Prefer an explicit format for exports, especially day/month ambiguity. The .dt namespace provides year, month, weekday, truncation, formatting, and duration calculations. A naive timestamp has no time zone, so its location and offset are not known.

PythonAccept two documented date formats and retain failures

import polars as pl

df = pl.DataFrame(
    {
        "raw": ["2026-09-01", "02/09/2026", "broken"],
        "shipped": ["2026-09-03", "2026-09-05", None],
    }
)
parsed = df.with_columns(
    pl.coalesce(
        pl.col("raw").str.to_date("%Y-%m-%d", strict=False),
        pl.col("raw").str.to_date("%d/%m/%Y", strict=False),
    ).alias("ordered"),
    pl.col("shipped").str.to_date("%Y-%m-%d").alias("shipped"),
)
result = parsed.with_columns(
    (pl.col("shipped") - pl.col("ordered")).dt.total_days().alias("delivery_days")
)
print(result.to_dicts())
print("bad dates:", result.filter(pl.col("ordered").is_null())["raw"].to_list())

TextExpected output

[{'raw': '2026-09-01', 'shipped': datetime.date(2026, 9, 3), 'ordered': datetime.date(2026, 9, 1), 'delivery_days': 2}, {'raw': '02/09/2026', 'shipped': datetime.date(2026, 9, 5), 'ordered': datetime.date(2026, 9, 2), 'delivery_days': 3}, {'raw': 'broken', 'shipped': None, 'ordered': None, 'delivery_days': None}]
bad dates: ['broken']

Calendar units and elapsed units differ

One calendar day can contain 23 or 25 hours during a daylight-saving change. Use calendar operations for local reporting dates and UTC timestamp subtraction for elapsed durations. Datetime units (ms, us, ns) express timestamp precision; converting to Python dictionaries can lose nanosecond precision.

Official reference: Parse dates and calculate intervals.

12 / 50

Time zones and daylight-saving changes#

Localize a wall-clock time; convert an already known instant to another zone.

A time zone, such as Asia/Dhaka or America/New_York, describes local clock rules. An offset such as +06:00 is the clock's difference from UTC at one time. UTC is the common reference used to store instants. A time-zone-aware column uses one zone for the whole column.

replace_time_zone interprets the displayed wall-clock values in a zone and can change the represented instant. convert_time_zone changes the display zone while preserving the instant. Daylight saving moves clocks seasonally: a fall-back time can occur twice (ambiguous), and a spring-forward time can never occur (nonexistent). Do not guess which event occurred from an ambiguous local timestamp alone.

PythonConvert UTC to Dhaka and handle two New York DST edge cases

import polars as pl

utc = pl.Series("ts", ["2026-09-01T12:00:00+00:00"]).str.to_datetime(time_zone="UTC")
print(
    "Dhaka:",
    utc.dt.convert_time_zone("Asia/Dhaka").dt.strftime("%Y-%m-%d %H:%M %z").to_list(),
)
fall = pl.Series("ts", ["2025-11-02 01:30"]).str.to_datetime("%Y-%m-%d %H:%M")
early = fall.dt.replace_time_zone("America/New_York", ambiguous="earliest")
late = fall.dt.replace_time_zone("America/New_York", ambiguous="latest")
print(
    "fall UTC instants:",
    early.dt.convert_time_zone("UTC").dt.strftime("%H:%M").to_list(),
    late.dt.convert_time_zone("UTC").dt.strftime("%H:%M").to_list(),
)
spring = pl.Series("ts", ["2025-03-09 02:30"]).str.to_datetime("%Y-%m-%d %H:%M")
print(
    "spring null:",
    spring.dt.replace_time_zone("America/New_York", non_existent="null").to_list(),
)
try:
    fall.dt.replace_time_zone("America/New_York")
except pl.exceptions.ComputeError:
    print("default: ambiguous local time rejected")

TextExpected output

Dhaka: ['2026-09-01 18:00 +0600']
fall UTC instants: ['05:30'] ['06:30']
spring null: [None]
default: ambiguous local time rejected

What changes

12:00 UTC is 18:00 in Dhaka. New York’s repeated 01:30 can represent two UTC instants; earliest and latest choose between them. The configured spring-gap policy returns null. Keep rejected values and source offsets for diagnosis.

Official reference: Time zones and daylight-saving changes.

Select columns and build expressions#

Add columns and make conditional decisions#

Filter rows and sort results#

Types, casting, and exact money#

Null, NaN, and missing-value policies#

Duplicates and the latest customer record#

Clean strings and messy exports#

Parse dates and calculate intervals#

Time zones and daylight-saving changes#

Help improve CodeRef