1. Core concepts¶

Veridelta checks two datasets for logical equality, not physical equality. Physical equality means the same names, types, and values. Logical equality is the relation you declare: after a rename, a currency strip, and a sentinel, two exports of the same accounts hold the same data.

This tutorial uses the Python API to declare a comparison, read its DiffResult, and isolate a real discrepancy. The YAML and CLI tutorial runs a comparison from a file, and Pushdown covers comparing inside a warehouse.

Open this tutorial in Google Colab. Its first cell installs Veridelta there.

In [ ]:

# A kernel without Veridelta, such as Colab's, installs the release this tutorial shows.
import importlib.util
import subprocess
import sys

if importlib.util.find_spec("veridelta") is None:
    subprocess.run(
        [sys.executable, "-m", "pip", "install", "--quiet", "veridelta==0.35.1"],
        check=True,
    )

# A kernel without Veridelta, such as Colab's, installs the release this tutorial shows. import importlib.util import subprocess import sys if importlib.util.find_spec("veridelta") is None: subprocess.run( [sys.executable, "-m", "pip", "install", "--quiet", "veridelta==0.35.1"], check=True, )

1. Two frames that hold the same accounts¶

The source is a legacy export, and the target is a warehouse table. They differ in the key's name, the currency symbols in balance, a sentinel for a missing status, and a two-cent rounding difference. Polars compares them physically:

In [ ]:

import polars as pl

from veridelta import DiffConfig, DiffEngine, DiffRule

source = pl.DataFrame(
    {
        "legacy_id": [1, 2, 3],
        "status": ["Active", "N/A", "Closed"],
        "balance": ["$10.00", "$20.50", "$5.00"],
        "tier": ["Premium", "Standard", "Enterprise"],
    }
)
target = pl.DataFrame(
    {
        "user_id": [1, 2, 3],
        "status": ["Active", None, "Closed"],
        "balance": [10.00, 20.48, 5.00],
        "tier": ["Premium", "Standard", "Enterprise"],
    }
)
print(f"Physical equality: {source.equals(target)}")

# Output:
# Physical equality: False

import polars as pl from veridelta import DiffConfig, DiffEngine, DiffRule source = pl.DataFrame( { "legacy_id": [1, 2, 3], "status": ["Active", "N/A", "Closed"], "balance": ["$10.00", "$20.50", "$5.00"], "tier": ["Premium", "Standard", "Enterprise"], } ) target = pl.DataFrame( { "user_id": [1, 2, 3], "status": ["Active", None, "Closed"], "balance": [10.00, 20.48, 5.00], "tier": ["Premium", "Standard", "Enterprise"], } ) print(f"Physical equality: {source.equals(target)}") # Output: # Physical equality: False

2. Declare the comparison¶

A DiffConfig names the primary keys, and each DiffRule says how to treat its columns. Rules apply in a fixed order of nine stages, not the order they are written in. The cast comes after the regular expression, so cast_to reads balance once regex_replace has removed the $. Transform order lists every stage.

With these three rules, the two frames match:

In [ ]:

config = DiffConfig(
    primary_keys=["user_id"],
    rules=[
        DiffRule(column_names=["legacy_id"], rename_to="user_id"),
        DiffRule(
            column_names=["balance"],
            regex_replace={"\\$": ""},
            cast_to="Float64",
            absolute_tolerance=0.05,
        ),
        DiffRule(column_names=["status"], null_values=["N/A"], treat_null_as_equal=True),
    ],
)
result = DiffEngine(config, source.lazy(), target.lazy()).run()
print(result.summary.report_summary)

# Output:
# Veridelta Execution Summary
# ===========================
# Status:        PASSED (Perfect Match)
# Match Rate:    100.0%
# Source Rows:   3
# Target Rows:   3
# Volume Shift:  +0 rows
#
# Row-Level Discrepancies:
# ---------------------------
# Added:         0
# Removed:       0
# Changed:       0
# Total Issues:  0

config = DiffConfig( primary_keys=["user_id"], rules=[ DiffRule(column_names=["legacy_id"], rename_to="user_id"), DiffRule( column_names=["balance"], regex_replace={"\\$": ""}, cast_to="Float64", absolute_tolerance=0.05, ), DiffRule(column_names=["status"], null_values=["N/A"], treat_null_as_equal=True), ], ) result = DiffEngine(config, source.lazy(), target.lazy()).run() print(result.summary.report_summary) # Output: # Veridelta Execution Summary # =========================== # Status: PASSED (Perfect Match) # Match Rate: 100.0% # Source Rows: 3 # Target Rows: 3 # Volume Shift: +0 rows # # Row-Level Discrepancies: # --------------------------- # Added: 0 # Removed: 0 # Changed: 0 # Total Issues: 0

3. Read the DiffResult¶

run() returns a DiffResult. Its summary is the verdict, ready to serialize as JSON. added, removed, and changed hold the rows behind each count. get_mismatches(column) narrows the changed rows to one column, with both values side by side:

In [ ]:

print("compared:", result.compared_columns)
print(
    f"added={result.added.height}  removed={result.removed.height}  changed={result.changed.height}"
)
print(result.get_mismatches("balance"))

# Output:
# compared: ('status', 'balance', 'tier')
# added=0  removed=0  changed=0
# shape: (0, 3)
# ┌─────────┬────────────────┬────────────────┐
# │ user_id ┆ balance_source ┆ balance_target │
# │ ---     ┆ ---            ┆ ---            │
# │ i64     ┆ f64            ┆ f64            │
# ╞═════════╪════════════════╪════════════════╡
# └─────────┴────────────────┴────────────────┘

print("compared:", result.compared_columns) print( f"added={result.added.height} removed={result.removed.height} changed={result.changed.height}" ) print(result.get_mismatches("balance")) # Output: # compared: ('status', 'balance', 'tier') # added=0 removed=0 changed=0 # shape: (0, 3) # ┌─────────┬────────────────┬────────────────┐ # │ user_id ┆ balance_source ┆ balance_target │ # │ --- ┆ --- ┆ --- │ # │ i64 ┆ f64 ┆ f64 │ # ╞═════════╪════════════════╪════════════════╡ # └─────────┴────────────────┴────────────────┘

4. Global defaults and per-column rules¶

A default_* setting on DiffConfig applies to every column, and a DiffRule that names a column overrides it there. Both configurations below forgive the two-cent drift: one with a default tolerance of 0.05, the other with a tighter 0.03 on balance alone:

In [ ]:

inherited = DiffConfig(
    primary_keys=["user_id"],
    default_absolute_tolerance=0.05,
    default_null_values=["N/A"],
    rules=[
        DiffRule(column_names=["legacy_id"], rename_to="user_id"),
        DiffRule(column_names=["balance"], regex_replace={"\\$": ""}, cast_to="Float64"),
    ],
)
print(
    "inherited default 0.05:",
    DiffEngine(inherited, source.lazy(), target.lazy()).run().summary.is_match,
)

tighter = DiffConfig(
    primary_keys=["user_id"],
    default_absolute_tolerance=0.05,
    default_null_values=["N/A"],
    rules=[
        DiffRule(column_names=["legacy_id"], rename_to="user_id"),
        DiffRule(
            column_names=["balance"],
            regex_replace={"\\$": ""},
            cast_to="Float64",
            absolute_tolerance=0.03,
        ),
    ],
)
print(
    "per-column 0.03:",
    DiffEngine(tighter, source.lazy(), target.lazy()).run().summary.is_match,
)

# Output:
# inherited default 0.05: True
# per-column 0.03: True

inherited = DiffConfig( primary_keys=["user_id"], default_absolute_tolerance=0.05, default_null_values=["N/A"], rules=[ DiffRule(column_names=["legacy_id"], rename_to="user_id"), DiffRule(column_names=["balance"], regex_replace={"\\$": ""}, cast_to="Float64"), ], ) print( "inherited default 0.05:", DiffEngine(inherited, source.lazy(), target.lazy()).run().summary.is_match, ) tighter = DiffConfig( primary_keys=["user_id"], default_absolute_tolerance=0.05, default_null_values=["N/A"], rules=[ DiffRule(column_names=["legacy_id"], rename_to="user_id"), DiffRule( column_names=["balance"], regex_replace={"\\$": ""}, cast_to="Float64", absolute_tolerance=0.03, ), ], ) print( "per-column 0.03:", DiffEngine(tighter, source.lazy(), target.lazy()).run().summary.is_match, ) # Output: # inherited default 0.05: True # per-column 0.03: True

5. Isolate a real discrepancy¶

Without a tolerance, the two-cent drift on row 2 is a mismatch. get_mismatches keeps only the balance column:

In [ ]:

strict = DiffConfig(
    primary_keys=["user_id"],
    rules=[
        DiffRule(column_names=["legacy_id"], rename_to="user_id"),
        DiffRule(column_names=["balance"], regex_replace={"\\$": ""}, cast_to="Float64"),
        DiffRule(column_names=["status"], null_values=["N/A"], treat_null_as_equal=True),
    ],
)
drifted = DiffEngine(strict, source.lazy(), target.lazy()).run()
print(drifted.summary.report_summary)
print(drifted.get_mismatches("balance"))

# Output:
# Veridelta Execution Summary
# ===========================
# Status:        FAILED
# Match Rate:    66.67%
# Source Rows:   3
# Target Rows:   3
# Volume Shift:  +0 rows
#
# Row-Level Discrepancies:
# ---------------------------
# Added:         0
# Removed:       0
# Changed:       1
# Total Issues:  1
#
# Top Column-Level Drifts:
# ---------------------------
# - balance: 1 mismatch
#
# shape: (1, 3)
# ┌─────────┬────────────────┬────────────────┐
# │ user_id ┆ balance_source ┆ balance_target │
# │ ---     ┆ ---            ┆ ---            │
# │ i64     ┆ f64            ┆ f64            │
# ╞═════════╪════════════════╪════════════════╡
# │ 2       ┆ 20.5           ┆ 20.48          │
# └─────────┴────────────────┴────────────────┘

strict = DiffConfig( primary_keys=["user_id"], rules=[ DiffRule(column_names=["legacy_id"], rename_to="user_id"), DiffRule(column_names=["balance"], regex_replace={"\\$": ""}, cast_to="Float64"), DiffRule(column_names=["status"], null_values=["N/A"], treat_null_as_equal=True), ], ) drifted = DiffEngine(strict, source.lazy(), target.lazy()).run() print(drifted.summary.report_summary) print(drifted.get_mismatches("balance")) # Output: # Veridelta Execution Summary # =========================== # Status: FAILED # Match Rate: 66.67% # Source Rows: 3 # Target Rows: 3 # Volume Shift: +0 rows # # Row-Level Discrepancies: # --------------------------- # Added: 0 # Removed: 0 # Changed: 1 # Total Issues: 1 # # Top Column-Level Drifts: # --------------------------- # - balance: 1 mismatch # # shape: (1, 3) # ┌─────────┬────────────────┬────────────────┐ # │ user_id ┆ balance_source ┆ balance_target │ # │ --- ┆ --- ┆ --- │ # │ i64 ┆ f64 ┆ f64 │ # ╞═════════╪════════════════╪════════════════╡ # │ 2 ┆ 20.5 ┆ 20.48 │ # └─────────┴────────────────┴────────────────┘

6. Map categorical codes¶

The modern system stores tier codes, such as PRM for Premium. A value_map translates source values before they are compared. It is a declared correspondence, not a fuzzy match:

In [ ]:

coded = target.with_columns(tier=pl.Series(["PRM", "STD", "ENT"]))
print(
    "without map, changed:",
    DiffEngine(config, source.lazy(), coded.lazy()).run().summary.changed_count,
)

mapped = DiffConfig(
    primary_keys=["user_id"],
    rules=[
        *config.rules,
        DiffRule(
            column_names=["tier"],
            value_map={"Premium": "PRM", "Standard": "STD", "Enterprise": "ENT"},
        ),
    ],
)
print(
    "with map, changed:",
    DiffEngine(mapped, source.lazy(), coded.lazy()).run().summary.changed_count,
)

# Output:
# without map, changed: 3
# with map, changed: 0

coded = target.with_columns(tier=pl.Series(["PRM", "STD", "ENT"])) print( "without map, changed:", DiffEngine(config, source.lazy(), coded.lazy()).run().summary.changed_count, ) mapped = DiffConfig( primary_keys=["user_id"], rules=[ *config.rules, DiffRule( column_names=["tier"], value_map={"Premium": "PRM", "Standard": "STD", "Enterprise": "ENT"}, ), ], ) print( "with map, changed:", DiffEngine(mapped, source.lazy(), coded.lazy()).run().summary.changed_count, ) # Output: # without map, changed: 3 # with map, changed: 0

Writing a map by hand needs someone who knows the codes. propose_value_maps drafts one from the data instead. It pairs rows on their keys, as run does, and proposes each source value for the target value its rows agree on, with the evidence. The veridelta crosswalk command prints the same proposals as YAML rules; see Proposing a value map.

Each tier appears once in this data, so the default min_support of 5 proposes nothing. On real data, keep the default: it stops a coincidence from becoming a proposal. Here, min_support=1 shows what a proposal holds:

In [ ]:

engine = DiffEngine(config, source.lazy(), coded.lazy())
print("default support:", engine.propose_value_maps())

(proposal,) = engine.propose_value_maps(min_support=1)
print(f"{proposal.column}: {proposal.value_map}")
for entry in proposal.entries:
    print(
        f"  {entry.source_value!r} -> {entry.target_value!r}: "
        f"{entry.agreeing_rows} of {entry.rows} rows ({entry.confidence:.0%})"
    )

# Output:
# default support: []
# tier: {'Enterprise': 'ENT', 'Premium': 'PRM', 'Standard': 'STD'}
#   'Enterprise' -> 'ENT': 1 of 1 rows (100%)
#   'Premium' -> 'PRM': 1 of 1 rows (100%)
#   'Standard' -> 'STD': 1 of 1 rows (100%)

engine = DiffEngine(config, source.lazy(), coded.lazy()) print("default support:", engine.propose_value_maps()) (proposal,) = engine.propose_value_maps(min_support=1) print(f"{proposal.column}: {proposal.value_map}") for entry in proposal.entries: print( f" {entry.source_value!r} -> {entry.target_value!r}: " f"{entry.agreeing_rows} of {entry.rows} rows ({entry.confidence:.0%})" ) # Output: # default support: [] # tier: {'Enterprise': 'ENT', 'Premium': 'PRM', 'Standard': 'STD'} # 'Enterprise' -> 'ENT': 1 of 1 rows (100%) # 'Premium' -> 'PRM': 1 of 1 rows (100%) # 'Standard' -> 'STD': 1 of 1 rows (100%)

The YAML and CLI tutorial runs a comparison from a configuration file. Configuration and Rules describe every setting.