Veridelta Quickstart¶

Veridelta is a high-performance semantic diffing engine built on Polars. Standard binary equality checks often fail in production pipelines due to trivial transformation artifacts like floating-point noise, rounding algorithm mismatches, or string formatting variance.

Veridelta enables data engineers to define logical equality through declarative configuration, ensuring resilient data validation at scale.

1. The Challenge: Physical vs. Logical Equality¶

Data migration between disparate computational environments (e.g., Python application layers to SQL data warehouses) introduces expected transformation artifacts. Two common examples include:

1. Rounding Algorithm Mismatches: Python utilizes "Banker's Rounding" (round half to even; 10.445 evaluates to 10.44). SQL systems typically utilize standard rounding (round half away from zero; 10.445 evaluates to 10.45). This generates an artificial $0.01 variance.
2. String Casing & Padding: Legacy systems may export Title Case ("Pending"), while modern warehouses enforce standardized, padded UPPERCASE (" PENDING ").

These datasets are logically identical, yet standard physical equality checks evaluate to false.

In [ ]:

import polars as pl

import veridelta as vd
from veridelta import DiffConfig, DiffEngine, DiffRule

# Source System (Legacy Python - Banker's Rounding, Title Case)
src_df = pl.DataFrame(
    {
        "transaction_id": ["TXN-001", "TXN-002", "TXN-003"],
        "tax_amount": [10.44, 250.50, 300.00],  # 10.445 rounded half-to-even
        "status": ["Cleared", "Pending", "Failed"],
    }
)

# Target System (Modern SQL - Standard Rounding, UPPERCASE with padding)
tgt_df = pl.DataFrame(
    {
        "transaction_id": ["TXN-001", "TXN-002", "TXN-003"],
        "tax_amount": [10.45, 250.50, 300.00],  # 10.445 rounded half-up
        "status": [" CLEARED ", " PENDING ", " FAILED "],
    }
)

is_physically_equal = src_df.equals(tgt_df)
print(f"Strict Physical Equality: {is_physically_equal}")

# Output:
# Strict Physical Equality: False

import polars as pl import veridelta as vd from veridelta import DiffConfig, DiffEngine, DiffRule # Source System (Legacy Python - Banker's Rounding, Title Case) src_df = pl.DataFrame( { "transaction_id": ["TXN-001", "TXN-002", "TXN-003"], "tax_amount": [10.44, 250.50, 300.00], # 10.445 rounded half-to-even "status": ["Cleared", "Pending", "Failed"], } ) # Target System (Modern SQL - Standard Rounding, UPPERCASE with padding) tgt_df = pl.DataFrame( { "transaction_id": ["TXN-001", "TXN-002", "TXN-003"], "tax_amount": [10.45, 250.50, 300.00], # 10.445 rounded half-up "status": [" CLEARED ", " PENDING ", " FAILED "], } ) is_physically_equal = src_df.equals(tgt_df) print(f"Strict Physical Equality: {is_physically_equal}") # Output: # Strict Physical Equality: False

2. Defining Semantic Boundaries¶

Veridelta resolves physical variance by establishing semantic boundaries. The DiffConfig object declaratively defines acceptable tolerances and normalization rules, masking expected migration artifacts without ignoring genuine data corruption.

In [ ]:

# Define the semantic comparison boundaries
config = DiffConfig(
    primary_keys=["transaction_id"],
    # Global Policies: Applied to all columns (unless overridden)
    # Allow a maximum of $0.01 variance to account for Banker's Rounding differences
    default_absolute_tolerance=0.01,
    # Global Whitespace Policy: Strip all leading/trailing whitespace
    default_whitespace_mode="both",
    # Granular Overrides: Column-specific normalization
    rules=[
        DiffRule(
            column_names=["status"],
            case_insensitive=True,  # Ensure 'Pending' matches ' PENDING '
        )
    ],
)

# Execute the comparison using the aligned DataFrames
engine = DiffEngine(config, src_df, tgt_df)
summary = engine.run()

print(f"Match Status: {summary.is_match}")
print(f"Changed Rows: {summary.changed_count}")

# Output:
# Match Status: True
# Changed Rows: 0

# Define the semantic comparison boundaries config = DiffConfig( primary_keys=["transaction_id"], # Global Policies: Applied to all columns (unless overridden) # Allow a maximum of $0.01 variance to account for Banker's Rounding differences default_absolute_tolerance=0.01, # Global Whitespace Policy: Strip all leading/trailing whitespace default_whitespace_mode="both", # Granular Overrides: Column-specific normalization rules=[ DiffRule( column_names=["status"], case_insensitive=True, # Ensure 'Pending' matches ' PENDING ' ) ], ) # Execute the comparison using the aligned DataFrames engine = DiffEngine(config, src_df, tgt_df) summary = engine.run() print(f"Match Status: {summary.is_match}") print(f"Changed Rows: {summary.changed_count}") # Output: # Match Status: True # Changed Rows: 0

3. Scale & Performance: Simulating Pipeline Drift¶

Veridelta is built on Polars' Rust backend, enabling semantic normalization to scale linearly across available CPU cores.

The following benchmark loads a sample of the NYC Yellow Taxi dataset, injects programmatic ETL drift, and executes a multi-threaded semantic comparison.

In [ ]:

# Load and deduplicate to ensure primary key integrity
src_df = vd.datasets.load_nyc_taxi().unique(  # pyright: ignore[reportUnknownMemberType]
    subset=["VendorID", "tpep_pickup_datetime"]
)

# Simulate ETL drift on the cleaned target:
# - Rounding noise on the fare (+$0.002)
# - A 0.5% calculation fluctuation on the tip amount (e.g., exchange rate drift)
# - Inconsistent string formatting
tgt_df = src_df.with_columns(  # pyright: ignore[reportUnknownMemberType]
    fare_amount=pl.col("fare_amount") + 0.002,
    tip_amount=pl.col("tip_amount") * 1.005,
    store_and_fwd_flag=pl.col("store_and_fwd_flag").str.to_uppercase(),
)

config = DiffConfig(
    primary_keys=["VendorID", "tpep_pickup_datetime"],
    # Global Policy: Ignore micro-penny rounding errors across the board
    default_absolute_tolerance=0.01,
    rules=[
        DiffRule(column_names=["store_and_fwd_flag"], case_insensitive=True),
        DiffRule(
            column_names=["tip_amount"],
            relative_tolerance=0.01,  # Accept up to 1% relative drift on tips
        ),
    ],
)

engine = DiffEngine(config, src_df, tgt_df)
summary = engine.run()

print(f"{' VERIDELTA EXECUTION SUMMARY ':=^50}")
print(f"Match Status:     {'SUCCESS' if summary.is_match else 'FAILED'}")
print(f"Total Rows:       {summary.total_rows_source:,}")
print(f"Semantic Diffs:   {summary.changed_count:,}")
print("=" * 50)

# Output:
# ========== VERIDELTA EXECUTION SUMMARY ===========
# Match Status:     SUCCESS
# Total Rows:       908
# Semantic Diffs:   0
# ==================================================

# Load and deduplicate to ensure primary key integrity src_df = vd.datasets.load_nyc_taxi().unique( # pyright: ignore[reportUnknownMemberType] subset=["VendorID", "tpep_pickup_datetime"] ) # Simulate ETL drift on the cleaned target: # - Rounding noise on the fare (+$0.002) # - A 0.5% calculation fluctuation on the tip amount (e.g., exchange rate drift) # - Inconsistent string formatting tgt_df = src_df.with_columns( # pyright: ignore[reportUnknownMemberType] fare_amount=pl.col("fare_amount") + 0.002, tip_amount=pl.col("tip_amount") * 1.005, store_and_fwd_flag=pl.col("store_and_fwd_flag").str.to_uppercase(), ) config = DiffConfig( primary_keys=["VendorID", "tpep_pickup_datetime"], # Global Policy: Ignore micro-penny rounding errors across the board default_absolute_tolerance=0.01, rules=[ DiffRule(column_names=["store_and_fwd_flag"], case_insensitive=True), DiffRule( column_names=["tip_amount"], relative_tolerance=0.01, # Accept up to 1% relative drift on tips ), ], ) engine = DiffEngine(config, src_df, tgt_df) summary = engine.run() print(f"{' VERIDELTA EXECUTION SUMMARY ':=^50}") print(f"Match Status: {'SUCCESS' if summary.is_match else 'FAILED'}") print(f"Total Rows: {summary.total_rows_source:,}") print(f"Semantic Diffs: {summary.changed_count:,}") print("=" * 50) # Output: # ========== VERIDELTA EXECUTION SUMMARY =========== # Match Status: SUCCESS # Total Rows: 908 # Semantic Diffs: 0 # ==================================================

Architectural Highlights¶

Veridelta is engineered for integration into high-throughput data pipelines and CI/CD validation gates.

• Zero-Import Configuration: Semantic modes utilize literal strings, enabling configuration via YAML/JSON without requiring Python imports.
• Strict Type Validation: Built on Pydantic V2, ensuring configuration logic is strictly validated before expensive I/O operations begin.
• Hardware-Accelerated Execution: Direct integration with Polars' Rust engine ensures that normalization and row-level diffing scales natively with available hardware.