Advanced Heuristics & Data Normalization¶
While Veridelta supports declarative YAML configurations for CI/CD pipelines, its native Python API provides programmatic flexibility for complex data environments. This notebook demonstrates how to construct advanced heuristics, deterministic value mapping, and rule chaining directly within Python using Veridelta's configuration models.
Operating directly on Polars DataFrames via the Python API is ideal for interactive exploratory data analysis, dynamic rule generation, and integration into existing Python-based orchestrators.
1. Simulating Systemic Formatting Discrepancies¶
During cross-platform migrations (e.g., legacy CRM to a modern SaaS platform), categorical enumerations and string formats routinely diverge.
The following DataFrames simulate a scenario where the target system utilizes abbreviated tier codes and strips all non-numeric characters from phone numbers.
import polars as pl
from veridelta.engine import DiffEngine
from veridelta.models import DiffConfig, DiffRule
# Source System: Legacy CRM (Verbose statuses, formatted phone numbers)
src_crm = pl.DataFrame(
{
"customer_id": ["CUST-801", "CUST-802"],
"subscription_tier": ["Premium", "Enterprise"],
"contact_number": ["(555) 123-4567", "+1 555-987-6543"],
}
)
# Target System: Modern Platform (Abbreviated tiers, numeric-only phones)
tgt_saas = pl.DataFrame(
{
"customer_id": ["CUST-801", "CUST-802"],
"subscription_tier": ["PRM", "ENT"],
"contact_number": ["5551234567", "15559876543"],
}
)
print("In-memory data artifacts initialized.")
# Output:
# In-memory data artifacts initialized.
The Baseline Failure¶
If we run a comparison using standard strict equality, the engine correctly flags that 100% of the rows contain semantic discrepancies due to the formatting drift.
baseline_config = DiffConfig(primary_keys=["customer_id"])
engine_baseline = DiffEngine(baseline_config, src_crm, tgt_saas)
print(engine_baseline.run().report_summary)
# Output:
# Veridelta Execution Summary
# ===========================
# Status: FAILED
# Match Rate: 0.0%
# Source Rows: 2
# Target Rows: 2
# Volume Shift: +0 rows
#
# Discrepancy Breakdown:
# ---------------------------
# Added: 0
# Removed: 0
# Changed: 2
# Total Issues: 2
#
# Top Column-Level Drifts:
# ---------------------------
# - subscription_tier: 2 mismatches
# - contact_number: 2 mismatches
2. Programmatic Transformation Rules¶
Value mappings and regular expression replacements execute natively within the Polars Rust engine, ensuring optimal execution speed. By instantiating DiffRule objects, we instruct the engine to sanitize the data prior to evaluating equality, bridging the gap between the two systems.
# Construct the configuration
mapping_config = DiffConfig(
primary_keys=["customer_id"],
rules=[
DiffRule(
column_names=["subscription_tier"], value_map={"Premium": "PRM", "Enterprise": "ENT"}
),
DiffRule(
column_names=["contact_number"],
regex_replace={
"[^0-9]": "" # Strip all non-numeric characters
},
),
],
)
engine = DiffEngine(mapping_config, src_crm, tgt_saas)
summary = engine.run()
print(summary.report_summary)
# Output:
# Veridelta Execution Summary
# ===========================
# Status: PASSED (Perfect Match)
# Match Rate: 100.0%
# Source Rows: 2
# Target Rows: 2
# Volume Shift: +0 rows
#
# Discrepancy Breakdown:
# ---------------------------
# Added: 0
# Removed: 0
# Changed: 0
# Total Issues: 0
3. Rule Chaining & Type Coercion¶
Real-world migrations often feature compounding discrepancies. For example, a legacy billing system might export invoice amounts as formatted strings (e.g., "$10.00"), while the modern system exports raw numeric values (e.g., 10.0).
If we compare these without rules, they immediately fail due to both Type mismatches (String vs Float) and Value mismatches ($10.00 vs 10.0).
# Source: Legacy Billing (Strings with currency symbols)
src_billing = pl.DataFrame(
{
"invoice_id": ["INV-001", "INV-002"],
"amount": ["$10.00", "$99.99"],
}
)
# Target: Modern Billing (Raw numbers, missing the penny on INV-002 due to a known tax bug)
tgt_billing = pl.DataFrame(
{
"invoice_id": ["INV-001", "INV-002"],
"amount": [10.00, 99.98],
}
)
# Baseline failure: Veridelta gracefully handles the String vs Float mismatch
# (preventing a pipeline crash) and correctly flags 100% of the rows as semantic failures.
baseline_billing = DiffEngine(DiffConfig(primary_keys=["invoice_id"]), src_billing, tgt_billing)
print(baseline_billing.run().report_summary)
# Output:
# Veridelta Execution Summary
# ===========================
# Status: FAILED
# Match Rate: 0.0%
# Source Rows: 2
# Target Rows: 2
# Volume Shift: +0 rows
#
# Discrepancy Breakdown:
# ---------------------------
# Added: 0
# Removed: 0
# Changed: 2
# Total Issues: 2
#
# Top Column-Level Drifts:
# ---------------------------
# - amount: 2 mismatches
Applying the Chain¶
Instead of writing complex row-by-row lambda functions in Pandas, Veridelta allows you to chain rules declaratively.
In the following configuration, we define a single DiffRule that strips the currency symbol, casts the strings to Polars Float64 objects, and applies an absolute tolerance to handle penny-rounding differences—all executed in sequence within the optimized C-level pipeline.
# Chain rules: Regex Clean -> Type Cast -> Numeric Tolerance
chained_config = DiffConfig(
primary_keys=["invoice_id"],
rules=[
DiffRule(
column_names=["amount"],
regex_replace={"\\$": ""}, # 1. Strip the dollar sign
cast_to="Float64", # 2. Coerce to Float for mathematical comparison
absolute_tolerance=0.01, # 3. Forgive the 1-cent tax rounding variance
)
],
)
engine_billing = DiffEngine(chained_config, src_billing, tgt_billing)
print(engine_billing.run().report_summary)
# Output:
# Veridelta Execution Summary
# ===========================
# Status: PASSED (Perfect Match)
# Match Rate: 100.0%
# Source Rows: 2
# Target Rows: 2
# Volume Shift: +0 rows
#
# Discrepancy Breakdown:
# ---------------------------
# Added: 0
# Removed: 0
# Changed: 0
# Total Issues: 0
4. Schema Asymmetry and Data Quality¶
Schema instability is a routine aspect of system evolution. Veridelta manages naming drifts and legacy data representations as semantic equivalents rather than execution failures.
- Schema Mapping:
rename_toaligns disparate field names between datasets, consolidating duplicated knowledge into a single representation. - Data Quality:
null_valuesandtreat_null_as_equalprovide an authoritative definition for missing data. This centralizes intent and prevents maintenance nightmares caused by implicit guesses or fragmented validation logic.
This declarative approach establishes a Single Source of Truth (SSoT).
# Scenario: Target system renamed the legacy ID and uses actual NULLs
src_legacy = pl.DataFrame({"id": [1, 2, 3], "status": ["Active", "UNKNOWN", "Active"]})
tgt_modern = pl.DataFrame(
{
"user_id": [1, 2, 3], # The authoritative column name (Target/Modern)
"status": ["Active", None, "Active"],
}
)
asymmetry_config = DiffConfig(
primary_keys=["user_id"], # Reference the authoritative Target name
rules=[
DiffRule(
column_names=["id"],
rename_to="user_id", # Source will be renamed to match Target
),
DiffRule(column_names=["status"], null_values=["UNKNOWN"], treat_null_as_equal=True),
],
)
summary = DiffEngine(asymmetry_config, src_legacy, tgt_modern).run()
print(summary.report_summary)
# Output:
# Veridelta Execution Summary
# ===========================
# Status: PASSED (Perfect Match)
# Match Rate: 100.0%
# Source Rows: 3
# Target Rows: 3
# Volume Shift: +0 rows
#
# Row-Level Discrepancies:
# ---------------------------
# Added: 0
# Removed: 0
# Changed: 0
# Total Issues: 0
Conclusion: Architecting for Maintainability¶
Veridelta’s programmatic API abstracts the boilerplate of cross-system data validation. By encapsulating semantic transformations, schema alignment, and tolerances within declarative DiffRule models, you prevent validation logic from fragmenting across custom Pandas scripts or SQL queries.
Key Architectural Takeaways:
- Single Source of Truth: Centralize migration assumptions (renames, null coercion, value maps) into version-controlled configuration objects.
- Performance at Scale: Pushing operations down to the Polars Rust engine ensures validation scales efficiently out-of-core without memory bottlenecks.
- Pipeline Integration: Use the Python API to dynamically generate configurations within orchestrators (e.g., Apache Airflow, Dagster) to gate downstream deployments based on strict data quality thresholds.
For automated CI/CD workflows, these Python models map 1:1 with Veridelta's YAML specification, allowing a seamless transition from exploratory notebook analysis to strict, automated pipeline execution.