From drift to rules¶
A first run of a rewritten pipeline against the old one often fails on most of its rows. Most of that drift is noise, and each kind has one cause, such as rounding, capital letters, or a code for each value. This guide finds those causes with veridelta suggest and veridelta crosswalk, and turns each into a rule you review. Then it accepts a change made on purpose with a baseline file. What still fails is drift that no rule and no baseline explains.
The guide writes two small CSV files, so it reads no data and needs no network. Run it where Veridelta is installed, such as after pip install veridelta.
1. Two exports¶
The legacy system and its rewrite export the same 40 accounts. The rewrite writes each region in capitals, codes each status as one letter, and writes N/A where the legacy file has no note. Some of its balances differ in the third decimal. It leaves out account 40, which was closed and purged on purpose. And it moves account 17 from the south region to the east, which nobody asked for:
import polars as pl
ids = range(1, 41)
legacy = pl.DataFrame(
{
"account_id": list(ids),
"region": [("north", "south", "east", "west")[i % 4] for i in ids],
"status": [("active", "paused", "closed")[i % 3] for i in ids],
"balance": [100 + i * 12.5 for i in ids],
"note": [None if i % 5 == 0 else f"renewal {2026 + i % 3}" for i in ids],
}
)
rewrite = legacy.filter(pl.col("account_id") != 40).with_columns(
pl.when(pl.col("account_id") == 17)
.then(pl.lit("EAST"))
.otherwise(pl.col("region").str.to_uppercase())
.alias("region"),
pl.col("status").replace_strict({"active": "A", "paused": "P", "closed": "C"}),
# Every third balance gains up to 0.004.
pl.when(pl.col("account_id") % 3 == 0)
.then(pl.col("balance") + (pl.col("account_id") % 4 + 1) / 1000)
.otherwise(pl.col("balance"))
.round(3)
.alias("balance"),
pl.col("note").fill_null("N/A"),
)
legacy.write_csv("legacy.csv")
rewrite.write_csv("rewrite.csv")
print(legacy.head(3))
print(rewrite.head(3))
# Output:
# shape: (3, 5)
# ┌────────────┬────────┬────────┬─────────┬──────────────┐
# │ account_id ┆ region ┆ status ┆ balance ┆ note │
# │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
# │ i64 ┆ str ┆ str ┆ f64 ┆ str │
# ╞════════════╪════════╪════════╪═════════╪══════════════╡
# │ 1 ┆ south ┆ paused ┆ 112.5 ┆ renewal 2027 │
# │ 2 ┆ east ┆ closed ┆ 125.0 ┆ renewal 2028 │
# │ 3 ┆ west ┆ active ┆ 137.5 ┆ renewal 2026 │
# └────────────┴────────┴────────┴─────────┴──────────────┘
# shape: (3, 5)
# ┌────────────┬────────┬────────┬─────────┬──────────────┐
# │ account_id ┆ region ┆ status ┆ balance ┆ note │
# │ --- ┆ --- ┆ --- ┆ --- ┆ --- │
# │ i64 ┆ str ┆ str ┆ f64 ┆ str │
# ╞════════════╪════════╪════════╪═════════╪══════════════╡
# │ 1 ┆ SOUTH ┆ P ┆ 112.5 ┆ renewal 2027 │
# │ 2 ┆ EAST ┆ C ┆ 125.0 ┆ renewal 2028 │
# │ 3 ┆ WEST ┆ A ┆ 137.504 ┆ renewal 2026 │
# └────────────┴────────┴────────┴─────────┴──────────────┘
2. The first run¶
The configuration names the two files and the primary key, and declares no rules yet:
%%writefile veridelta.yaml
source:
path: legacy.csv
target:
path: rewrite.csv
primary_keys: [account_id]
!veridelta run -c veridelta.yaml --quiet
# Output:
#
# Veridelta Execution Summary
# ===========================
# Status: FAILED
# Match Rate: 0.0%
# Source Rows: 40
# Target Rows: 39
# Volume Shift: -1 row
#
# Row-Level Discrepancies:
# ---------------------------
# Added: 0
# Removed: 1
# Changed: 39
# Total Issues: 40
#
# Top Column-Level Drifts:
# ---------------------------
# - region: 39 mismatches
# - status: 39 mismatches
# - balance: 13 mismatches
# - note: 7 mismatches
Every account the rewrite kept differs, and four columns drift. That is too many rows to read one by one.
3. Suggest rules¶
veridelta suggest runs the comparison, then tries each kind of rule it knows on the columns that differ. It prints the rules that explain the differences as YAML on stdout. On stderr, it prints the evidence for each: how many differing rows it explains, example keys, and for a tolerance, the largest gap. No model is called:
!veridelta suggest -c veridelta.yaml
# Output:
# Loading configuration from veridelta.yaml...
# Comparing, and trying each rule...
# region: case_insensitive true explains 38 of 39 differing rows
# for example account_id=1; account_id=2; account_id=3
# balance: absolute_tolerance 0.005 explains 13 of 13 differing rows, the largest gap 0.004
# for example account_id=3; account_id=6; account_id=9
# note: null_values ["N/A"] explains 7 of 7 differing rows
# for example account_id=5; account_id=10; account_id=15
# rules:
# - column_names:
# - region
# case_insensitive: true
# - column_names:
# - balance
# absolute_tolerance: 0.005
# - column_names:
# - note
# null_values:
# - N/A
Read each rule before you paste it, since a rule forgives its kind of difference in every later run too:
region: case folding explains 38 of the 39 differing rows. Account 17 differs in more than case, so it stays reported.balance: the largest gap is 0.004, so a tolerance of 0.005 forgives the rounding and nothing larger. A balance off by a cent still fails.note: the rewrite writesN/Awhere the legacy file has no value.
status gets no suggestion, since active and A differ in more than case or spacing. Suggesting rules lists every kind of rule suggest tries.
4. Map codes with crosswalk¶
A value map translates codes, and veridelta crosswalk proposes one from the pairs the data agrees on. Run it once the rules above are in place. Before them, it also proposes a map from each region to its capitals. Case folding says the same in one setting, and covers regions the data does not hold yet:
%%writefile veridelta.yaml
source:
path: legacy.csv
target:
path: rewrite.csv
primary_keys: [account_id]
rules:
- column_names: [region]
case_insensitive: true
- column_names: [balance]
absolute_tolerance: 0.005
- column_names: [note]
null_values: ["N/A"]
!veridelta crosswalk -c veridelta.yaml
# Output:
# Loading configuration from veridelta.yaml...
# Lining up source and target values...
# status: 3 new value_map entries
# 'active' -> 'A': 13 of 13 rows (100.0%)
# 'closed' -> 'C': 13 of 13 rows (100.0%)
# 'paused' -> 'P': 13 of 13 rows (100.0%)
# rules:
# - column_names:
# - status
# value_map:
# active: A
# closed: C
# paused: P
An entry needs 5 agreeing rows by default, and 95% of the rows that hold its source value. Each code here agrees on all 13 of its rows. Proposing a value map explains how entries are chosen.
5. Run with the rules¶
The configuration now holds all four rules:
%%writefile veridelta.yaml
source:
path: legacy.csv
target:
path: rewrite.csv
primary_keys: [account_id]
rules:
- column_names: [region]
case_insensitive: true
- column_names: [balance]
absolute_tolerance: 0.005
- column_names: [note]
null_values: ["N/A"]
- column_names: [status]
value_map:
active: A
closed: C
paused: P
!veridelta run -c veridelta.yaml --quiet
# Output:
#
# Veridelta Execution Summary
# ===========================
# Status: FAILED
# Match Rate: 95.0%
# Source Rows: 40
# Target Rows: 39
# Volume Shift: -1 row
#
# Row-Level Discrepancies:
# ---------------------------
# Added: 0
# Removed: 1
# Changed: 1
# Total Issues: 2
#
# Top Column-Level Drifts:
# ---------------------------
# - region: 1 mismatch
Two rows remain. Account 40 is the purge, made on purpose. Account 17 is the defect: south in the legacy file and EAST in the rewrite.
6. Accept the change made on purpose¶
--save-baseline writes every row of drift the run finds to a file, and --baseline accepts the drift a file lists. The verdict and the exit code are the same with --save-baseline as without it, so this cell drops the summary:
!veridelta run -c veridelta.yaml --quiet --save-baseline accepted.json > /dev/null
import json
from pathlib import Path
accepted = json.loads(Path("accepted.json").read_text(encoding="utf-8"))
print("removed:", accepted["removed"])
print("changed:", accepted["changed"])
# Output:
# removed: [{'account_id': 40}]
# changed: [{'key': {'account_id': 17}, 'columns': ['region']}]
The file accepts both rows, the defect too, so read it before you commit it. Here, drop the entry for account 17, keep the purge, and run against the file:
accepted["changed"] = []
Path("accepted.json").write_text(json.dumps(accepted, indent=2) + "\n", encoding="utf-8")
!veridelta run -c veridelta.yaml --quiet --baseline accepted.json; echo "exit=$?"
# Output:
#
# Veridelta Execution Summary
# ===========================
# Status: FAILED
# Match Rate: 97.5%
# Source Rows: 40
# Target Rows: 39
# Volume Shift: -1 row
#
# Row-Level Discrepancies:
# ---------------------------
# Added: 0
# Removed: 0
# Changed: 1
# Total Issues: 1
# Accepted: 1
#
# Top Column-Level Drifts:
# ---------------------------
# - region: 1 mismatch
#
# exit=1
The run accepts the purge, fails on account 17 alone, and exits 1, so CI holds the rewrite until the defect is fixed. Commit veridelta.yaml and accepted.json together, and run with --baseline accepted.json, or pass baseline: accepted.json to the GitHub Action. Accepting drift describes the file, and Rules describes every rule. The cell below removes the files this guide wrote:
for name in ("legacy.csv", "rewrite.csv", "veridelta.yaml", "accepted.json"):
Path(name).unlink(missing_ok=True)
print("cleaned")
# Output:
# cleaned