Veridelta CLI: End-to-End Execution¶

The Command Line Interface (CLI) is the primary deployment mechanism for production CI/CD pipelines and data orchestration systems.

This document outlines the standard operating procedure for generating local test data, defining a declarative YAML configuration, and executing the validation engine via the CLI.

1. Local Data Generation¶

Generate representative source and target CSV artifacts to simulate schema and formatting drift between systems.

In [ ]:

import polars as pl

# Source System Export
pl.DataFrame(
    {
        "user_id": [1, 2, 3],
        "status": ["Active", "Pending", "Closed"],
        "balance": ["$100.50", "$50.00", "$0.00"],
    }
).write_csv("source_export.csv")

# Target System Export (Exhibits categorical and numeric type drift)
pl.DataFrame(
    {"user_id": [1, 2, 3], "status": ["ACT", "PND", "CLS"], "balance": [100.50, 50.00, 0.00]}
).write_csv("target_export.csv")

print("SYSTEM: Data artifacts provisioned to local filesystem.")

# Output:
# SYSTEM: Data artifacts provisioned to local filesystem.

import polars as pl # Source System Export pl.DataFrame( { "user_id": [1, 2, 3], "status": ["Active", "Pending", "Closed"], "balance": ["$100.50", "$50.00", "$0.00"], } ).write_csv("source_export.csv") # Target System Export (Exhibits categorical and numeric type drift) pl.DataFrame( {"user_id": [1, 2, 3], "status": ["ACT", "PND", "CLS"], "balance": [100.50, 50.00, 0.00]} ).write_csv("target_export.csv") print("SYSTEM: Data artifacts provisioned to local filesystem.") # Output: # SYSTEM: Data artifacts provisioned to local filesystem.

2. Configuration Definition¶

The YAML configuration file dictates the validation rules, string sanitization, and type coercions. Parameters map directly to the root of the configuration models.

In [ ]:

%%writefile veridelta.yaml
source:
  path: "source_export.csv"
  format: "csv"

target:
  path: "target_export.csv"
  format: "csv"

primary_keys: ["user_id"]
strict_types: false
output_path: "./diff_results"
output_format: "parquet"

rules:
  - column_names: ["status"]
    value_map:
      "Active": "ACT"
      "Pending": "PND"
      "Closed": "CLS"
      
  - column_names: ["balance"]
    regex_replace:
      "\\$": ""
    cast_to: "Float64"

%%writefile veridelta.yaml source: path: "source_export.csv" format: "csv" target: path: "target_export.csv" format: "csv" primary_keys: ["user_id"] strict_types: false output_path: "./diff_results" output_format: "parquet" rules: - column_names: ["status"] value_map: "Active": "ACT" "Pending": "PND" "Closed": "CLS" - column_names: ["balance"] regex_replace: "\\$": "" cast_to: "Float64"

3. CLI Execution¶

Execute the Veridelta engine using the staged configuration.

A 0 exit code denotes pipeline success (mismatches within threshold). A 1 exit code denotes failure, halting the pipeline and persisting discrepancy artifacts to the designated output_path.

In [ ]:

!veridelta run -c veridelta.yaml

# Output:
# Loading configuration from veridelta.yaml...
# Ingesting and aligning datasets...
# Executing semantic diff...
#
# Veridelta Execution Summary
# ===========================
# Status:        PASSED (Perfect Match)
# Match Rate:    100.0%
# Source Rows:   3
# Target Rows:   3
# Volume Shift:  +0 rows
#
# Row-Level Discrepancies:
# ---------------------------
# Added:         0
# Removed:       0
# Changed:       0
# Total Issues:  0

!veridelta run -c veridelta.yaml # Output: # Loading configuration from veridelta.yaml... # Ingesting and aligning datasets... # Executing semantic diff... # # Veridelta Execution Summary # =========================== # Status: PASSED (Perfect Match) # Match Rate: 100.0% # Source Rows: 3 # Target Rows: 3 # Volume Shift: +0 rows # # Row-Level Discrepancies: # --------------------------- # Added: 0 # Removed: 0 # Changed: 0 # Total Issues: 0

4. Teardown & Housekeeping¶

Purge local testing artifacts and generated reporting directories to maintain a clean workspace.

In [ ]:

import pathlib
import shutil

# Remove local file artifacts
for file in ["source_export.csv", "target_export.csv", "veridelta.yaml"]:
    pathlib.Path(file).unlink(missing_ok=True)

# Remove generated artifacts directory
shutil.rmtree("diff_results", ignore_errors=True)

print("SYSTEM: Local artifacts cleaned.")

# Output:
# SYSTEM: Local artifacts cleaned.

import pathlib import shutil # Remove local file artifacts for file in ["source_export.csv", "target_export.csv", "veridelta.yaml"]: pathlib.Path(file).unlink(missing_ok=True) # Remove generated artifacts directory shutil.rmtree("diff_results", ignore_errors=True) print("SYSTEM: Local artifacts cleaned.") # Output: # SYSTEM: Local artifacts cleaned.