Configuration Guide

Veridelta is driven by a declarative YAML configuration file or Python object. This specification defines data ingestion parameters, schema alignment constraints, and the semantic rules for engine evaluation.

Core Architecture

Execution parameters and datasets are defined at the root level of the configuration. The engine requires deterministic identifiers to perform record alignment.

source:
  path: "legacy_system.csv"
  format: "csv"

target:
  path: "modern_system.parquet"
  format: "parquet"

# Mandatory alignment key
primary_keys: ["user_id"]

Engine Directives

Global directives control the strictness of the underlying Polars evaluation engine.

Directive	Description
`schema_mode`	Enforces column structure constraints. Options: `intersection` (default, compares common columns only), `exact`, `allow_additions`, `allow_removals`.
`strict_types`	If `false` (default), the engine dynamically soft-casts target columns to source types to prevent execution halts on mismatched types. If `true`, type mismatches automatically fail the row.
`normalize_column_names`	If `true`, strips whitespace and lowercases all column headers prior to schema alignment.
`threshold`	The allowable mismatch ratio (0.0 to 1.0) before the pipeline exits with a failure code.

Column-Level Overrides (Rules)

The rules array defines granular, per-column or regex-pattern tolerances.

1. Numeric Tolerances

Bypass floating-point anomalies or acceptable system rounding differences.

rules:
  - column_names: ["total_amount", "tax"]
    absolute_tolerance: 0.01
    relative_tolerance: 0.005

2. String Normalization & Sanitization

Execute string mutations prior to type evaluation. regex_replace is processed first, ensuring text is sanitized before any subsequent cast_to operations.

rules:
  - column_names: ["user_email"]
    case_insensitive: true
    whitespace_mode: "both"

  - column_names: ["balance"]
    regex_replace:
      "\\$": ""  # Strip currency symbols before casting
    cast_to: "Float64"

3. Value Mapping (Crosswalks)

Translate legacy enumerations or system-specific codes to modern equivalents during evaluation.

rules:
  - column_names: ["status_code"]
    value_map:
      "0": "INACTIVE"
      "1": "ACTIVE"
      "2": "PENDING"

4. Exclusion Routing

Explicitly drop volatile or irrelevant columns (e.g., auto-generated timestamps) from the comparison matrix.

rules:
  - pattern: "^etl_loaded_at_.*"
    ignore: true