# Veridelta

> Compare two datasets on their primary keys under rules you declare, on a laptop, in CI, or inside a warehouse.

Source: https://veridelta.github.io/veridelta/

# Veridelta

Veridelta compares two datasets on their primary keys and reports every row that differs under the rules you declare. Nothing is forgiven unless a rule says so, and the exit code tells CI whether the datasets match. Use it to verify a migration, a pipeline change, or a model's new evaluation run, on a laptop, in CI, or inside a warehouse.

![A terminal prints a five-line veridelta.yaml and two three-row CSV files, validates the configuration, runs the comparison, shows one added, one removed, and one changed row, and prints the exit code for CI, 1, beside what 0, 1, and 3 mean.](https://veridelta.github.io/veridelta/assets/demo.gif)

Files, lakehouse tables, databases, and DuckDB files are read and compared on [Polars](https://pola.rs/). Two tables in one warehouse are compared inside it, as are two Postgres or DuckDB tables that set `pushdown`. Only counts and keys come back, unless [`pushdown_sample_rows`](https://veridelta.github.io/veridelta/pushdown/#row-samples) asks for a sample of the changed rows.

## Status

Veridelta is in alpha, so a minor version can still change the configuration or the Python API. Its tests run on generated data, including a suite that seeds known drift into two datasets and checks that each run reports exactly that drift, in a local run and in pushdown. Nobody but its maintainer is known to have run it yet. Its pushdown SQL has run in DuckDB and Postgres 16, but not yet in a live cloud warehouse, as [Pushdown](https://veridelta.github.io/veridelta/pushdown/) says.

## Install

```bash
uv add veridelta                # or: pip install veridelta
uv add 'veridelta[snowflake]'   # extras: snowflake, databricks, bigquery, delta, iceberg, database, duckdb, excel, fuzzy, mcp, all
```

## Quick start

The smallest configuration names the two files and the keys that pair their rows. The suffix of each path says what format it is, and the recording above runs this file on two three-row files:

```yaml
# veridelta.yaml
primary_keys: [id]
source:
  path: legacy.csv
target:
  path: modern.csv
```

`validate` checks the file without reading any rows, and `run` compares the two files. It exits 0 when they match, 1 when rows differ, and 3 when the run could not finish. [Command line](https://veridelta.github.io/veridelta/cli/) lists every code and flag:

```bash
veridelta validate -c veridelta.yaml
veridelta run -c veridelta.yaml
```

Two files that need no rules need no configuration file either. Name them and the key that pairs their rows, and each format still follows its suffix:

```bash
veridelta run legacy.csv modern.csv --key id
```

[Configuration](https://veridelta.github.io/veridelta/configuration/) lists every setting, and [Rules](https://veridelta.github.io/veridelta/rules/) say what counts as a match, column by column.

## Where to start

The tutorials build a comparison step by step. Each one opens in Google Colab from its first paragraph, where its first cell installs Veridelta:

1. [Core concepts](https://veridelta.github.io/veridelta/examples/01_core_concepts/): the Python API, `DiffResult`, and rules.
2. [YAML and CLI](https://veridelta.github.io/veridelta/examples/02_yaml_and_cli/): a configuration file, `--json`, and artifacts.
3. [Advanced rules](https://veridelta.github.io/veridelta/examples/03_advanced_rules/): resolving drift in real data.
4. [HTML reports](https://veridelta.github.io/veridelta/examples/04_html_reports/): a report to hand to reviewers.
5. [Validate and CI](https://veridelta.github.io/veridelta/examples/05_validate_and_ci/): a database source, `veridelta validate`, and the GitHub Action.
6. [Model evaluation runs](https://veridelta.github.io/veridelta/examples/06_model_evaluation_runs/): two runs of a model's evaluation, compared on the example ID, with a tolerance on scores and fuzzy matching on answers.

A how-to guide solves one task from start to end:

- [From drift to rules](https://veridelta.github.io/veridelta/how-to/from-drift-to-rules/): turn a failing first run into rules you review, and accept a change made on purpose.
- [Troubleshooting](https://veridelta.github.io/veridelta/how-to/troubleshooting/): what a failed or surprising run means, and the change that fixes it.

The user guide is the reference:

- [Configuration](https://veridelta.github.io/veridelta/configuration/): the file, its settings, and environment variables.
- [Sources](https://veridelta.github.io/veridelta/sources/): files, lakehouse tables, databases, DuckDB, and warehouses.
- [Rules](https://veridelta.github.io/veridelta/rules/): what counts as a match, column by column.
- [Pushdown](https://veridelta.github.io/veridelta/pushdown/): comparing two tables inside the warehouse that stores them.
- [Results](https://veridelta.github.io/veridelta/results/): the summary, reports, metrics, and files a run produces.
- [Command line](https://veridelta.github.io/veridelta/cli/): commands, flags, and exit codes.
- [CI integrations](https://veridelta.github.io/veridelta/ci/): the GitHub Action and the GitLab CI template.
- [AI agents](https://veridelta.github.io/veridelta/agents/): running Veridelta from an agent, through the command line or the MCP server.

## How it works

```mermaid
flowchart LR
  subgraph sources [Sources]
    files[Files]
    lakehouse[Delta Iceberg]
    databases[Postgres MySQL SQLite]
    duckdb[DuckDB MotherDuck]
    warehouse[Snowflake Databricks BigQuery]
  end
  files --> loader[LoaderFactory]
  lakehouse --> loader
  databases --> loader
  duckdb --> loader
  warehouse --> compiler[SQLPushdownCompiler]
  databases -. pushdown .-> compiler
  duckdb -. pushdown .-> compiler
  loader --> engine["DiffEngine"]
  engine --> result[DiffResult]
  compiler --> warehouseSql[Warehouse SQL]
  warehouseSql --> result
  result --> artifacts[Artifacts]
  result --> reports["HTML JSON"]
  result --> exitCode[Exit code]
```

File, lakehouse, database, and DuckDB sources load through `LoaderFactory` into a local `DiffEngine` run. A pair of tables in one warehouse, or two Postgres or DuckDB tables that set `pushdown`, compiles to SQL and runs in place. Both paths return a `DiffResult`.

Source: https://veridelta.github.io/veridelta/how-to/troubleshooting/

# Troubleshooting

Each section starts from what you see, then gives the cause and the change that fixes it. A command that cannot finish exits `3`. It prints the error's type and message on stderr, and with `--json`, as one object on stdout; see [Exit codes](https://veridelta.github.io/veridelta/cli/#exit-codes).

## Find the cause first

- `veridelta validate -c veridelta.yaml` checks the file without reading a row. With `--schemas`, it also reads each side's columns and checks every rule against them.
- The error's type says where to look. A `ConfigError` is in the configuration, a `ConnectorError` comes from a source, and a `DataIntegrityError` comes from the data. Any other type is a bug: please [report it](https://github.com/Veridelta/veridelta/issues).
- `-v` logs each file opened, each connection, and each statement on stderr.

## An extra is not installed

```text
Reading the source needs the optional 'snowflake' extra, which is not installed. Install it with: uv add 'veridelta[snowflake]'
```

The core package reads files. Every other kind of source, and Excel files, reads through an optional extra. Install the one the message names, with `uv add 'veridelta[snowflake]'` or `pip install 'veridelta[snowflake]'`. `veridelta validate` reports every missing extra without connecting. [Sources](https://veridelta.github.io/veridelta/sources/) names the extra for each kind of source.

## A primary key repeats

```text
DataIntegrityError
Primary keys ['id'] are not unique in SOURCE dataset. Found 2 duplicate rows. Clean your data before diffing.
```

The keys together must name one row on each side. Two causes are common:

- The key is incomplete. An order line needs `order_id` and `line_no`, not `order_id` alone. List every key column in `primary_keys`.
- A rule on a key column made two keys one. `case_insensitive` reads `A1` and `a1` as the same key, and the check runs after the rules.

To see the repeated keys, read the side with Polars:

```python
import polars as pl

frame = pl.read_csv("legacy.csv")
print(frame.filter(frame["id"].is_duplicated()))
```

## Every row is added and removed

```text
Added:         2
Removed:       2
Changed:       0
```

No key pairs, since each side spells its keys another way, such as `A1` against `a1`, or `7` against `007` in text. Rules on a key column apply before rows are paired. Give the key the rule that makes both sides spell it one way, such as `case_insensitive: true`, `whitespace_mode: both`, or `pad_zeros: 3`. See [Primary keys](https://veridelta.github.io/veridelta/configuration/#primary-keys).

The match rate reads below zero in this case. It is one minus the mismatch ratio, and the ratio counts added and removed rows over the source rows, so it can pass 1.

## A key holds two types

```text
Configuration Error
Rows pair only on keys of one type, or of two integer or two float types, and these primary keys hold two other types: 'id' (String in the source, Int64 in the target). Give each a rule with cast_to, such as cast_to: Int64, so both sides hold one type.
```

Two exports often type one key differently. A CSV with padded keys, such as `" 1"`, reads them as text, where a Parquet file stored integers. Give the key a rule with `cast_to`, and trim it first if it is padded:

```yaml
rules:
  - column_names: [id]
    whitespace_mode: both
    cast_to: Int64
```

Before 0.33.2, a run failed on such a key with an unexpected error from inside the join.

## Most values in a column differ

One systematic difference is the usual cause: rounding, case, padding, a placeholder for NULL, or a code for each value. `veridelta suggest` and `veridelta crosswalk` find these, with the evidence for each rule they propose. [From drift to rules](https://veridelta.github.io/veridelta/how-to/from-drift-to-rules/) walks through both.

A column stored as two types compares as [Column types](https://veridelta.github.io/veridelta/configuration/#column-types) says: two numbers by value, and any other pair by casting the target to the source type.

## Timestamps differ by whole hours

When one side stores a time zone and the other does not, the side without one is read as UTC. A `09:00` without a zone matches `09:00` UTC, and differs from `09:00` in New York, which is `14:00` UTC.

If the side without a zone holds local time, export it with its zone. Where it is text, parse it with a `datetime_format` that reads the offset, such as `%Y-%m-%dT%H:%M:%S%z`. A `timezone` rule converts zoned timestamps only, and refuses a column without a zone; see [Dates and timezones](https://veridelta.github.io/veridelta/rules/#dates-and-timezones).

## A SQLite read fails, or its numbers drift

- `sqlite:///legacy.db` names `/legacy.db`, at the root of the file system. Write the full path after `sqlite://`, such as `sqlite:///srv/data/legacy.db`.
- `Cannot infer type from null for SQLite` names a column declared without a type, whose first rows are NULL. Declare its type, or select it with a `CAST` in a `query`.
- A column declared `NUMERIC` arrives as `Float64`. A value a float cannot hold exactly, such as one computed as `0.1 + 0.2`, then differs from the same decimal on the other side. A small `absolute_tolerance`, such as `0.000001`, forgives that and nothing a person would see.
- `validate --schemas` reads no rows, so it shows such a `NUMERIC` column as `String`, where a run reads `Float64`.

[Databases](https://veridelta.github.io/veridelta/sources/#databases) lists how each declared type arrives.

## A rule does nothing

```text
warning: Neither side has a column named 'vv', so the rules that name it do nothing. Check the spelling.
```

`veridelta validate --schemas` warns about a rule that names a column neither side has. Without `--schemas`, it reads no columns, so it cannot tell. With `normalize_column_names`, write the names in lower case.

## Pushdown refuses a column of two types

A comparison inside a database refuses a column whose two sides hold two types, unless both are numeric, since the database would convert one side by its own rules. Give the column a `cast_to`, or set `strict_types`; [Upgrading](https://veridelta.github.io/veridelta/upgrading/#0330) shows both.

Source: https://veridelta.github.io/veridelta/configuration/

# Configuration

A configuration file declares the two datasets to compare, the primary keys that pair their rows, and the rules that decide when two values match. In Python, the same settings are the fields of `DiffConfig`.

The smallest file names a source, a target, and the primary keys:

```yaml
source:
  path: "legacy_system.csv"

target:
  path: "modern_system.parquet"

primary_keys: ["user_id"]
```

A source without a `type` is a file, read from `path` in the `format` its suffix names, or the one you give, with optional reader `options`. Set `type` to read a lakehouse table, a database, or a warehouse table instead; see [Sources](https://veridelta.github.io/veridelta/sources/). To change how particular columns are compared, add `rules`; see [Rules](https://veridelta.github.io/veridelta/rules/).

## Primary keys

`primary_keys` names the columns that pair each source row with its target row. It must name at least one column, and the keys together must be unique on each side. A key that repeats raises `DataIntegrityError` before any values are compared.

Each key must hold one type on both sides, or two integer types, or two float types. Any other pair, such as text against a number or a date against a timestamp, raises `ConfigError` before rows are paired. Give such a key a rule with `cast_to`, such as `cast_to: Int64`.

Rules on key columns apply before rows are paired: a `case_insensitive` key pairs `ABC` with `abc`. See [Transform order](https://veridelta.github.io/veridelta/rules/#transform-order). Write a renamed key with its target name; see [Renaming columns](https://veridelta.github.io/veridelta/rules/#renaming-columns).

## Settings

Every setting except `primary_keys` is optional:

| Setting | Default | Description |
| :--- | :--- | :--- |
| `primary_keys` | required | Columns that pair rows. See [Primary keys](https://veridelta.github.io/veridelta/configuration/#primary-keys). |
| `schema_mode` | `intersection` | Which columns both sides must have. See [Schema mode](https://veridelta.github.io/veridelta/configuration/#schema-mode). |
| `strict_types` | `false` | Whether a column stored as two different types fails. See [Column types](https://veridelta.github.io/veridelta/configuration/#column-types). |
| `normalize_column_names` | `false` | Whether to strip and lowercase column names before the sides are aligned. See [Column names](https://veridelta.github.io/veridelta/configuration/#column-names). |
| `threshold` | `0.0` | Largest mismatch ratio, from 0.0 to 1.0, that still counts as a match. The ratio is added, removed, and changed rows over source rows. |
| `default_absolute_tolerance` | `0.0` | Absolute tolerance for each numeric column without its own `absolute_tolerance`. |
| `default_relative_tolerance` | `0.0` | Relative tolerance for each numeric column without its own `relative_tolerance`. |
| `default_treat_null_as_equal` | `true` | Whether two NULLs match, for each column without its own `treat_null_as_equal`. |
| `default_whitespace_mode` | `none` | Whitespace to strip from each text column without its own `whitespace_mode`: `none`, `left`, `right`, or `both`. |
| `default_null_values` | `[]` | Values to read as NULL. Each applies only to columns whose type can hold it. |
| `rules` | `[]` | Rules for particular columns. See [Rules](https://veridelta.github.io/veridelta/rules/). |
| `report_top_columns_limit` | `5` | Drifting columns to list in `report_summary` and the Markdown summary. `0` hides the list in both. |
| `pushdown_sample_rows` | `0` | Pushdown only. Changed rows to fetch with their values. `0` fetches none, and no value leaves the warehouse. Local runs ignore it, since they hold every row. See [Row samples](https://veridelta.github.io/veridelta/pushdown/#row-samples). |
| `output_path` | none | Directory for discrepancy files. Without it, no files are written. See [Artifacts](https://veridelta.github.io/veridelta/results/#artifacts). |
| `output_format` | `parquet` | Format of the discrepancy files: `parquet`, `csv`, `json`, `ndjson`, or `arrow`. |

A `default_*` setting fills in for every column whose rule leaves that field unset. The tolerances loosen only columns that are numeric after normalization; every other column is compared exactly.

This file passes when at most 1% of rows differ, forgives numeric differences up to 0.01, and writes the differing rows to `./artifacts`:

```yaml
primary_keys: ["user_id"]
threshold: 0.01
default_absolute_tolerance: 0.01
default_relative_tolerance: 0.0
default_treat_null_as_equal: true
report_top_columns_limit: 5
output_path: "./artifacts"
output_format: parquet
```

### Schema mode

`schema_mode` sets which columns the two sides must share. Every mode compares only the columns both sides have, and every mode requires the primary keys on both sides.

| Mode | Passes when |
| :--- | :--- |
| `intersection` | Always. Columns on one side only are left out. |
| `exact` | Both sides have the same set of columns. Column order is not compared. |
| `allow_additions` | The target has every source column. It may add more. |
| `allow_removals` | The target adds no column. It may drop source columns. |

A violation raises `ConfigError` before any rows are read.

### Column types

With `strict_types: false`, a column stored as different types on the two sides is still compared:

- Two numeric types compare by value. An integer `10` and a float `10.7` differ, and a `Float32` `0.1` differs slightly from a `Float64` `0.1`. Add a tolerance to forgive precision gaps.
- Any other pair casts the target to the source type: the text `"10"` matches the integer `10`.

Pushdown compares two different types only when both are numeric, and refuses any other pair; see [Differences from a local run](https://veridelta.github.io/veridelta/pushdown/#differences-from-a-local-run).

With `strict_types: true`, a column whose two sides hold different types after normalization fails every row. A `cast_to` or `datetime_format` that brings both sides to one type keeps such a column comparable. Under `treat_null_as_equal`, two NULLs still match.

### Column names

`normalize_column_names: true` strips whitespace from every column name and lowercases it before the two sides are aligned. It applies on every entry point, `DiffEngine(...).run()` and `validate_schemas` included.

The names in `primary_keys`, `column_names`, and `rename_to` are normalized the same way. A `pattern` is not: write it against the lowercase names. Two names that become equal once normalized raise `ConfigError`. Pushdown refuses the setting when it would rename a stored column; see [Columns](https://veridelta.github.io/veridelta/pushdown/#columns).

## Environment variables

Any string inside `source` or `target` can read an environment variable, which keeps credentials and per-environment paths out of the file:

```yaml
source: &warehouse
  type: snowflake
  table: ANALYTICS.PUBLIC.LEGACY_EVENTS
  account: xy12345
  user: ${SNOWFLAKE_USER}
  password: ${SNOWFLAKE_PASSWORD}
  role: ${SNOWFLAKE_ROLE:-ANALYST}
  warehouse: COMPUTE_WH
  database: ANALYTICS
  schema_name: PUBLIC

target:
  <<: *warehouse
  table: ANALYTICS.PUBLIC.MODERN_EVENTS

primary_keys: ["event_id"]
```

- `${NAME}` is replaced by the variable's value. It can sit inside longer text, as in `s3://${LAKE_BUCKET}/events`. A variable that is set but empty gives empty text.
- `${NAME:-default}` uses `default` when the variable is unset or empty. The default is literal text and cannot contain `}` or another reference.
- `$${` writes a literal `${`. Write any value that contains `${` this way.
- A value read from the environment is never expanded again. A secret that contains `$` or `${` arrives intact.
- Nested values such as `storage_options` and file `options` are expanded too. Keys, numbers, and booleans are not.
- Root settings and `rules` are read as written. A `${1}` in a `regex_replace` replacement stays as it is.

A reference to an unset variable without a default raises `ConfigError` when the file loads. So does a malformed reference: `${1}`, `${NAME`, `${NAME-x}`, or a nested `${A:-${B}}`. The error names the field, such as `source -> password`, and the variable, but never a value. Validation errors for `source` and `target` leave out their input for the same reason.

Expanded values are text:

- `version` and `snapshot_id` accept only YAML integers. Write them literally.
- File `options` reach the reader as they are: an expanded option arrives as a string.
- A database `password` is percent-encoded when it joins the URI, but a `${VAR}` expanded inside `uri` is not. Keep database passwords in `password`.

## Loading a file in Python

`load_config` reads a file into the three objects a run needs, and `DiffEngine.run_from_configs` runs them through the same path as the CLI:

```python
from veridelta import DiffEngine, load_config

diff, source, target = load_config("veridelta.yaml")
result = DiffEngine.run_from_configs(diff, source, target)
summary = result.summary
```

`DiffEngine(config, source_frame, target_frame).run()` compares two Polars frames you have already loaded.

## Schema checks

`DiffEngine.validate_schemas` checks `schema_mode` and the primary keys against two schemas without reading any rows. It accepts zero-row frames and unevaluated scans, which makes it cheap enough to gate a deployment:

```python
import polars as pl

from veridelta import ConfigError, DiffConfig, DiffEngine

contract = DiffConfig(primary_keys=["user_id"], schema_mode="exact")
try:
    DiffEngine.validate_schemas(contract, pl.scan_parquet("source.parquet"), pl.scan_parquet("target.parquet"))
except ConfigError as exc:
    print(exc)
```

It raises `ConfigError` on a violation and returns nothing otherwise.

`DiffEngine.validate_rules` takes the same arguments and goes one step further. It resolves every rule against the aligned columns and builds each column's comparison, still without reading a row, then returns the columns a run would compare. A rule the run could not honor fails here, such as a null sentinel the column's type cannot hold, or a similarity limit without the `fuzzy` extra. Repeated keys and invalid regular expressions surface only once rows are read.

`DiffEngine.read_schema(source)` reads one side's columns and their types, as a run reads them before its first row, and returns no rows. A database or DuckDB side that sets `query` is refused, since only running the query would name its columns.

To check a whole configuration file from the command line, use `veridelta validate`; see [Checking a configuration](https://veridelta.github.io/veridelta/cli/#checking-a-configuration).

## Editor support

Veridelta publishes a JSON Schema for configuration files. It serves an editor that uses the YAML language server, such as VS Code with the `redhat.vscode-yaml` extension. The editor then completes keys, shows each field's description, and flags a typo such as `primary_key` or `absolute_tolerence` as you type.

Point a file at the schema with a comment on its first line:

```yaml
# yaml-language-server: $schema=https://veridelta.github.io/veridelta/schema/veridelta.schema.json
source:
  path: "legacy_system.csv"

target:
  path: "modern_system.parquet"

primary_keys: ["user_id"]
```

A `$schema:` key does not work, because the loader rejects keys it does not know.

To apply the schema to every configuration in a workspace without a modeline, map a file pattern to it in the YAML extension's settings, such as in `.vscode/settings.json`:

```json
{
  "yaml.schemas": {
    "https://veridelta.github.io/veridelta/schema/veridelta.schema.json": "**/veridelta*.yaml"
  }
}
```

The site's copy of the schema follows the main branch. To pin it to the release you run, use the copy in that release's tag, such as `https://raw.githubusercontent.com/Veridelta/veridelta/v0.35.1/docs/schema/veridelta.schema.json`. Or print the installed version's schema to a file and point at that:

```bash
veridelta schema > veridelta.schema.json
```

The schema is slightly stricter than the loader. The loader converts `threshold: "0.1"` to a number, and the schema flags the quotes. In `source` and `target`, every text field also accepts a `${NAME}` reference.

### Validate from a task

The schema flags a wrong key or value as you type. `veridelta validate` also checks what the schema cannot, such as a missing extra or a pattern Polars rejects; see [Checking a configuration](https://veridelta.github.io/veridelta/cli/#checking-a-configuration). This VS Code task runs it on the file in the editor and lists the verdict in the Problems panel. Save it as `.vscode/tasks.json`, then run it from the Terminal menu with Run Task:

```json
{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "Veridelta: validate this file",
      "type": "shell",
      "command": "veridelta",
      "args": ["validate", "-c", "${file}"],
      "presentation": {
        "reveal": "always",
        "clear": true
      },
      "problemMatcher": [
        {
          "owner": "veridelta",
          "source": "veridelta",
          "severity": "error",
          "fileLocation": ["autoDetect", "${workspaceFolder}"],
          "pattern": {
            "regexp": "^(.+?): (\\d+ errors?, \\d+ warnings?\\.)$",
            "kind": "file",
            "file": 1,
            "message": 2
          }
        },
        {
          "owner": "veridelta",
          "source": "veridelta",
          "severity": "warning",
          "fileLocation": ["autoDetect", "${workspaceFolder}"],
          "pattern": {
            "regexp": "^(.+?): (valid, with \\d+ warnings?\\.)$",
            "kind": "file",
            "file": 1,
            "message": 2
          }
        }
      ]
    }
  ]
}
```

The panel shows one entry per file, with its counts of errors and warnings. The terminal shows each finding, as the command prints it. A valid file with no warnings adds no entry.

Source: https://veridelta.github.io/veridelta/sources/

# Sources

A source block describes one side of a comparison: a file, a lakehouse table, a database or DuckDB table or query, or a warehouse table. Its `type` selects the connector, and a block without a `type` is a file.

## Connection fields

Each connector accepts the fields below and rejects any other key:

| `type` | Required | Optional |
| :--- | :--- | :--- |
| `file` (default) | `path` | `format` (from the suffix, else `csv`), `options` |
| `snowflake` | `table`, `account`, `user`, `warehouse`, `database`, `schema_name` | `password`, `private_key_path`, `private_key_passphrase`, `role` |
| `databricks` | `table`, `server_hostname`, `http_path` | `access_token`, `catalog`, `schema_name` |
| `bigquery` | `table`, `project` | `dataset`, `location`, `credentials_path`, `maximum_bytes_billed` |
| `delta` | `table_uri` | `version`, `storage_options` |
| `iceberg` | `table_uri` | `snapshot_id`, `storage_options` |
| `database` | `uri`, and exactly one of `table` or `query` | `password`, `pushdown`, `partition_on`, `partitions` |
| `duckdb` | `database`, and exactly one of `table` or `query` | `motherduck_token`, `pushdown` |

`version` and `snapshot_id` must be non-negative integers, and `maximum_bytes_billed` a positive one. A quoted number is rejected, not converted, because each value goes straight to a scan or a job. Warehouse, lakehouse, database, and DuckDB blocks cannot be changed once loaded.

## Extras

The core package reads files. Every other connector, and Excel files, needs an optional extra:

```bash
uv add 'veridelta[snowflake]'
uv add 'veridelta[databricks]'
uv add 'veridelta[bigquery]'
uv add 'veridelta[delta]'
uv add 'veridelta[iceberg]'
uv add 'veridelta[database]'
uv add 'veridelta[duckdb]'
uv add 'veridelta[excel]'
uv add 'veridelta[all]'
```

## Files

A file source reads `path` in one of these formats: `csv`, `parquet`, `json`, `ndjson`, `arrow`, `avro`, or `excel`. Any other `format` is rejected when the configuration loads. When `format` is absent, the file's suffix decides it, in any case and before any `?` in the path. The suffixes are `.csv`; `.parquet` or `.pq`; `.json`; `.ndjson` or `.jsonl`; `.arrow`, `.ipc`, or `.feather`; `.avro`; and `.xlsx` or `.xls`. A suffix not in that list, or none, reads as `csv`. A `format` you write always wins. A file read as the wrong format seldom holds the primary keys, and the error that stops the run names the file, the format it was read as, and the columns it found.

`options` go to the matching Polars reader. `{"separator": ";"}` reaches `scan_csv`, and `{"sheet_name": "Q3"}` reaches `read_excel`.

Most formats are read lazily, in streaming batches. Three are read whole into memory, because Polars has no lazy reader for them:

- `json`: a JSON document is one array, which cannot be parsed in parts. Prefer `ndjson` for anything large.
- `excel`: a spreadsheet is a random-access container. Reading one needs the `excel` extra.
- `avro`: Polars reads Avro eagerly.

Avro columns keep the types in the file's schema. Its `options` take `columns` and `n_rows`. The Avro reader takes a local path only; copy a file from object storage, such as `s3://`, before reading it.

Veridelta reads each file's schema before the comparison starts. So a missing file, a pattern that matches none, or a file that cannot be read fails at once, with a `ConnectorError` that names it.

## Lakehouse tables

A Delta Lake or Iceberg table is scanned lazily, as an unevaluated Polars `LazyFrame`. Install the `delta` or `iceberg` extra. This pair compares version 12 of a Delta table with one snapshot of an Iceberg table:

```yaml
source:
  type: delta
  table_uri: s3://lake/legacy_events
  version: 12
  storage_options:
    AWS_REGION: us-east-1

target:
  type: iceberg
  table_uri: s3://lake/iceberg/modern_events
  snapshot_id: 883142
  storage_options:
    AWS_REGION: us-east-1

primary_keys: ["event_id"]
```

`storage_options` is a map of strings passed to the scanner, such as credentials, the region, and other object store settings.

The scan reads the table's transaction log or metadata when it opens. So a missing table, version, or snapshot fails before the comparison starts, with the table's URI in the error. Checking an Iceberg `snapshot_id` reads one row.

## Databases

A `database` source reads a table, or the result of a query, from an operational database through [ConnectorX](https://github.com/sfu-db/connector-x). Install the `database` extra. The rows are compared locally. A database pairs with a file, a lakehouse table, or another database, and `crosswalk` reads it too. This pair compares a Postgres table with the result of a MySQL query:

```yaml
source:
  type: database
  uri: postgresql://analyst@legacy-db.internal:5432/sales
  password: ${LEGACY_DB_PASSWORD}
  table: public.orders

target:
  type: database
  uri: mysql://analyst@modern-db.internal:3306/sales
  password: ${MODERN_DB_PASSWORD}
  query: SELECT order_id, total, status FROM orders WHERE placed >= '2024-01-01'

primary_keys: ["order_id"]
```

Two Postgres tables on one server can instead be compared inside Postgres; see [Postgres](https://veridelta.github.io/veridelta/pushdown/#postgres).

### Connection string

`uri` is a ConnectorX connection string that starts with `postgresql://` or `postgres://`, `mysql://` (MariaDB too), `mssql://`, `oracle://`, `redshift://`, `clickhouse://`, or `sqlite://`.

A SQLite URI is followed by a file path, as in `sqlite:///srv/data/legacy.db`, or `sqlite://C:/data/legacy.db` on Windows. The path must name an existing file. Veridelta refuses a missing one, which ConnectorX would otherwise create as an empty database.

A SQL Server URI takes connection options as parameters. `?encrypt=true` requires TLS for the whole connection. A server with a self-signed certificate, such as a development container, also needs `trust_server_certificate=true`, which accepts the certificate without checking it.

### Table or query

Set exactly one of `table` and `query`:

- `table` is one to three identifier segments, such as `public.orders`. Each segment is quoted for its database: double quotes for Postgres, Redshift, Oracle, and SQLite, backticks for MySQL and ClickHouse, and brackets for SQL Server. Quoting keeps case: write names as they are stored. Any other scheme needs `query`.
- `query` is sent to the database as written. Veridelta cannot tell a read from a write: connect with a role that can only read. A `query` is expanded like any other `source` string. Write a literal `${` in it as `$${`.

### Password

`password` is percent-encoded into the URI. It can contain `@`, `:`, `/`, or any other character, and needs a user name in `uri`.

A password written into `uri` itself must already be percent-encoded, which an expanded `${VAR}` is not. Setting both fails when the file loads. Logs and errors print the URI without its query, and mask the value of a parameter named like a credential, such as `?password=`, where a driver's message repeats it. Use `password` all the same.

### Reading and types

The rows are read into memory once, before the comparison starts, because Polars has no lazy database reader. Select columns and filter rows in `query` instead of reading a whole table.

Column types come from the database driver. For SQLite, that means the declared types:

| Declared type | Polars type |
| :--- | :--- |
| `INTEGER` | `Int64` |
| `REAL` | `Float64` |
| `TEXT` | `String` |
| `DATE` | `Date` |
| `DATETIME` | `Datetime` |
| `BOOLEAN` | `Boolean` |
| `NUMERIC` | `Float64` |

A SQLite column declared without a type cannot be typed when its first rows are NULL, and the read fails.

A Postgres `table` keeps the declared precision and scale of each `numeric` column, so `numeric(10, 2)` arrives as `Decimal(10, 2)` with every stored digit. Veridelta reads the declarations from the `pg_attribute` catalog before the rows, so a Postgres-compatible server without that catalog needs a `query`. A `NaN` has no decimal form and fails the read; leave it out with a `query`.

Any other Postgres `numeric` arrives as `Decimal(38, 10)`. Its values are rounded to ten decimal places, and a value with more than 18 digits before the decimal point fails the read. That applies to every column of a `query`, to a `numeric` declared without a precision, and to one with a precision above 38 or a negative scale.

A MySQL or SQL Server `DECIMAL` arrives as `Decimal(38, 10)` too, with the same limits, whether a `table` or a `query` reads it.

MySQL has no boolean type, so a flag arrives as a number. These MySQL types arrive as follows:

| MySQL type | Polars type |
| :--- | :--- |
| `TINYINT(1)`, which `BOOLEAN` stands for | `Int8` |
| `BIT` | `Binary` |
| `INT UNSIGNED` | `UInt32` |
| `DATETIME`, `TIMESTAMP` | `Datetime`, with no time zone |
| `JSON` | `String` |

A `cast_to` rule cannot turn `Binary` into a number or a boolean, and a run refuses one before it reads a row. To compare a `BIT` column as a number, read it with `CAST(column AS UNSIGNED)` in a `query`.

SQL Server has a boolean type, `BIT`, and these of its types arrive as follows:

| SQL Server type | Polars type |
| :--- | :--- |
| `INT`, `TINYINT` | `Int64` |
| `BIT` | `Boolean` |
| `DATETIME2` | `Datetime`, with no time zone |
| `DATETIMEOFFSET` | `Datetime` in UTC |

ConnectorX applies the offset of a `DATETIMEOFFSET` twice, which shifts any value with a nonzero offset. A `table` read corrects for it. Veridelta reads the table's columns and no rows first, then selects each `DATETIMEOFFSET` column as `SWITCHOFFSET(column, '+00:00')`. So every value arrives at the instant it holds.

A `query` is sent as written, so its values shift: `12:00 +02:00` arrives as `08:00` in UTC instead of `10:00`. To read the instant each value holds, select the column as `SWITCHOFFSET(column, '+00:00')` in the `query`.

### Parallel reads

A large `table` reads faster in ranges, each over its own connection. Name an integer column to split on, and how many ranges to read:

```yaml
source:
  type: database
  uri: postgresql://analyst@legacy-db.internal:5432/sales
  password: ${LEGACY_DB_PASSWORD}
  table: public.orders
  partition_on: order_id
  partitions: 4
```

Veridelta reads the column's lowest and highest values, and ConnectorX splits that span into `partitions` ranges and reads them in parallel. Each range opens its own connection, so the database must accept that many more.

The column must hold integers and no NULL. A NULL falls in no range, so ConnectorX would leave its row out. Veridelta counts the column's NULLs first and fails the read if it finds any. An empty table is read in one piece.

ConnectorX writes the column name into each range's statement without quotes, so the database folds its case as it does for any unquoted name. On Postgres, partition on a column whose name is all lowercase.

On SQL Server, a table with a `DATETIMEOFFSET` column reads through a select that names every column. ConnectorX loses the escape of a `]` in a name when it splits that select. So a split read of such a table fails when a column name holds `]`. Read that table in one piece instead.

Only a `table` read splits. A `query` runs as written, and a `pushdown` table reads no rows. A schema check with `validate --schemas` reads the columns in one statement.

## DuckDB and MotherDuck

A `duckdb` source reads a table, or the result of a query, from a DuckDB file or a MotherDuck database. Install the `duckdb` extra. The rows are compared locally, so a DuckDB source pairs with any source but a warehouse. Two tables in one database can instead be compared inside DuckDB; see [DuckDB and MotherDuck](https://veridelta.github.io/veridelta/pushdown/#duckdb-and-motherduck). This pair compares a DuckDB file with a Parquet export:

```yaml
source:
  type: duckdb
  database: warehouse.duckdb
  table: main.orders

target:
  path: exports/orders.parquet

primary_keys: ["order_id"]
```

`database` is a file path, which resolves from the working directory, or a MotherDuck database written as `md:name` or `motherduck:name`. Set exactly one of `table` and `query`:

- `table` is one to three identifier segments, such as `main.orders` or `warehouse.main.orders`. Each segment is double-quoted, which keeps its case.
- `query` is sent to DuckDB as written, so it can also read files, as in `SELECT * FROM read_parquet('orders/*.parquet')`. A statement that returns no rows, such as `SET`, fails the read.

### DuckDB files

A file opens read-only, so a `query` that writes fails and the file never changes. A missing file fails the read rather than leaving an empty database behind.

DuckDB lets one process write to a file, or several read it, but not both at once. Close a notebook's read-write connection before a run reads the same file.

### MotherDuck

A MotherDuck database needs a token. Set `motherduck_token`, or the `MOTHERDUCK_TOKEN` or `motherduck_token` environment variable. Without one, MotherDuck opens a browser sign-in, which a CI job cannot finish, so Veridelta refuses to connect. Write `motherduck_token: ${MOTHERDUCK_TOKEN}` rather than the token itself, and never put the token in `database`, which is printed and logged.

The connection opens read-write, because a read-only MotherDuck connection needs a read-scaling token. A `query` therefore runs with all of the token's permissions. Use a read-scaling token for a source, as you would give a database `query` a role that can only read.

Connecting downloads MotherDuck's DuckDB extension, so the machine needs network access to DuckDB's extension repository. The MotherDuck connection has not yet run against a live account.

### DuckDB types

Column types come from DuckDB: `INTEGER` arrives as `Int32`, `DECIMAL(10, 2)` as `Decimal(10, 2)`, and `TIMESTAMPTZ` as a `Datetime` in UTC. The session reads time in UTC, so a timestamp with a time zone, or a date cast in a `query`, does not depend on the machine.

Polars has no type for `INTERVAL` or `UNION`, alone or inside a `STRUCT`, list, or map. A column holding one fails the read with its name. Cast it in a `query`, as in `CAST(span AS VARCHAR)`, or in a view when the source sets `pushdown`.

## Warehouses

A Snowflake, Databricks, or BigQuery source names a `table` in that warehouse. Two tables on one connection are compared inside the warehouse, and a warehouse table pairs with nothing else; see [Pushdown](https://veridelta.github.io/veridelta/pushdown/).

For Snowflake and Databricks, `table` is one to three unquoted identifier segments: `EVENTS`, `schema.table`, or `catalog.schema.table`. This pair compares two Snowflake tables:

```yaml
source:
  type: snowflake
  table: ANALYTICS.PUBLIC.LEGACY_EVENTS
  account: xy12345
  user: SVC_VERIDELTA
  private_key_path: ${SNOWFLAKE_KEY_FILE}
  private_key_passphrase: ${SNOWFLAKE_KEY_PASSPHRASE}
  warehouse: COMPUTE_WH
  database: ANALYTICS
  schema_name: PUBLIC

target:
  type: snowflake
  table: ANALYTICS.PUBLIC.MODERN_EVENTS
  account: xy12345
  user: SVC_VERIDELTA
  private_key_path: ${SNOWFLAKE_KEY_FILE}
  private_key_passphrase: ${SNOWFLAKE_KEY_PASSPHRASE}
  warehouse: COMPUTE_WH
  database: ANALYTICS
  schema_name: PUBLIC

primary_keys: ["event_id"]
```

Snowflake requires strong sign-in for scripted users, so a password alone may be refused. A service user signs in with a key pair instead:

- `private_key_path` names the PEM file of the user's private key.
- `private_key_passphrase` decrypts that file, if it is encrypted.
- A programmatic access token also works in `password`, but by default it needs a network policy that allows the client's address.

Set either `password` or `private_key_path`, not both. In GitHub Actions, write the key from a secret to a file before the step that runs Veridelta:

```yaml
- name: Write the Snowflake key
  run: printf '%s\n' "$SNOWFLAKE_PRIVATE_KEY" > "$RUNNER_TEMP/snowflake_key.p8"
  env:
    SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_PRIVATE_KEY }}
```

Then give the step that runs Veridelta `SNOWFLAKE_KEY_FILE: ${{ runner.temp }}/snowflake_key.p8` in its `env`.

This pair compares two Databricks tables:

```yaml
source:
  type: databricks
  table: main.default.legacy_events
  server_hostname: adb.azuredatabricks.net
  http_path: /sql/1.0/warehouses/abc
  catalog: main
  schema_name: default

target:
  type: databricks
  table: main.default.modern_events
  server_hostname: adb.azuredatabricks.net
  http_path: /sql/1.0/warehouses/abc
  catalog: main
  schema_name: default

primary_keys: ["event_id"]
```

### BigQuery

A `bigquery` source names a `project` and a `table`. The project runs the queries and holds the data. It never appears in SQL, so a project id with hyphens works. The table is `dataset.table`, or `table` alone when `dataset` names the default dataset:

```yaml
source:
  type: bigquery
  project: analytics-prod
  table: legacy.events
  location: US
  maximum_bytes_billed: 50000000000

target:
  type: bigquery
  project: analytics-prod
  table: modern.events
  location: US
  maximum_bytes_billed: 50000000000

primary_keys: ["event_id"]
```

Credentials come from Application Default Credentials, such as `gcloud auth application-default login` on a workstation or the attached service account on Google Cloud. Set `credentials_path` to a service account key file to use that instead.

`maximum_bytes_billed` makes BigQuery refuse any statement that would bill more bytes, which caps what a run can cost.

Project ids follow Google's rules: six to thirty lowercase letters, digits, or hyphens. Older domain-scoped ids, such as `example.com:project`, are refused.

## Credentials

Do not commit a `password`, a `private_key_passphrase`, an `access_token`, or a `motherduck_token` in YAML, a database `password` included. Write `${NAME}` so the loader reads the value from the environment; see [Environment variables](https://veridelta.github.io/veridelta/configuration/#environment-variables). Or build the connection in Python, as in `SnowflakeConfig(..., password=os.environ["SNOWFLAKE_PASSWORD"])`, and pass it to `DiffEngine.run_from_configs`.

Printing a connection config, or formatting one into a log line, leaves out its credentials:

- `password`, for Snowflake and databases;
- `private_key_path` and `private_key_passphrase`, for Snowflake;
- `access_token`, for Databricks;
- `credentials_path`, for BigQuery;
- `motherduck_token`, for MotherDuck;
- `storage_options`, for Delta Lake and Iceberg;
- a `storage_options` map inside a file source's `options`. The other reader options still print.

A password written inside a database `uri` prints as `***`, and the rest of the URI prints as written. The credentials stay readable as attributes and in `model_dump()`, because the connectors and readers need them. Remove them before logging a dump.

## Logging

Connectors log under `veridelta.connectors.warehouse`, `veridelta.connectors.lakehouse`, `veridelta.connectors.database`, and `veridelta.connectors.duckdb`. Each logger has a `NullHandler` and prints nothing until you configure logging, or pass `--verbose` on the [command line](https://veridelta.github.io/veridelta/cli/#logging):

```python
import logging

logging.basicConfig(level=logging.INFO)
logging.getLogger("veridelta.connectors").setLevel(logging.INFO)
```

- `INFO` records a session or scan opening and closing. It also records each database read, Postgres pushdown statements included, with its row count and the URI with its password masked. Each DuckDB read, pushdown statements included, is recorded with its row count and `database`.
- `INFO` also records each warehouse pushdown statement by its kind, with its duration. The kinds are `schema`, `duplicates`, `count`, `mismatch`, `added`, `missing`, `columns`, and `samples`.
- `WARNING` records a connection, a read, or a statement that fails, before the error reports it. A warehouse session that does not close cleanly is recorded with the error's type only.
- `DEBUG` adds the driver's traceback for a session that does not close cleanly.

At `INFO` and `WARNING`, log lines never contain SQL text, row values, `storage_options`, passwords, or tokens. The `DEBUG` traceback holds the driver's own text, which can echo connection details. A warehouse session closes when the run finishes, whether the run succeeded or raised.

Source: https://veridelta.github.io/veridelta/rules/

# Rules

A rule changes how the columns it selects are compared. It can clean values before the comparison, loosen the comparison, rename a column, or leave a column out.

Rules are listed under `rules`. Each one selects columns by name or by pattern, then sets any of the fields below:

```yaml
rules:
  - column_names: ["total_amount", "tax"]
    absolute_tolerance: 0.01
  - pattern: "^etl_"
    ignore: true
```

## Rule fields

The stage column gives each field's place in the [transform order](https://veridelta.github.io/veridelta/rules/#transform-order):

| Field | Stage | Description |
| :--- | :--- | :--- |
| `column_names` | | Exact source column names the rule governs. |
| `pattern` | | Regular expression matched against the start of each column name. |
| `ignore` | | Whether to leave the governed columns out of the comparison. |
| `rename_to` | | Target name for a single source column. |
| `null_values` | 1 | Values to read as NULL. Overrides `default_null_values`, and must fit the column's type. |
| `regex_replace` | 2 | Map of regular expression to replacement, applied in order to text. |
| `whitespace_mode` | 3 | Whitespace to strip from text: `none`, `left`, `right`, or `both`. Overrides `default_whitespace_mode`. |
| `case_insensitive` | 3 | Whether to lowercase text before comparing. |
| `value_map` | 4 | Map from a source value to the target value it stands for. |
| `pad_zeros` | 5 | Width to pad each value to with leading zeros, after converting it to text. |
| `datetime_format` | 6 | `strptime` pattern that parses text into timestamps. |
| `timezone` | 6 | Zone to convert timezone-aware timestamps to. |
| `cast_to` | 7 | Type to convert to: `Int64`, `Float64`, `String`, `Boolean`, `Date`, or `Datetime`. |
| `absolute_tolerance` | 8 | Largest absolute difference between two numbers that still matches. Overrides `default_absolute_tolerance`. |
| `relative_tolerance` | 8 | Largest difference relative to the source value, where `0.01` is 1%. Overrides `default_relative_tolerance`. |
| `max_levenshtein_distance` | 8 | Most single-character edits between two text values that still match. |
| `min_jaro_winkler_similarity` | 8 | Lowest Jaro-Winkler similarity, above 0 and at most 1, between two text values that still match. Local runs only. |
| `treat_null_as_equal` | 9 | Whether two NULLs match. Overrides `default_treat_null_as_equal`. |

## Selecting columns

A rule selects columns with `column_names`, a list of exact source column names, or with `pattern`, a regular expression matched against the start of each name. Every other field is optional.

One rule governs each column. When several rules could select it, the first rule that lists it by name wins, then the first rule whose `pattern` matches it. An `ignore` rule follows the same order: a rule that names a column keeps it even when a broader `ignore` pattern matches it.

A renamed column answers to both of its names. A rule that lists its target name wins, then a rule that lists its source name. Local runs and pushdown resolve rules the same way.

## Excluding columns

`ignore: true` leaves the columns a rule governs out of the comparison. Use it for volatile columns, such as load timestamps:

```yaml
rules:
  - pattern: "^etl_loaded_at_.*"
    ignore: true
```

## Renaming columns

`rename_to` pairs a source column with a target column of a different name. Use it when a field was renamed between systems:

```yaml
rules:
  - column_names: ["legacy_customer_id"]
    rename_to: "customer_id"
```

The rule's other fields apply to the renamed pair on both sides. A rename can carry a tolerance or a transform:

```yaml
rules:
  - column_names: ["legacy_amt"]
    rename_to: "amount"
    absolute_tolerance: 0.01
```

Write a renamed primary key in `primary_keys` with its target name. Local runs and pushdown both accept it, and pushdown reads the source column under its stored name.

## Transform order

Rules apply in a fixed order, not the order they are written in. Each column passes through nine stages, in a local run and in a warehouse alike:

1. Null sentinels (`null_values`)
2. Regular expression replacement (`regex_replace`)
3. Whitespace, then case (`whitespace_mode`, `case_insensitive`)
4. Value map, source side only (`value_map`)
5. Zero padding (`pad_zeros`)
6. Date parsing, then timezone (`datetime_format`, `timezone`)
7. Cast (`cast_to`)
8. Comparison: equality, a numeric tolerance, or a text similarity limit
9. Null equality (`treat_null_as_equal`)

Stages 1 to 7 normalize each side on its own, before rows are paired. They apply to primary keys too: a `case_insensitive` rule on a key column changes which rows pair, not only how they compare. If normalized keys make two rows on one side share a key, `DataIntegrityError` is raised instead of a join that multiplies rows.

Stages 2, 3, and 4 apply to text columns and skip every other type: a global `default_whitespace_mode` is safe on a mixed schema. Stage 1 checks each sentinel against each column's type; see [Null sentinels](https://veridelta.github.io/veridelta/rules/#null-sentinels).

## What rules cannot do

A rule says how one column on each side becomes comparable. That leaves some checks outside the rules:

- **No arithmetic.** A rule cannot compute a value, such as rounding up to the nearest multiple of 7 or turning cents into dollars. A [tolerance](https://veridelta.github.io/veridelta/rules/#numeric-tolerances) says how far apart two numbers may be and still match, which covers most rounding: `absolute_tolerance: 0.005` forgives up to half a cent.
- **One column at a time.** Each stage works on one column, so no rule compares two columns, such as a total with the sum of its parts.
- **Rows pair by their primary key.** Two rows with no key in common are one added and one removed, never matched by their other values. The cleaning stages apply to keys too, so a rule can pair keys that differ only in case or padding.
- **Patterns are those of Polars.** `regex_replace` has no look-around and no backreferences; see [Regular expressions](https://veridelta.github.io/veridelta/rules/#regular-expressions).

A difference that no rule and no default covers is reported. To check something a rule cannot express, change the data before Veridelta reads it, such as in a SQL view or a step of the pipeline, then compare the result.

## Null sentinels

`null_values` lists the placeholder values a system writes in place of NULL. A list can mix text, numbers, and booleans:

```yaml
default_null_values: ["N/A", "", -999]

rules:
  - column_names: ["is_verified"]
    null_values: [false]
```

Each sentinel applies only to columns whose type can hold it:

- Text reaches string, categorical, and enum columns.
- A number reaches any numeric column, decimals included.
- A boolean reaches boolean columns only.

Quoting therefore decides the type. `-999` is a number: it nulls `-999` in an integer column and is skipped on a text column. `"-999"` is text and behaves the other way around. List both if a value appears in both forms.

`default_null_values` spans a mixed schema, and it skips each sentinel that does not fit a column. An explicit `null_values` on a named column is a direct instruction: if none of its sentinels fits the column's type, Veridelta raises `ConfigError`. Pushdown applies the same check to the probed schema.

`.nan` and `.inf` are rejected when the file loads. NaN never equals itself, and infinity has no portable SQL literal.

## Regular expressions

`regex_replace` maps patterns to replacements and applies them in order to text columns. It runs before `cast_to`, which converts the cleaned text:

```yaml
rules:
  - column_names: ["balance"]
    regex_replace:
      "\\$": ""  # Strip currency symbols before casting
    cast_to: "Float64"
```

Patterns use the regular expression syntax of Polars, which has no look-around and no backreferences, unlike Python's `re`.

A replacement refers to capture groups as Polars does: `$1` or `${1}`, with `$0` for the whole match and `$$` for a dollar sign. `$1a` refers to a group named `1a`; write `${1}a` for group 1 followed by `a`. A backslash in a replacement is plain text. Pushdown rewrites each reference for the warehouse's SQL; see [Rules in SQL](https://veridelta.github.io/veridelta/pushdown/#rules-in-sql).

## Whitespace and case

`whitespace_mode` strips whitespace from the `left` end, the `right` end, or `both` ends of text. It strips spaces, tabs, line breaks, no-break spaces, and every other Unicode whitespace character. `case_insensitive` then lowercases the text:

```yaml
rules:
  - column_names: ["user_email"]
    case_insensitive: true
    whitespace_mode: "both"
```

## Value maps

`value_map` translates source values into the target's terms before the comparison, such as legacy status codes into names. It applies to the source side of text columns. A value without an entry stays as it is:

```yaml
rules:
  - column_names: ["status_code"]
    value_map:
      "0": "INACTIVE"
      "1": "ACTIVE"
      "2": "PENDING"
```

The map reads values after stages 1 to 3: a `case_insensitive` column needs lowercase keys.

### Proposing a value map

`veridelta crosswalk` drafts `value_map` entries from the data. It aligns, normalizes, and pairs the configured datasets as `run` does, then proposes each source value for the target value it lines up with:

```bash
veridelta crosswalk -c veridelta.yaml > proposed.yaml
```

The evidence goes to stderr:

```text
gender: 2 new value_map entries
  'M' -> 'Male': 599 of 600 rows (99.8%)
  'F' -> 'Female': 400 of 400 rows (100.0%)
```

The rules go to stdout, ready to paste into the configuration:

```yaml
rules:
- column_names:
  - gender
  value_map:
    M: Male
    F: Female
```

An entry needs both of these:

- **Confidence**, set by `--min-confidence` (default 0.95): the share of the source value's paired rows whose target is the proposed value. Every row with that source value counts, including rows that already match and rows whose target is NULL. An entry that would break a matching row pays for it. The floor must be above 0.5, which leaves at most one candidate per source value.
- **Support**, set by `--min-support` (default 5): how many rows agree. A coincidence in a handful of rows is never proposed.

Values are read as the `value_map` stage sees them, after null sentinels, `regex_replace`, whitespace, and case. A `case_insensitive` column gets lowercase entries.

Only compared text columns qualify, and only when no later stage changes the mapped value. A column with `pad_zeros`, `datetime_format`, or a `cast_to` other than `String` is skipped, as are primary keys and ignored columns. A non-text target is read as the text it is compared as: a `Y`/`N` source against a `1`/`0` target proposes `Y: '1'`, unless `strict_types` rules the pair out.

An existing `value_map` is kept and extended. Rows it already translates are left out of the counts, so a raw value equal to one of its outputs cannot receive an entry.

One rule governs each column. When a rule already governs a column, the command says so on stderr, even with `--quiet`, and the new entries belong in that rule's `value_map`. When that rule also governs other columns, through several names or a `pattern`, the column needs a rule of its own first. A map merged into a shared rule applies to every column it governs. The note says which case applies.

`--sample-fraction` (default 1.0) reads that share of the source rows, chosen by a hash of the primary keys. Rerunning on the same data with the same Polars version samples the same rows. `--json` prints each proposal with its evidence instead of YAML.

Two tables on one warehouse connection are counted inside the warehouse, and no row leaves it. The connection rules are those of a warehouse run: one backend, one connection, and two different tables. A warehouse paired with a file or a database is refused. After the column probes and the duplicate key checks, one statement counts every candidate column. Veridelta then applies the confidence floor itself, so a warehouse proposes exactly what a local run would from the same rows. Two differences remain:

- Only columns stored as text on both sides qualify. A local run also proposes text entries for a non-text target, such as `Y: '1'`. Each engine writes numbers and timestamps as text its own way, and a warehouse compares such an entry against the integer column, not its text.
- A sample hashes the normalized keys with the warehouse's own hash function. Rerunning against the same tables samples the same rows, but not the rows a local run of the same fraction samples.

In Python, `DiffEngine(config, source, target).propose_value_maps()` returns `ValueMapProposal` objects. `DiffEngine.propose_value_maps_from_configs(diff, source, target)` takes what `load_config` returns and reads the data first. Each proposal's `to_rule()` returns it as a standalone rule.

## Zero padding

`pad_zeros` converts each value to text and pads it with leading zeros to a fixed width. A numeric `123` in one system then matches the text `"00123"` in the other:

```yaml
rules:
  - column_names: ["account_number"]
    pad_zeros: 10
```

A negative value keeps its sign in front, as in `-012`, and a value longer than the width stays whole. The width must be a YAML integer: `pad_zeros: "5"` is rejected, not converted.

## Dates and timezones

`datetime_format` parses text into timestamps with a [strptime](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) pattern, and the column is compared as timestamps. `timezone` then converts timezone-aware timestamps to one zone:

```yaml
rules:
  - column_names: ["created_at"]
    datetime_format: "%Y-%m-%d %H:%M:%S%z"
    timezone: "UTC"
```

A value that does not fit the pattern becomes NULL. Two such values match only under `treat_null_as_equal`, and one against a parsed timestamp is a mismatch. Pushdown translates a fixed set of directives into each warehouse's format language; see [Rules in SQL](https://veridelta.github.io/veridelta/pushdown/#rules-in-sql).

`%f` is a fraction of a second, as in Python: `.5` is half a second. A local run and a warehouse both read one to six digits. A local run also reads seven to nine digits, keeping microseconds, and parses a value with no fraction at all when the format writes `.%f`. A warehouse may read either as NULL.

`timezone` requires timezone-aware data. Naive timestamps raise `ConfigError`, because assuming a zone for them would shift every value by a real offset without a warning. To compare text timestamps that carry an offset, parse them with a format that contains `%z`.

The conversion changes only the timezone label. Comparisons and casts read the underlying instant, not the wall clock time in the new zone, so a `timezone` rule cannot change a verdict on its own. It makes two differently zoned columns comparable, and it rejects data that carries no zone.

## Casts

`cast_to` converts a column to `Int64`, `Float64`, `String`, `Boolean`, `Date`, or `Datetime`. Any other type name is rejected when the file loads. A cast the column's type cannot take, such as a date or text to `Boolean`, or binary data to a number, stops the run before it reads a row, and `validate --schemas` reports it.

Converting a float to `Int64` truncates toward zero, as Polars does, where SQL would round. Local runs and pushdown both truncate.

## Numeric tolerances

A tolerance forgives small differences between numbers, such as rounding between two systems:

```yaml
rules:
  - column_names: ["total_amount", "tax"]
    absolute_tolerance: 0.01
    relative_tolerance: 0.005
```

Two numbers match when they are equal, or when their difference is at most `absolute_tolerance + relative_tolerance * |source|`. A tolerance loosens only columns that are numeric after normalization, such as a text column with `cast_to: Float64`. Text, boolean, and temporal columns are compared exactly.

Only finite values are loosened. `NaN` matches only `NaN`, and an infinity matches only the same infinity, however wide the tolerance. Integer differences are exact and never wrap around the column's type: Int8 `100` and `-100` differ by 200.

A tolerance must be finite. To stop comparing a column, use `ignore`.

`veridelta suggest` proposes a tolerance, trimming, case folding, a date format, or null sentinels from the data, with the rows each explains; see [Suggesting rules](https://veridelta.github.io/veridelta/cli/#suggesting-rules).

## Fuzzy text matching

A similarity limit forgives typos in free text, such as names typed by hand into two systems, without a `regex_replace` for each one. Scores come from [RapidFuzz](https://github.com/rapidfuzz/RapidFuzz), which the `fuzzy` extra installs:

```bash
uv add 'veridelta[fuzzy]'
```

A rule sets one of two limits:

```yaml
rules:
  - column_names: ["customer_name"]
    case_insensitive: true
    max_levenshtein_distance: 1
  - column_names: ["city"]
    min_jaro_winkler_similarity: 0.95
```

- `max_levenshtein_distance` counts single-character insertions, deletions, and substitutions. `Jon` and `John` are one edit apart and match at `1`. `kitten` and `sitting` need `3`.
- `min_jaro_winkler_similarity` scores two values from 0 to 1 and rewards a shared prefix. `MARTHA` and `MARHTA` score 0.961: they match at `0.96` but not at `0.97`.

A limit loosens only columns that are text after normalization, as a tolerance loosens only numbers. A column cast to a number or parsed as a date is still compared exactly, while `pad_zeros` or `cast_to: String` makes a column text.

Equal values still match outright. A NULL is never similar to anything, so `treat_null_as_equal` alone decides NULLs. Scores are case-sensitive: `case_insensitive` lowercases both sides first, in stage 3, which puts `ABD` and `abc` one edit apart. Primary keys are never loosened, because rows pair on equal keys.

A rule sets at most one of the two limits, and neither has a global default. Loosening every text column would also forgive identifiers and codes that must match exactly. Without the extra, a run that needs a score raises `ConfigError` with the install command before comparing any rows.

Pushdown compiles `max_levenshtein_distance` for Snowflake, Databricks, and BigQuery, while Postgres and DuckDB refuse it. Every warehouse refuses `min_jaro_winkler_similarity`; see [Rules in SQL](https://veridelta.github.io/veridelta/pushdown/#rules-in-sql).

## Null equality

`treat_null_as_equal` decides whether two NULLs match. A rule without it uses `default_treat_null_as_equal`, which is `true`. A NULL never matches a value, whatever the setting. A value that an earlier stage turns into NULL, such as text that `datetime_format` cannot parse, follows the same rule.

Source: https://veridelta.github.io/veridelta/pushdown/

# Pushdown

Pushdown compares two tables inside the database that stores them. Veridelta compiles the comparison to SQL, runs it there, and reads back counts and primary keys instead of rows.

Every rule gives the same verdict in a warehouse as in a local run, except where this page says otherwise. The generated SQL has run in DuckDB and in Postgres 16, but not yet in live BigQuery, Databricks, MotherDuck, or Snowflake services.

## When a comparison is pushed down

A pair of warehouse tables is always compared in place. Both sides must use the same warehouse and the same connection, and name two different tables:

| `type` | Fields that must match on both sides |
| :--- | :--- |
| `snowflake` | `account`, `user`, `warehouse`, `database`, `schema_name`, `role`, `password` |
| `databricks` | `server_hostname`, `http_path`, `access_token`, `catalog`, `schema_name` |
| `bigquery` | `project`, `dataset`, `location`, `credentials_path`, `maximum_bytes_billed` |

Naming the same table on both sides raises `ConfigError`, since a table compared with itself always matches. A warehouse paired with a file, a lakehouse table, or a database, or with a different warehouse, raises `ConnectorError`.

Database and DuckDB sources are read into memory and compared locally. Two Postgres tables on one server, or two tables in one DuckDB database, are compared in place when both set `pushdown`; see [Postgres](https://veridelta.github.io/veridelta/pushdown/#postgres) and [DuckDB and MotherDuck](https://veridelta.github.io/veridelta/pushdown/#duckdb-and-motherduck).

## Statements

A pushdown run issues these statements:

1. A zero-row column probe, a duplicate key check, and a `COUNT(*)` for each side.
2. The changed rows: an inner join that finds mismatches.
3. The added rows, found only in the target, and the removed rows, found only in the source.
4. A tally of mismatches per column. It is skipped when no column is compared.

With `pushdown_sample_rows` set, one more statement fetches a sample of the changed rows with their values; see [Row samples](https://veridelta.github.io/veridelta/pushdown/#row-samples).

These counts fill every `DiffSummary` field, `column_mismatches` included. `threshold`, `match_rate_percentage`, and the drift report mean what they mean in a local run.

The duplicate key check costs one grouped scan of each table over its normalized keys. A key that repeats raises `DataIntegrityError` before any count or join runs, as a local run refuses repeated keys.

## Columns

The column probes check `schema_mode` and the primary keys before any comparison runs, and raise `ConfigError` on a violation. Rules that select columns by `pattern` match against the probed names before any SQL is compiled.

Probed names are compared exactly as the compiler quotes them, with no case folding. Write names in the case the warehouse stores them; Snowflake stores unquoted names in uppercase. `normalize_column_names` cannot change that: pushdown raises `ConfigError` if the setting would rename a stored column.

## Rules in SQL

All nine transform stages compile for compared columns, and stages 1 to 7 for primary keys. One setting is refused everywhere: `min_jaro_winkler_similarity` raises `ConfigError` before any comparison runs. Snowflake's `JAROWINKLER_SIMILARITY` ignores case and returns a whole number from 0 to 100, and Databricks has no Jaro-Winkler function. Neither can reproduce a local verdict. Postgres also refuses `datetime_format` and `max_levenshtein_distance`, and DuckDB refuses `max_levenshtein_distance`; see [Postgres](https://veridelta.github.io/veridelta/pushdown/#postgres) and [DuckDB and MotherDuck](https://veridelta.github.io/veridelta/pushdown/#duckdb-and-motherduck).

### Edit distance

`max_levenshtein_distance` compiles to Snowflake's `EDITDISTANCE`, Databricks' `levenshtein`, or BigQuery's `EDIT_DISTANCE`, which count characters as a local run does. A warehouse run needs no `fuzzy` extra. Postgres and DuckDB refuse the setting, each for its own reason.

### Text in SQL

Write `regex_replace` patterns, `value_map` entries, and text `null_values` as you would for a local run. Each is escaped for the warehouse's string literals, so a backslash in `\d` or `\N` and the apostrophe in `O'Brien` arrive intact. Do not double them yourself.

Escaping keeps the text, but each warehouse runs its own regular expression engine. Capture group references in a replacement, written as Polars reads them, are rewritten in the warehouse's spelling, such as `\1` on Snowflake, BigQuery, DuckDB, and Postgres. Refer to groups by number, 0 to 9. No warehouse can refer to a group by name in a replacement, so a named reference raises `ConfigError`. That includes `$1a`, which Polars reads as the group named `1a`.

`whitespace_mode` strips the same characters in every warehouse as in a local run: spaces, tabs, line breaks, no-break spaces, and the rest of Unicode's whitespace.

`pad_zeros` compiles to an expression that keeps the sign in front and never truncates. A bare `LPAD` would pad in front of a minus sign, giving `0-12` where a local run gives `-012`, and would cut characters past the width.

### Dates and timezones

`datetime_format` is translated directive by directive into the warehouse's own format language, from a fixed table. `%Y-%m-%d` becomes `YYYY"-"MM"-"DD` on Snowflake and `yyyy'-'MM'-'dd` on Databricks, whose parser reads Java `DateTimeFormatter` patterns.

The table covers `%Y`, `%m`, `%d`, `%H`, `%M`, `%S`, `%f`, `%z`, and `%%`, separated by spaces or any of `-` `/` `:` `.` `,` `_` `T`. Any other directive raises `ConfigError`. An untranslated directive would parse nothing and return NULL for every row, which would read as a clean match.

`timezone` emits no SQL. In a local run it rewrites only a column's timezone label, and every later cast and comparison reads the underlying UTC instant, so it cannot change a verdict. Warehouses have no per-column label to rewrite, and Spark's `TIMESTAMP` is a bare instant. Functions that look equivalent shift the value to a wall clock time instead, which would make pushdown disagree with a local run. The rule's precondition still holds: a column that is not a timezone-aware timestamp once `pad_zeros` and `datetime_format` apply raises `ConfigError`, as it does locally. Text parsed with a `%z` format is aware, and text parsed without one is naive.

## Differences from a local run

- **Artifacts hold primary keys only**, since the comparison SQL never selects whole rows. They are written as `added_rows_pks_only`, `removed_rows_pks_only`, and `changed_rows_pks_only`, so they cannot be mistaken for local artifacts, which hold whole records. A [row sample](https://veridelta.github.io/veridelta/pushdown/#row-samples), when requested, is written as `changed_rows_sample`.
- **A column of two different types is refused unless both are numeric.** A local run casts the target to the source type. A database converts one side by its own rules, so the text `"007"` against the integer `7`, or a date against a timestamp, could reach another verdict. Pushdown raises `ConfigError` before reading a row, naming each such column. Give it a `cast_to` that brings both sides to one type, or set `strict_types` to fail it; [Upgrading](https://veridelta.github.io/veridelta/upgrading/#0330) shows both. Two numeric types still compare by value.
- **A primary key of two types pairs as the database converts it.** A local run refuses a key whose two sides hold two types it cannot pair, such as text against a number; see [Primary keys](https://veridelta.github.io/veridelta/configuration/#primary-keys). A database converts one side, so the text `"007"` pairs with the integer `7`. Give such a key a `cast_to`, and both pair rows the same way.
- **`strict_types` compares the types the driver reports** for each side, after normalization. It fails every row of a column whose two types differ, as a local run does. These are the driver's types, not the declared ones. Snowflake's `NUMBER(38,0)`, for one, arrives as a decimal, so it meets a `NUMBER(38,0)` column but not a `FLOAT`.

## BigQuery

BigQuery differs from the other warehouses in three ways a comparison can notice:

- `datetime_format` cannot use `%f`, since BigQuery spells fractional seconds only as part of the seconds. A format with `%z` parses to an aware timestamp, and one without it to a naive one, as in Polars.
- Comparing columns of different types fails the statement instead of converting one side. Give such a pair a `cast_to`, or set `strict_types`.
- `GEOGRAPHY` and `JSON` columns cannot be compared. Mark them `ignore`.

## Postgres

Two Postgres tables on one server can be compared where they are stored, instead of being read into memory. Set `pushdown: true` on both sides:

```yaml
source:
  type: database
  uri: postgresql://analyst@sales-db.internal:5432/sales
  password: ${SALES_DB_PASSWORD}
  table: legacy.orders
  pushdown: true

target:
  type: database
  uri: postgresql://analyst@sales-db.internal:5432/sales
  password: ${SALES_DB_PASSWORD}
  table: modern.orders
  pushdown: true

primary_keys: ["order_id"]
```

Veridelta then runs each statement inside Postgres through ConnectorX, as in a warehouse, and only counts and primary keys come back. Everything above applies, with these additions:

- Both `uri` values start with `postgresql://` or `postgres://` and match, as do both `password` values, so one connection reaches both tables. Each side names a `table`; a `query` cannot be compared in place. Other databases, Redshift included, are always read and compared locally.
- Set `pushdown` on both sides or on neither. A pair where only one side sets it raises `ConfigError` instead of reading both.
- The server must read string literals by the SQL standard, which is the Postgres default (`standard_conforming_strings` on). Veridelta checks before the first statement and raises `ConnectorError` if it is off, since a backslash in a value would otherwise be read as an escape.
- `datetime_format` and `max_levenshtein_distance` raise `ConfigError` before any statement runs. Postgres has no date parse that returns NULL for text it cannot read, so one bad value would fail the whole statement. Its `levenshtein` needs the `fuzzystrmatch` extension and refuses text longer than 255 characters. Leave `pushdown` off to compare such columns locally; `veridelta validate` warns about both.
- After the column probes, one catalog query per table reads the declared precision and scale of each `numeric` column, which a local read also keeps. `strict_types` tells `numeric(10, 2)` from `numeric(12, 4)`, and `pad_zeros` or `cast_to: String` writes seven in a `numeric(20, 0)` column as `7` with or without `pushdown`.
- A `numeric` declared without a precision, with one above 38, or with a negative scale has the type ConnectorX reads it as: `Decimal(38, 10)`. Turned into text, seven in such a column is `7` with `pushdown` and `7.0000000000` without it.
- Primary keys and row samples come back through ConnectorX too. A `numeric` value among them is rounded to ten decimal places, and one with more than 18 digits before the decimal point fails the statement.

## DuckDB and MotherDuck

Two tables in one DuckDB file or MotherDuck database can be compared inside DuckDB, instead of being read into memory. Set `pushdown: true` on both sides:

```yaml
source:
  type: duckdb
  database: warehouse.duckdb
  table: legacy.orders
  pushdown: true

target:
  type: duckdb
  database: warehouse.duckdb
  table: modern.orders
  pushdown: true

primary_keys: ["order_id"]
```

Veridelta opens one connection, as a read opens it, and runs each statement there. A file opens read-only, and every session reads time in UTC; see [DuckDB and MotherDuck](https://veridelta.github.io/veridelta/sources/#duckdb-and-motherduck). Everything above applies, with these additions:

- Both sides name a `table`, and their `database` and `motherduck_token` values match, so one connection reaches both tables. Write `database` the same way on both sides, since `./warehouse.duckdb` and `warehouse.duckdb` count as different. A `query` cannot be compared in place.
- Set `pushdown` on both sides or on neither. A pair where only one side sets it raises `ConfigError` instead of reading both.
- `max_levenshtein_distance` raises `ConfigError` before any statement runs. DuckDB's `levenshtein` counts UTF-8 bytes, not characters, so `é` against `e` is two edits where a local run counts one. Leave `pushdown` off to compare such columns locally; `veridelta validate` warns about it.
- `case_insensitive` lowercases with DuckDB's `lower`, which differs from Polars for a few letters. `İ` becomes `i` rather than `i̇`, and a final `Σ` becomes `σ` rather than `ς`.
- The column probe reads the type of every column, ignored ones included. A column that Polars cannot read, such as an `INTERVAL`, fails the probe. Compare a view that casts it, such as to `VARCHAR`.
- MotherDuck pushdown has not yet run against a live account, and neither have MotherDuck reads.

## Row samples

A pushdown run reads back counts and keys, so its report can say which rows changed but not how. Set `pushdown_sample_rows` to see values for some of them:

```yaml
primary_keys: ["order_id"]
pushdown_sample_rows: 50
```

After the counts, one more statement fetches up to that many changed rows, lowest keys first, so the same tables give the same sample. Each row holds its keys and, for every compared column, `{column}_source`, `{column}_target`, and `{column}_is_match`, as a local run's changed rows do. The values are the ones the comparison saw, after every rule up to the comparison itself, and each flag is the result that decided the row. The counts do not change, because the sample comes from the same changed rows.

The sample reaches these places:

- the HTML report, whose changed rows table shows it in place of the bare keys;
- `DiffResult.changed_sample`;
- with `output_path` set, a `changed_rows_sample` artifact;
- with `--markdown-max-rows` above 0, the [Markdown summary](https://veridelta.github.io/veridelta/results/#markdown-summary), which the [CI integrations](https://veridelta.github.io/veridelta/ci/) post as a pull request comment.

It never reaches a log line or the `--json` summary. Values do leave the warehouse, though, and the GitHub Action uploads the HTML report as a workflow artifact. Set `pushdown_sample_rows` only where everyone who can open the report may read the data. `0`, the default, fetches nothing.

Source: https://veridelta.github.io/veridelta/results/

# Results

`DiffEngine.run()` and `DiffEngine.run_from_configs()` return a `DiffResult`. It holds a summary of what differs on `summary`, and the rows behind it on `added`, `removed`, and `changed`, so a notebook can inspect the drift without writing files.

The summary prints as a plain-text report, and one call isolates the rows where a single column disagrees:

```python
import polars as pl

from veridelta import DiffConfig, DiffEngine

diff = DiffConfig(primary_keys=["trip_id"])
source_df = pl.scan_parquet("legacy_trips.parquet")
target_df = pl.scan_parquet("modern_trips.parquet")
result = DiffEngine(diff, source_df, target_df).run()

print(result.summary.report_summary)

# The rows where one column disagreed, with both values side by side.
result.get_mismatches("total_amount")

# The changed rows as a pandas DataFrame, if pandas is installed.
result.to_pandas()
```

## Summary

`summary` is a `DiffSummary`, a Pydantic model without any frames. `summary.model_dump_json()` returns the same JSON that `veridelta run --json` prints:

| Field | Description |
| :--- | :--- |
| `total_rows_source` | Rows in the source. |
| `total_rows_target` | Rows in the target. |
| `added_count` | Rows only in the target. |
| `removed_count` | Rows only in the source. |
| `changed_count` | Rows in both with at least one compared value that differs. |
| `column_mismatches` | For each column with at least one mismatch, the changed rows where it differs. |
| `total_mismatches` | Added, removed, and changed rows together. |
| `mismatch_ratio` | `total_mismatches` over `total_rows_source`. |
| `match_rate_percentage` | One minus `mismatch_ratio`, as a percentage rounded to two places. |
| `is_match` | Whether `mismatch_ratio` is at most `threshold`. |
| `accepted_count` | Rows of drift a [baseline](https://veridelta.github.io/veridelta/cli/#accepting-drift) accepted, which the counts above leave out. `0` without one. |
| `is_perfect_match` | Whether nothing differs. |
| `volume_shift` | `total_rows_target` minus `total_rows_source`. |
| `report_summary` | A plain-text report of these counts and the most drifting columns. |

[`schema/run.schema.json`](https://veridelta.github.io/veridelta/schema/run.schema.json) is the JSON Schema of this object, and `veridelta schema run` prints it; see [Printing the schema](https://veridelta.github.io/veridelta/cli/#printing-the-schema).

## Rows

`added` holds the rows found only in the target, and `removed` the rows found only in the source. `changed` holds the rows found in both with at least one difference. Each changed row carries its primary keys and, for every compared column, `{column}_source`, `{column}_target`, and `{column}_is_match`.

`get_mismatches(column)` returns the primary keys and both values for the rows where that column differs. It raises `ConfigError` for a column that was not compared. `to_pandas()` converts `changed` to pandas; it needs pandas and pyarrow, which Veridelta does not install.

### Pushdown results

[Pushdown](https://veridelta.github.io/veridelta/pushdown/) compares in place and never selects values, so its frames hold primary keys only and the result sets `keys_only`. `get_mismatches` then returns every changed key instead of one column's values, and still rejects a column that was not compared.

A run with `pushdown_sample_rows` set also carries `changed_sample`: up to that many changed rows with each compared column's values, laid out like a local run's `changed`. See [Row samples](https://veridelta.github.io/veridelta/pushdown/#row-samples).

## HTML report

![The top of an HTML report from a comparison of 120 orders: a FAILED verdict, a match rate of 87.5%, 120 source and 121 target rows, 3 added, 2 removed, and 10 changed, drift in status and amount, and the changed rows with both values side by side.](https://veridelta.github.io/veridelta/assets/report-light.png#only-light)
![The top of an HTML report from a comparison of 120 orders: a FAILED verdict, a match rate of 87.5%, 120 source and 121 target rows, 3 added, 2 removed, and 10 changed, drift in status and amount, and the changed rows with both values side by side.](https://veridelta.github.io/veridelta/assets/report-dark.png#only-dark)

`write_html` writes the standalone report that `veridelta run` writes with `--html`. It embeds its own styles and script, so it opens offline. Every row is in the page itself, and the script splits long tables into pages of 25 rows. `max_rows` caps every table, as `--html-max-rows` does:

```python
from veridelta.report import write_html, write_markdown

write_html(result, "reports/nightly.html", max_rows=1000)
write_markdown(result, "reports/summary.md")
```

## Markdown summary

`write_markdown` writes the short summary that CI posts to a job summary or a pull request, and `render_markdown` returns it as text. It holds the verdict, a table of counts, and the drifting columns, up to `report_top_columns_limit`. Column names are written as code, so a name from the data cannot break the table or the page it lands on.

The summary ends with the same counts as JSON, inside an HTML comment that GitHub and GitLab hide from readers. Its first line names the [run schema](https://veridelta.github.io/veridelta/schema/run.schema.json) the JSON follows:

```text
<!-- veridelta-summary https://veridelta.github.io/veridelta/schema/run.schema.json
{"total_rows_source":3,"total_rows_target":3,"added_count":1,"removed_count":1,"changed_count":1,"column_mismatches":{"status":1},"is_match":false,...}
-->
```

The JSON is what `veridelta run --json` prints, except that `column_mismatches` holds only the columns the drift table lists, and is left out when `report_top_columns_limit` is `0`. A column name's `<`, `>`, and `&` are written as the JSON escapes `\u003c`, `\u003e`, and `\u0026`, so a name cannot close the comment, and a JSON parser reads them back as the same name. A script, or an agent that reads a pull request comment through the API, parses the line after the marker:

```python
import json
import re

found = re.search(r"^<!-- veridelta-summary \S+\n(.+)$", body, re.MULTILINE)
summary = json.loads(found.group(1))
```

The summary lists no values unless asked. With `max_rows` above 0, or `--markdown-max-rows` on the command line, a "Changed values" table follows the drift table. It has one row per differing value: the primary keys, the column, and the source and target values, lowest keys first. From Python:

```python
from veridelta.report import write_markdown

write_markdown(result, "summary.md", max_rows=20)
```

Each value is written as code. Text is quoted, so a trailing space or an empty string shows, and NULL reads as _null_. A value longer than 60 characters is cut and ends in `...`. The table stops at `max_rows` values, or before the summary, its JSON comment included, reaches 60,000 bytes, which keeps a pull request comment under GitHub's limit. A last line then says how many values it showed. A pushdown run lists the values of its [row sample](https://veridelta.github.io/veridelta/pushdown/#row-samples), so it needs `pushdown_sample_rows` too.

CI posts the summary where more people may read it than may read the data. Ask for values only where every reader of the job summary and the pull request may see them.

## OpenTelemetry metrics

`veridelta run --otel otel-metrics.json` writes the run's metrics to a file, and `--otel-send` sends them to an OTLP/HTTP endpoint. Either reaches an observability backend, such as Datadog or Grafana, through an OpenTelemetry Collector or the backend's own OTLP intake. Neither needs an OpenTelemetry package. From Python:

```python
from veridelta import DiffEngine, load_config
from veridelta.telemetry import send_otlp_metrics, write_otlp_metrics

diff, source, target = load_config("veridelta.yaml")
result = DiffEngine.run_from_configs(diff, source, target)
write_otlp_metrics(
    result, "reports/otel-metrics.json", config_path="veridelta.yaml", source=source, target=target
)
send_otlp_metrics(result, config_path="veridelta.yaml", source=source, target=target)
```

Every metric is a gauge stamped with the time of the run's export, so each run adds one point to each series:

| Metric | Unit | Attributes | Value |
| :--- | :--- | :--- | :--- |
| `veridelta.dataset.rows` | `{row}` | `veridelta.side`: `source` or `target` | Rows in each dataset. |
| `veridelta.diff.rows` | `{row}` | `veridelta.diff.kind`: `added`, `removed`, or `changed` | Rows only in the target, only in the source, or in both with a value that differs. |
| `veridelta.column.mismatched_rows` | `{row}` | `veridelta.column.name` | Changed rows whose value in that column differs. Every compared column reports, so a column that stops drifting reads `0` instead of disappearing. |
| `veridelta.diff.mismatch_ratio` | `1` | | Added, removed, and changed rows over the source rows, as `mismatch_ratio` is. |
| `veridelta.diff.match` | | | `1` when the run matched within `threshold`, `0` when it drifted. It has no unit, since a backend such as Prometheus reads a unit of `1` as a ratio. |

Resource attributes say which comparison ran:

- `service.name`, which is `veridelta` unless the environment renames it, and `service.version`;
- `veridelta.config.path`, the configuration file;
- `veridelta.source.type` and `veridelta.target.type`, such as `file` or `snowflake`;
- `veridelta.source.name` and `veridelta.target.name`: the table, or the file or lakehouse path.

A URL keeps only its scheme, host, and path, so a token in its user part or a signature in its query never leaves the run. An Azure `abfss://container@account` path keeps its container, which sits where a user would. A database `query` source has no name.

Like the Markdown summary by default, the export holds counts and column names, never row values, connection URIs, credentials, or SQL. A run that fails before it has a result writes no file and sends nothing.

The file is one line of JSON in OTLP's JSON encoding, as the [OpenTelemetry file exporter format](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md) specifies. The Collector's OTLP JSON file receiver, in its contrib distribution, can read it. The same line is the body `--otel-send` posts.

### Sending to an endpoint

`--otel-send` posts the export to an OTLP/HTTP endpoint, such as a Collector or a vendor's OTLP intake. The standard OpenTelemetry variables configure it:

| Variable | Default | Meaning |
| :--- | :--- | :--- |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | Base URL. `/v1/metrics` is added to its path. |
| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | | Full URL, used as written in place of the base URL. |
| `OTEL_EXPORTER_OTLP_HEADERS` | | Request headers, such as an API key, as comma-separated `key=value` items. |
| `OTEL_EXPORTER_OTLP_TIMEOUT` | `10000` | Milliseconds to wait for the endpoint. |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/json` | The protocol. Only `http/json` is supported. |

`HEADERS`, `TIMEOUT`, and `PROTOCOL` also have a metrics form, such as `OTEL_EXPORTER_OTLP_METRICS_HEADERS`. The metrics form wins over the general one, as it does in the OpenTelemetry SDKs.

Header keys and values are percent-encoded, as in `OTEL_RESOURCE_ATTRIBUTES`, so `Bearer <token>` is written `Bearer%20<token>`. No header value appears in a log line or an error, and neither does the endpoint's query. The send follows no redirect, so a header never reaches a second host. Over HTTPS, the endpoint's certificate is checked against the system's trust store.

The send is one attempt, with no retry. When it fails, such as on an unreachable endpoint, an HTTP error, or the timeout, the run exits `3` after writing its files. Under `--json`, stdout then holds the error object instead of the summary. An unusable variable, or a protocol other than `http/json`, fails the run the same way before anything is sent.

`--otel` and `--otel-send` together write and send the same export.

### Attributes from the environment

The standard OpenTelemetry variables add resource attributes, so a run can carry its environment or its team:

- `OTEL_RESOURCE_ATTRIBUTES` holds comma-separated `key=value` items, such as `deployment.environment=prod,team=data`. Percent-encode a comma, an equals sign, or a percent sign inside a key or value.
- `OTEL_SERVICE_NAME` replaces `service.name`. It wins over a `service.name` in `OTEL_RESOURCE_ATTRIBUTES`, as the OpenTelemetry specification requires.

When the environment sets a key that Veridelta also sets, Veridelta's value wins, except for `service.name`. The instrumentation scope stays `veridelta`. The values are exported as written, so keep secrets out of them.

A malformed `OTEL_RESOURCE_ATTRIBUTES`, such as an item without `=` or a `%` that starts no valid escape, is ignored whole, as the specification recommends. The `veridelta.telemetry` logger then warns without repeating the value. On the command line, `--verbose` prints that warning. Without it, the variable is dropped silently.

## Artifacts

With `output_path` set, a run writes the differing rows to that directory as `added_rows`, `removed_rows`, and `changed_rows`, in the `output_format`. An empty set writes no file. Pushdown writes primary keys only, under other names; see [Differences from a local run](https://veridelta.github.io/veridelta/pushdown/#differences-from-a-local-run).

`output_format` is `parquet` (the default), `csv`, `json`, `ndjson`, or `arrow`. CSV and JSON have no type for bytes, so `csv`, `json`, and `ndjson` write a binary column, such as a MySQL `BIT`, as hexadecimal text, and refuse a column that nests binary values in a list or a struct. Excel is not offered, because writing a workbook needs a second dependency that a discrepancy file does not justify. Neither is Avro: the Polars writer cannot store several types a discrepancy frame can hold, such as `Int8`, unsigned integers, nanosecond or timezone-aware timestamps, and all-NULL columns.

Source: https://veridelta.github.io/veridelta/cli/

# Command line

The `veridelta` command runs a comparison, checks a configuration, proposes value maps and other rules, prints the JSON Schemas of its files and its output, and serves its checks and comparisons to an AI agent. `run`, `validate`, `crosswalk`, and `suggest` read `veridelta.yaml` unless `-c` names another file. `run` also takes two files and `--key` in place of a file.

| Command | Description |
| :--- | :--- |
| `veridelta run` | Compare the two datasets and report the result. |
| `veridelta validate` | Report what would stop a run, without reading any rows. |
| `veridelta crosswalk` | Propose `value_map` entries from the data. |
| `veridelta suggest` | Suggest rules that would explain the differences, each with its evidence. |
| `veridelta schema` | Print the JSON Schema of the configuration file, or of what a command prints with `--json`. |
| `veridelta mcp` | Serve checks and comparisons to an AI agent as Model Context Protocol tools, over stdio. |
| `veridelta --version`, `-V` | Print the installed version. |

## Running a comparison

`veridelta run` compares the configured datasets, prints a report, and exits with a [status code](https://veridelta.github.io/veridelta/cli/#exit-codes):

```bash
veridelta run -c veridelta.yaml
veridelta run -c veridelta.yaml --json
veridelta run legacy.csv modern.csv --key id
veridelta run -c veridelta.yaml --html report.html --markdown summary.md --otel otel-metrics.json
```

| Flag | Description |
| :--- | :--- |
| `-c`, `--config PATH` | Configuration file. Default `veridelta.yaml`. |
| `FILE FILE` | Two files to compare in place of a configuration file, the source and then the target. Needs `--key`, and refuses `-c`. See [Two files without a configuration](https://veridelta.github.io/veridelta/cli/#two-files-without-a-configuration). |
| `-k`, `--key COLUMN` | A primary key column of the two files. Repeat it for a key of several columns, such as `--key id --key day`. |
| `-v`, `--verbose` | Print each file opened, connection, read, and pushdown statement on stderr; see [Logging](https://veridelta.github.io/veridelta/cli/#logging). |
| `--json` | Print the [summary](https://veridelta.github.io/veridelta/results/#summary) as JSON on stdout instead of the text report. |
| `-q`, `--quiet` | Suppress progress messages on stderr. The report or the JSON still prints. |
| `--baseline PATH` | Accept the drift the JSON file lists, and fail only on drift it does not list. See [Accepting drift](https://veridelta.github.io/veridelta/cli/#accepting-drift). |
| `--save-baseline PATH` | Also write the drift this run finds as the file `--baseline` reads. |
| `--html PATH` | Also write a standalone [HTML report](https://veridelta.github.io/veridelta/results/#html-report), which loads nothing from a CDN. |
| `--html-max-rows N` | Rows per table in the HTML report, zero or more. Default 1000, so a large diff cannot produce a file too large to open. |
| `--markdown PATH` | Also write the [Markdown summary](https://veridelta.github.io/veridelta/results/#markdown-summary) that the [CI integrations](https://veridelta.github.io/veridelta/ci/) post. |
| `--markdown-max-rows N` | Changed values to list in the Markdown summary, lowest keys first. Default 0, which lists none. |
| `--otel PATH` | Also write the run's [OpenTelemetry metrics](https://veridelta.github.io/veridelta/results/#opentelemetry-metrics) to a file. |
| `--otel-send` | Also send the metrics to an OTLP/HTTP endpoint. See [Sending to an endpoint](https://veridelta.github.io/veridelta/results/#sending-to-an-endpoint). |

Progress messages always go to stderr, so `veridelta run --json | jq` needs no filtering. Reports and summaries from a pushdown run are labeled as holding primary keys only. An HTML report shows a [row sample](https://veridelta.github.io/veridelta/pushdown/#row-samples)'s values when the run fetched one, and so does a Markdown summary that lists values.

### Two files without a configuration

`veridelta run legacy.csv modern.csv --key id` runs what a file holding only `primary_keys: [id]` and the two paths would run, with the same summary and exit codes. Each file's format follows its suffix, as in a configuration. Rules, `threshold`, discrepancy files, and every other setting need a configuration file. The output flags, such as `--html` and `--json`, work the same either way.

An error then names the arguments rather than a configuration file. Two files without `--key`, `--key` without two files, or two files with `-c` exit `2`, as any invalid argument does.

In Python, `files_config(source, target, primary_keys)` from `veridelta.config` returns what `load_config` returns for that file.

## Exit codes

Every command exits `2` for invalid arguments. Every command but `schema`, which only prints, exits `3` when it cannot finish:

| Code | `run` | `validate` | `crosswalk` | `suggest` | `mcp` |
| :--- | :--- | :--- | :--- | :--- | :--- |
| `0` | A match within `threshold`. | No errors. Warnings are allowed. | Proposals computed, whether or not any were found. | Suggestions computed, whether or not any were found. | The host disconnected, or Ctrl-C stopped the server. |
| `1` | Drift. | At least one error. | Not used. | Not used. | Not used. |
| `2` | Invalid arguments. | Invalid arguments. | Invalid arguments. | Invalid arguments. | Invalid arguments, such as a `--root` that is not a directory. |
| `3` | The run could not finish, such as on a configuration error, an unreachable source, a missing extra, or metrics `--otel-send` could not send. | The check could not finish, such as when `--schemas` cannot reach a source. | The proposals could not be computed. | The suggestions could not be computed, such as for a pair compared where it is stored. | The server could not start, such as without the `mcp` extra. |

A command that cannot finish explains why on stderr. With `--json`, it also prints the error on stdout, as one JSON object in place of its usual output:

```json
{
  "error": {
    "type": "ConfigError",
    "message": "Environment variable 'SNOWFLAKE_PASSWORD' is not set, but source -> password references it. Set it, or write ${NAME:-default} to give a fallback."
  }
}
```

`type` names the error. A `ConfigError` is a problem with the configuration file. A `ConnectorError` comes from a source, or from the endpoint `--otel-send` posts to. A `DataIntegrityError` comes from a source's data. Any other type is a bug or an unsupported input: please [report it](https://github.com/Veridelta/veridelta/issues). An error `validate` finds in the file is a finding at exit `1`, in its usual output, not an error object.

`veridelta schema error` prints the JSON Schema of this object; see [Printing the schema](https://veridelta.github.io/veridelta/cli/#printing-the-schema).

## Logging

`-v` or `--verbose` prints Veridelta's own log lines on stderr, for `run`, `validate`, `crosswalk`, and `mcp`. Each line records a file opened, a connection, a read, or a pushdown statement. Reads and statements carry their timings:

```text
INFO veridelta.connectors.database: Read 1200 rows of table 'orders' from postgresql://analyst@db.internal/sales in 0.412s
```

A line names its source by host, account, project, database, or table, and masks a password inside a URI. It never holds SQL text, row values, or a credential. [Logging](https://veridelta.github.io/veridelta/sources/#logging) lists what each level records. The drivers' own loggers stay silent.

When a read or a statement fails, a warning names it and how long it ran, and the error follows. `--verbose` changes nothing else: stdout keeps the report or the JSON, and `--quiet` still drops the progress messages.

## Checking a configuration

`veridelta validate` reports what would stop a run, without reading any rows:

```bash
veridelta validate -c veridelta.yaml
veridelta validate -c veridelta.yaml --allow-missing-env --json
veridelta validate -c veridelta.yaml --schemas
```

| Flag | Description |
| :--- | :--- |
| `-c`, `--config PATH` | Configuration file. Default `veridelta.yaml`. |
| `-v`, `--verbose` | Print each file opened, connection, read, and pushdown statement on stderr; see [Logging](https://veridelta.github.io/veridelta/cli/#logging). |
| `--schemas` | Also connect, read each side's columns, never its rows, and check the rules against them. |
| `--allow-missing-env` | Read an unset `${NAME}` as the text `NAME`, with a warning, instead of failing. |
| `--json` | Print the findings as one JSON object on stdout. |
| `-q`, `--quiet` | Suppress the verdict line on stderr. |

By default, `validate` connects to nothing. It checks that:

- the source and target are a pair an engine can compare;
- every optional extra the run needs is installed, in the environment `validate` runs in;
- a database `table` uses a URI scheme Veridelta can quote;
- each `regex_replace` pattern compiles in Polars, whose regular expressions have no look-around and no backreferences, unlike Python's `re`.

On a pair compared in place, such as two warehouse tables, it also warns about settings the backend refuses only for some stored names or types:

- `normalize_column_names`;
- `min_jaro_winkler_similarity`;
- a `datetime_format` the backend cannot parse with, such as any `datetime_format` on Postgres;
- `max_levenshtein_distance` on Postgres and DuckDB;
- a `regex_replace` replacement that refers to a group by name.

A pattern Polars rejects is only a warning there, since the backend runs it with its own engine.

With `--schemas`, `validate` also connects and checks the rules against each side's columns:

- Files and lakehouse tables are opened as a run opens them, then checked with `DiffEngine.validate_rules`. JSON, Excel, and Avro files have no lazy reader, so they are read whole.
- A database `table` is read as a run reads it, with `WHERE 1 = 0` added, so no row is fetched. SQLite reports a `NUMERIC` column as text in that probe, although a full read returns numbers. A `query` is not run. If either side reads one, the rules are checked against neither side, and a warning names each `query` side.
- A pair compared in place runs the column probes a run starts with, then compiles every comparison statement without running it. That settles each warning above one way or the other.
- A name in a rule's `column_names` that neither side has, after `normalize_column_names` when it is on, is a warning. Such a rule does nothing. That is safe, since it only fails to forgive, but a misspelled name would otherwise go unnoticed.

Errors print to stdout as `error:` lines and warnings as `warning:` lines. With `--json`, they print as one object with `config`, `valid`, `errors`, and `warnings`. The verdict goes to stderr.

<details markdown>
<summary>Watch <code>validate --schemas</code> catch a wrong primary key</summary>

![A terminal shows a configuration whose primary key is order_id, and the first rows of a CSV file keyed by id. veridelta validate with --schemas reports that order_id is not among the columns read, and exits 1.](https://veridelta.github.io/veridelta/assets/demo-validate.gif)

```text
> cat orders.yaml
primary_keys: [order_id]
source:
  path: legacy.csv
target:
  path: modern.csv
> head -n 2 legacy.csv
id,status,amount
1,open,10.0
> veridelta validate -c orders.yaml --schemas
error: The primary key 'order_id' is not among the columns of the source, `legacy.csv` read as csv. The columns read are: 'id', 'status', 'amount'.
orders.yaml: 1 error, 0 warnings.
> echo "exit code: $?"
exit code: 1
```

</details>

`--allow-missing-env` checks a file without its secrets, such as in a pull request job. An unset `${NAME}` with no default is read as the text `NAME`, with a warning. That works for fields that take a name, such as `table`, `account`, `password`, or `path`. A field with a required shape, such as a database `uri`, still fails unless its variable is set.

In Python, `DiffEngine.check_config_file(path, schemas=False, allow_missing_env=False)` checks a file as `validate` does, and `DiffEngine.check_configs(diff, source, target, schemas=False)` checks models already loaded. Both return the findings as `ConfigFinding` models. `load_config(path, unset_env=[])` reads unset variables as their names and appends each name to the list.

## Proposing value maps

`veridelta crosswalk` prints `value_map` entries that the data supports, as YAML rules ready to paste into the configuration. [Proposing a value map](https://veridelta.github.io/veridelta/rules/#proposing-a-value-map) explains how entries are chosen:

```bash
veridelta crosswalk -c veridelta.yaml
veridelta crosswalk -c veridelta.yaml --min-confidence 0.99 --json
```

| Flag | Description |
| :--- | :--- |
| `-c`, `--config PATH` | Configuration file. Default `veridelta.yaml`. |
| `-v`, `--verbose` | Print each file opened, connection, read, and pushdown statement on stderr; see [Logging](https://veridelta.github.io/veridelta/cli/#logging). |
| `--min-confidence SHARE` | Share of a source value's rows that must agree on one target value, above 0.5. Default 0.95. |
| `--min-support N` | Agreeing rows an entry needs. Default 5. |
| `--sample-fraction SHARE` | Share of source rows to read, chosen by primary key. Default 1.0. |
| `--json` | Print the proposals and their evidence as JSON on stdout instead of YAML. |
| `-q`, `--quiet` | Suppress progress and evidence on stderr. |

<details markdown>
<summary>Watch <code>crosswalk</code> propose a value map</summary>

![A terminal shows the first rows of two CSV files whose active column holds Y and N on one side and true and false on the other. veridelta crosswalk proposes Y to true from 10 of 10 rows and N to false from 5 of 5 rows, and prints the rule as YAML.](https://veridelta.github.io/veridelta/assets/demo-crosswalk.gif)

```text
> head -n 4 crm.csv
id,active,tier
1,Y,silver
2,Y,bronze
3,N,gold
> head -n 4 warehouse.csv
id,active,tier
1,true,silver
2,true,bronze
3,false,gold
> veridelta crosswalk -c crosswalk.yaml
Loading configuration from crosswalk.yaml...
Lining up source and target values...
active: 2 new value_map entries
  'Y' -> 'true': 10 of 10 rows (100.0%)
  'N' -> 'false': 5 of 5 rows (100.0%)
rules:
- column_names:
  - active
  value_map:
    Y: 'true'
    N: 'false'
```

</details>

## Suggesting rules

`veridelta suggest` runs the comparison, then suggests rules that would explain the differences it finds, each with its evidence. It prints the rules as YAML on stdout, ready to paste first in `rules`, and the evidence on stderr. No model is called:

```bash
veridelta suggest -c veridelta.yaml
veridelta suggest -c veridelta.yaml --max-share 0.001 --json
```

| Flag | Description |
| :--- | :--- |
| `-c`, `--config PATH` | Configuration file. Default `veridelta.yaml`. |
| `-v`, `--verbose` | Print each file opened, connection, read, and pushdown statement on stderr; see [Logging](https://veridelta.github.io/veridelta/cli/#logging). |
| `--max-share SHARE` | Largest gap a tolerance may explain, as a share of the larger of its two values, above 0 and at most 1. Default 0.01. |
| `--json` | Print the suggestions and their evidence as JSON on stdout instead of YAML. |
| `-q`, `--quiet` | Suppress progress and evidence on stderr. |

A numeric column gets a tolerance when every gap between its differing values is at most `--max-share` of the larger of the two. A larger gap is a change, not noise, so that column gets no suggestion. The kind of tolerance follows the gaps:

- An absolute tolerance, when the gaps stay about one size whatever the values, as rounding leaves them.
- A relative tolerance, when the gaps grow with the values, as a rate change leaves them.

The tolerance is the first round value, such as 0.005 or 0.01, above the largest gap.

A text column gets trimming, `whitespace_mode: both`, when its differing values match once both ends are stripped, and case folding, `case_insensitive: true`, when they match once lowercased. It gets both only when some rows need both. Text that differs in anything else is a change, so no rule explains it.

A column with text on one side and dates or timestamps on the other gets `datetime_format` when the text reads as those dates under one format. The common formats are tried, such as `%Y-%m-%d`, `%d/%m/%Y`, and `%m/%d/%Y`, with or without a time, and the one that matches the most rows wins. A timestamp with a zone is left out, since text without an offset names no instant.

A column of any type gets `null_values` when a common spelling of NULL stands on one side where the other side is NULL. The spellings are an empty string, `N/A`, `NA`, `#N/A`, `NULL`, `(null)`, `<null>`, `None`, `nil`, `NaN`, `-`, `--`, and `?`, in any case, and the numbers -1, -9, -99, -999, -9999, and -99999. A suggested sentinel joins the column's sentinels today, from its rule or from `default_null_values`. A real value beside NULL, such as a city, is a change, and `false` is never suggested, since it is too often meant.

Then the comparison runs again with the rule, and the evidence counts the rows it makes match, of the rows that differ in that column, with up to three example keys, and for a tolerance, the largest gap. A row with a NULL on one side is explained only by a null sentinel. A rule that would make a row that matches today differ is not suggested, such as case folding ahead of a `value_map` whose keys are capitals.

Each printed rule names its column alone. When a rule governs the column today, such as one that matches it by `pattern`, the suggestion keeps that rule's settings, and a note says to put it first in `rules`, where it wins over the other. In Python, `DiffEngine.suggest_rules()` returns the suggestions as `RuleSuggestion` models.

`suggest` reads both sides locally, so it refuses a pair compared where it is stored, such as two warehouse tables. Suggest rules on files exported from them instead.

## Accepting drift

`veridelta run --baseline accepted.json` accepts the drift the file lists and fails only on drift it does not list, so a change made on purpose stops failing the run while any new drift still fails it:

```json
{
  "primary_keys": ["order_id"],
  "added": [{"order_id": 1121}],
  "removed": [{"order_id": 1017}],
  "changed": [{"key": {"order_id": 1034}, "columns": ["amount"]}]
}
```

`added` and `removed` list rows by their primary key. `changed` lists a row with the columns whose drift is accepted on it: drift in any other column of that row still counts. The keys are the run's `primary_keys`, as the run compares them, after any rule normalizes them. JSON has no type for a date, so a date key is written as text, such as `"2026-10-07"`, and read back as the key column's type.

Accepted drift is left out of the counts, the verdict, the artifacts, and the reports, and the summary's `accepted_count` says how many rows the file accepted. A run compared where its data is stored refuses `--baseline`, since its rows stay where they are. `veridelta schema baseline` prints the file's JSON Schema.

`veridelta run --save-baseline accepted.json` writes the file from a run, so a change made on purpose is accepted in one step: every row of drift the run finds, in key order, with each changed row's differing columns. Read the file before you commit it, since it accepts all of that drift. With `--baseline` too, the new file keeps what the old one accepted, and leaves out entries for rows that no longer drift. The run's verdict and exit code are the same as without the flag.

In Python, `Baseline` and its `AcceptedChange` entries import from `veridelta`. `Baseline.of(result)` builds the file's contents from a run, `write` saves them, `Baseline.read` loads a file, and `run_from_configs` and `run` take it as `baseline`:

```python
from veridelta import Baseline, DiffEngine, load_config

diff, source, target = load_config("veridelta.yaml")
result = DiffEngine.run_from_configs(diff, source, target)
Baseline.of(result).write("accepted.json")

accepted = Baseline.read("accepted.json")
result = DiffEngine.run_from_configs(diff, source, target, baseline=accepted)
```

## Printing the schema

`veridelta schema` prints the configuration file's JSON Schema, which editors use to complete and check a file; see [Editor support](https://veridelta.github.io/veridelta/configuration/#editor-support):

```bash
veridelta schema > veridelta.schema.json
```

Given the name of an output, it prints the JSON Schema of what a command prints with `--json` instead, so a script or an agent can check what it parses:

```bash
veridelta schema run > run.schema.json
```

| Name | The JSON it describes |
| :--- | :--- |
| `run` | The summary `veridelta run --json` prints: the row counts, the verdict, and the drifting columns. |
| `validate` | The report `veridelta validate --json` prints: whether the file is valid, its errors, and its warnings. |
| `crosswalk` | The list of value maps `veridelta crosswalk --json` proposes. |
| `suggest` | The list of rules `veridelta suggest --json` suggests, with their evidence. |
| `baseline` | The file that `run` reads with `--baseline`: the drift a run accepts. See [Accepting drift](https://veridelta.github.io/veridelta/cli/#accepting-drift). |
| `error` | The one object `run`, `validate`, `crosswalk`, or `suggest` prints with `--json` in place of its usual output when it [exits 3](https://veridelta.github.io/veridelta/cli/#exit-codes). |

The docs site serves each one too, at the URL its `$id` names, such as [`schema/run.schema.json`](https://veridelta.github.io/veridelta/schema/run.schema.json). A schema changes with the release that changes its output, and [the changelog](https://veridelta.github.io/veridelta/changelog/) says so.

## Serving tools to an agent

`veridelta mcp` serves Veridelta's checks and comparisons to an AI agent as [Model Context Protocol](https://modelcontextprotocol.io/) tools. The agent's host, such as Claude Code or Cursor, starts the command and speaks to it over stdin and stdout. It needs the `mcp` extra, and [AI agents](https://veridelta.github.io/veridelta/agents/#mcp-server) shows how to register it with a host and lists its tools. This serves the configuration files in the current directory:

```bash
veridelta mcp --root .
```

| Flag | Description |
| :--- | :--- |
| `--root DIR` | A folder the tools may read configuration files and data on this machine from. Repeat it for more folders. Default: the current directory. |
| `--allow-row-values` | Let `read_discrepancies`, `propose_value_maps`, and `suggest_rules` return values from the data. Off by default. |
| `--allow-queries` | Let a tool run a side's `query`, which runs as written. A DuckDB file still reads other files only from under the roots. Off by default. |
| `--max-rows N` | The most rows, value map entries, or example keys one call returns, at least 1. Default: 50. |
| `-v`, `--verbose` | Print each file opened, connection, read, and pushdown statement on stderr; see [Logging](https://veridelta.github.io/veridelta/cli/#logging). |

A tool refuses a configuration file, or data on this machine, outside every root, and a tool that runs the comparison refuses a configuration whose `output_path` lies outside them. Without `--allow-row-values`, no tool returns a value from the data, and without `--allow-queries`, no tool runs a side's `query`. The server runs in the first root, so a relative path in a tool call, or in a configuration file, resolves there. Stdout carries the protocol and nothing else, and log lines go to stderr, which the host keeps. The server stops when the host disconnects, or on Ctrl-C.

Source: https://veridelta.github.io/veridelta/ci/

# CI integrations

The GitHub Action and the GitLab CI template run `veridelta run` in a pipeline, then report the result:

- they post the Markdown summary where reviewers look;
- they keep the JSON summary, the HTML report, and the OpenTelemetry metrics as artifacts;
- they fail the job on drift or on an error.

The examples pin `v0.35.1`.

## GitHub Actions

This workflow compares the datasets on every pull request and comments with the summary:

```yaml
name: Data parity
on: pull_request

permissions:
  contents: read
  pull-requests: write   # only needed for the summary comment

jobs:
  compare:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: Veridelta/veridelta@v0.35.1
        with:
          config: veridelta.yaml
          extras: snowflake
        env:
          SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
```

The comment looks like this one, from an [example pull request](https://github.com/Veridelta/veridelta-media/pull/13) that compares the demo's 40 accounts with a [baseline](https://veridelta.github.io/veridelta/ci/#accepting-drift):

![A pull request comment by github-actions titled "Veridelta: FAILED": a match rate of 97.5%, 40 source and 39 target rows, 1 changed, 1 accepted by the baseline, drift in region, and the changed value, account 17's region from south to east.](https://veridelta.github.io/veridelta/assets/action-comment-light.png#only-light)
![A pull request comment by github-actions titled "Veridelta: FAILED": a match rate of 97.5%, 40 source and 39 target rows, 1 changed, 1 accepted by the baseline, drift in region, and the changed value, account 17's region from south to east.](https://veridelta.github.io/veridelta/assets/action-comment-dark.png#only-dark)

**Version.** Pin the action to a release tag such as `v0.35.1`, or to a commit SHA. The action installs Veridelta from its own ref, so the tag you pin is the version that runs. To keep the action at one ref and install a different version from PyPI, set `version`.

**Credentials.** Pass credentials as step environment variables, as above, and reference them from the configuration as `${SNOWFLAKE_PASSWORD}`. See [Environment variables](https://veridelta.github.io/veridelta/configuration/#environment-variables).

**What it does.** The action:

- appends the summary to the job summary;
- uploads `summary.json`, `summary.md`, `report.html`, and `otel-metrics.json` as one artifact;
- on `pull_request` and `pull_request_target` events, keeps one comment per `config` path up to date on the pull request. Matrix legs that run the same path write over each other's comment, so set `comment` to `false` on all but one.

**Forks.** A pull request from a fork gets a read-only token, so the comment step logs a warning instead of failing.

**Untrusted configurations.** A configuration can read any URL and put any environment variable in it, so it can send a secret to whoever wrote it. Run the action with secrets in its environment only on a configuration from a ref you trust. A `pull_request` from a fork gets no secrets, which keeps that event safe. Never run the action under `pull_request_target` with the pull request's head checked out: that event has your secrets and a write token, while the head holds whatever the pull request's author wrote.

### Inputs

| Input | Default | Meaning |
| :--- | :--- | :--- |
| `config` | `veridelta.yaml` | Configuration file, relative to `working-directory`. |
| `baseline` | empty | Baseline file of drift the run accepts, relative to `working-directory`. See [Accepting drift](https://veridelta.github.io/veridelta/ci/#accepting-drift). |
| `working-directory` | `.` | Directory the comparison runs in. |
| `extras` | empty | Optional extras to install, comma-separated, such as `database,fuzzy`. |
| `version` | empty | Install this version from PyPI instead of the action's own ref. |
| `python-version` | `3.12` | Python to run Veridelta with. |
| `html-max-rows` | `1000` | Rows per table in the HTML report. |
| `markdown-max-rows` | `0` | Changed values to list in the summary. See [Values in the summary](https://veridelta.github.io/veridelta/ci/#values-in-the-summary). |
| `otel-send` | `false` | Send the run's OpenTelemetry metrics to an OTLP/HTTP endpoint. See [Sending metrics to an observability backend](https://veridelta.github.io/veridelta/ci/#sending-metrics-to-an-observability-backend). |
| `fail-on-mismatch` | `true` | Fail the step on drift. An error always fails it. |
| `comment` | `true` | Keep a summary comment on the pull request. |
| `github-token` | `github.token` | Token used to comment. |
| `upload-artifact` | `true` | Upload the reports as an artifact. |
| `artifact-name` | derived | Artifact name. Set it when a matrix runs one configuration more than once, since artifact names must be unique within a run. |

`extras` takes lowercase extra names, `version` a release number, and `artifact-name` letters, digits, spaces, dots, underscores, and hyphens. The action refuses any other value before it installs anything: `exit-code` is `2`, the command line's code for invalid arguments, and `status` is `error`.

### Outputs

| Output | Meaning |
| :--- | :--- |
| `status` | `match`, `drift`, or `error`. |
| `is-match` | `true` when the comparison matched within its threshold. |
| `exit-code` | Exit code of `veridelta run`. |
| `summary-json`, `summary-markdown`, `report-html` | Paths to the reports. Empty when the run did not finish. |
| `otel-metrics` | Path to the run's [OpenTelemetry metrics](https://veridelta.github.io/veridelta/results/#opentelemetry-metrics). Empty when the run did not finish. |

To act on drift in a later step instead of failing, set `fail-on-mismatch: false` and read `status`.

The pull request comment ends with the run's counts as JSON, in an HTML comment that readers never see, so an agent that reads the pull request through the API can parse the result; see [Markdown summary](https://veridelta.github.io/veridelta/results/#markdown-summary).

### Accepting drift

A change made on purpose, such as an account closed for good, fails every pull request until a baseline accepts it. Write the file once with [`veridelta run --save-baseline accepted.json`](https://veridelta.github.io/veridelta/cli/#accepting-drift), read it, commit it next to the configuration, and pass it to the action:

```yaml
      - uses: Veridelta/veridelta@v0.35.1
        with:
          config: veridelta.yaml
          baseline: accepted.json
```

The run then fails only on drift the file does not list, and the summary counts the rows it accepted. A missing or invalid file ends the run with `status` set to `error`, as does a pair compared where it is stored, which refuses a baseline.

### Values in the summary

The summary lists counts and column names, never values, unless `markdown-max-rows` is above `0`. It then lists up to that many [changed values](https://veridelta.github.io/veridelta/results/#markdown-summary), lowest keys first. They appear in the job summary and the pull request comment, where anyone who can read the pull request can read them. Set it only where every such reader may see the data.

### Sending metrics to an observability backend

The action writes the run's [OpenTelemetry metrics](https://veridelta.github.io/veridelta/results/#opentelemetry-metrics) on every run. Set `otel-send: true` to also send them to an OTLP/HTTP endpoint, such as a Collector or a vendor's OTLP intake. The step's `env` names the endpoint, and any header your backend requires, from secrets:

```yaml
      - uses: Veridelta/veridelta@v0.35.1
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: ${{ secrets.OTLP_ENDPOINT }}
          OTEL_EXPORTER_OTLP_HEADERS: ${{ secrets.OTLP_HEADERS }}
          OTEL_RESOURCE_ATTRIBUTES: deployment.environment=ci,team=data
        with:
          config: veridelta.yaml
          otel-send: true
```

`OTLP_HEADERS` holds items such as `api-key=<key>`. [Sending to an endpoint](https://veridelta.github.io/veridelta/results/#sending-to-an-endpoint) lists every variable. `OTEL_RESOURCE_ATTRIBUTES` tags the run with [attributes from the environment](https://veridelta.github.io/veridelta/results/#attributes-from-the-environment). A send that fails makes the run an `error`, which fails the step even when the comparison matched.

## GitLab CI

Include the template from a release tag, with its inputs:

```yaml
include:
  - remote: https://raw.githubusercontent.com/Veridelta/veridelta/v0.35.1/ci/gitlab/veridelta.yml
    inputs:
      config: veridelta.yaml
      extras: snowflake
```

The template defines one job, named `veridelta` by default, which:

- installs the release the template ships with;
- prints the summary to the job log;
- keeps the reports and the OpenTelemetry metrics as artifacts, exposed on the merge request as "Veridelta report", unless `upload-artifact` is `false`;
- with `otel-send` set to `true`, also sends the metrics, to the endpoint that masked CI/CD variables such as `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` name.

**Merge request notes.** To keep a summary note on the merge request, add a project access token with the `api` scope as a masked CI/CD variable named `VERIDELTA_GITLAB_TOKEN`. Without it, the job still runs and reports. The job keeps one note per `config` path. Two jobs that run the same path from different working directories write over each other's note.

**Inputs:**

- `config`, `working-directory`, `extras`, `html-max-rows`, `markdown-max-rows`, `otel-send`, `fail-on-mismatch`, `comment`, and `upload-artifact` mean what they do for the GitHub Action, with the same defaults. With `markdown-max-rows` above `0`, values appear in the job log and the merge request note.
- `version` defaults to the release the template ships with.
- `python-version` is empty by default, which runs the image's Python. Set it, such as to `"3.13"`, and uv downloads that version when the image lacks it.
- `stage`, `job-name`, and `image` place the job in your pipeline.

There is no `github-token` input, since the note reads the masked `VERIDELTA_GITLAB_TOKEN` variable. There is no `artifact-name` either, since GitLab keeps each job's artifacts apart. The template has no `baseline` input: it stays as it is unless a GitLab user asks for more, so open an issue to ask for one.

With `upload-artifact` set to `false`, the reports go to a temporary directory instead. GitLab then logs that no files match the artifact path, which does not change the job's result.

**Credentials.** Set them as masked CI/CD variables; the configuration reads them as `${NAME}`.

## Checking configurations in review

`veridelta validate` catches a configuration that cannot run before the comparison job does, and needs no credentials:

```yaml
- uses: astral-sh/setup-uv@v10
- run: uvx veridelta@0.35.1 validate -c veridelta.yaml --allow-missing-env
```

`--allow-missing-env` reads each unset `${NAME}` as the text `NAME`, with a warning, so the job needs no secrets. The job exits `1` on an error, and a warning never fails it. Install the same extras the comparison uses, such as `uvx --from 'veridelta[snowflake]==0.35.1' veridelta validate ...`: `validate` checks the environment it runs in. See [Checking a configuration](https://veridelta.github.io/veridelta/cli/#checking-a-configuration).

## Exit codes and statuses

Both integrations read `veridelta run`'s exit code together with its JSON summary:

| Status | Exit code | Meaning |
| :--- | :--- | :--- |
| `match` | 0 | The comparison is within its threshold. |
| `drift` | 1 | The JSON summary reports `is_match: false`. |
| `error` | 3, or any other | The run did not finish, such as a configuration error, an unreachable source, a missing extra, or metrics `otel-send` could not send. There is no summary, and the job always fails. |

A pinned older release also exits `1` when it fails, so `1` counts as drift only when the summary says so.

Source: https://veridelta.github.io/veridelta/agents/

# AI agents

An AI agent, such as a coding assistant, runs Veridelta through its command line or through its MCP server. This page gives the steps that keep its runs predictable and its replies free of row values.

## Steps

1. Check the configuration file. `validate` reads no rows, and by default connects to nothing:

    ```bash
    veridelta validate -c veridelta.yaml --json
    ```

    The JSON holds the `config` it checked, `valid`, `errors`, and `warnings`, and the command exits `1` when there is an error. Fix each error before a run. A check that cannot finish exits `3` and prints an `error` object instead. `--allow-missing-env` checks a file whose secrets are not set, such as in a pull request job.

2. Run the comparison with `--json`, so stdout carries only the [summary](https://veridelta.github.io/veridelta/results/#summary):

    ```bash
    veridelta run -c veridelta.yaml --json
    ```

3. Read the run's [exit code](https://veridelta.github.io/veridelta/cli/#exit-codes) before its output:

    | Code | Meaning | stdout |
    | :--- | :--- | :--- |
    | `0` | A match within `threshold`. | The summary. |
    | `1` | Drift. | The summary. |
    | `2` | Invalid arguments. | Nothing. |
    | `3` | The run could not finish. | One object, `{"error": {"type": ..., "message": ...}}`, which stderr explains too. |

    Read `type` before acting on `message`. A `ConfigError` means the configuration file needs a fix, and [Exit codes](https://veridelta.github.io/veridelta/cli/#exit-codes) says what the other types mean. Any type that is not a Veridelta error, such as one from a driver, is worth reporting to the user as a possible bug.

4. Report counts and column names, which is all the summary holds. Leave row values out of a reply unless the user asks for them.

5. When two columns hold the same values in different encodings, such as `Y` and `true`, propose a `value_map` from the data:

    ```bash
    veridelta crosswalk -c veridelta.yaml --json
    ```

    Show the proposals and their evidence to the user before adding them to the configuration.

6. When a column differs by small amounts, such as rounding, padding, case, a date format, or a spelling of NULL, suggest a rule from the data:

    ```bash
    veridelta suggest -c veridelta.yaml --json
    ```

    Each suggestion names the rows it explains, and its example keys come from the data. Show the suggestions to the user before adding a rule: a rule forgives what it explains in later runs too, such as every gap below a tolerance.

## MCP server

`veridelta mcp` serves steps 1, 2, 5, and 6 above as [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) tools, so an agent's host can call them without a shell. Two more tools list a side's columns and read the rows that differ. It needs the `mcp` extra:

```bash
uv add 'veridelta[mcp]'
```

Register the server with the host from the folder that holds the configuration files. In Claude Code, this command does it:

```bash
claude mcp add veridelta -- uv run veridelta mcp --root .
```

A host that reads its servers from a file takes the same command as an entry. Claude Code reads `.mcp.json` at the project's root, and Cursor reads `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "veridelta": {
      "command": "uv",
      "args": ["run", "veridelta", "mcp", "--root", "."]
    }
  }
}
```

Where Veridelta is installed without uv, the command is `veridelta mcp` itself. To let the agent read rows, add `--allow-row-values` to the command. The server has six tools, and each takes `path`, the configuration file, which a relative path reads against the first root:

| Tool | Returns | Reads |
| :--- | :--- | :--- |
| `validate_config` | What `veridelta validate --json` prints: `config`, `valid`, `errors`, and `warnings`. Its `schemas` and `allow_missing_env` arguments work as the command's flags do. | No rows. With `schemas`, each side's columns. |
| `run_comparison` | What `veridelta run --json` prints, the [summary](https://veridelta.github.io/veridelta/results/#summary), with `verdict`, `match` or `drift`, the `exit_code` the command gives, 0 or 1, and `artifacts_written`. | Both sides. |
| `describe_schema` | One side's `columns`, each name as stored mapped to its type as Polars names it, such as `Int64`, and the `side`. Its `side` argument is `source` or `target`. | That side's columns, and no rows. A side that reads a `query` is refused, since only running it would name its columns. |
| `read_discrepancies` | The rows of one `kind`, `added`, `removed`, or `changed`, up to `limit`, 20 by default: `kind`, `total`, `rows`, `truncated`, and `keys_only`, which says the pair was compared in place, so each row holds its primary key alone. | Both sides, since each call runs the comparison. Only on a server started with `--allow-row-values`. |
| `propose_value_maps` | What `veridelta crosswalk --json` prints, as `proposals`, with `total` and `truncated`. Its `min_confidence`, `min_support`, and `sample_fraction` arguments work as the command's flags do. | Both sides. Only on a server started with `--allow-row-values`. |
| `suggest_rules` | What `veridelta suggest --json` prints, as `suggestions`, with `total` and `truncated`. Its `max_share` argument works as the command's `--max-share` does. Each suggestion's `examples` hold primary keys from the data. | Both sides, read locally, so a pair compared in place is refused. Only on a server started with `--allow-row-values`. |

<details markdown>
<summary>Watch a client call three tools, as an agent's host does</summary>

![A terminal runs a script that starts veridelta mcp with row values allowed and calls three tools. validate_config answers valid with no errors. run_comparison answers drift, exit code 1, with one added, one removed, and one changed row, and one mismatch in status. read_discrepancies returns the changed row, id 2, whose status is closed in the source and shipped in the target.](https://veridelta.github.io/veridelta/assets/demo-mcp.gif)

```text
> # A client calls the MCP server's tools, as an agent's host does.
> python mcp_client.py
-> validate_config {"path": "veridelta.yaml"}
<- {
  "valid": true,
  "errors": [],
  "warnings": []
}
-> run_comparison {"path": "veridelta.yaml"}
<- {
  "verdict": "drift",
  "exit_code": 1,
  "added_count": 1,
  "removed_count": 1,
  "changed_count": 1,
  "column_mismatches": {
    "status": 1
  }
}
-> read_discrepancies {"path": "veridelta.yaml", "kind": "changed"}
<- {
  "total": 1,
  "truncated": false,
  "rows": [
    {
      "id": 2,
      "status_source": "closed",
      "amount_source": 20.5,
      "status_target": "shipped",
      "amount_target": 20.5,
      "status_is_match": false,
      "amount_is_match": true
    }
  ]
}
```

</details>

The person who starts the server decides what it may read, and no tool call can change that:

- Each `--root` names a folder the tools may read configuration files from, and a path outside every root fails the call. The server runs in the first root, so a relative path resolves there.
- A tool that opens a side reads files on this machine only from under the roots, `~` and links included, since a column name or an error can carry a file's text as a row does. Data elsewhere, such as an object store or a warehouse, is read as the command line reads it.
- A run writes the rows that differ to the configuration's `output_path`, so `run_comparison` and `read_discrepancies` refuse one outside every root before they read a row.
- A side's `query` runs as written, with the configuration's credentials, so a tool that would run one refuses unless the server is started with `--allow-queries`. Even then, a DuckDB file's connection reads other files, attaches databases, and loads extensions only from under the roots. A MotherDuck connection is not held to the roots.
- A tool returns findings, counts, and column names, never the configuration or a value from the data, unless the server is started with `--allow-row-values`. A password inside an error is masked, as on the command line, and so is every value of four characters or more that the configuration takes from an environment variable, wherever it appears in an answer.
- With `--allow-row-values`, `read_discrepancies` returns at most `--max-rows` rows per call, 50 by default. `propose_value_maps` returns at most that many value map entries, counting the entries a column's rule already has, and a proposal comes back whole or not at all. `suggest_rules` returns at most that many example keys, and a suggestion comes back whole or not at all.
- With `schemas`, a check reads each side's columns as `veridelta validate --schemas` does, and without it a check opens no data. `describe_schema` reads one side's columns the same way, and a warehouse table with the probe a run starts with.
- A host may start the server with only some of the user's environment variables. A `${NAME}` that the configuration references must reach the server, through the host's `env` setting for it if need be, or the check reports it unset. `allow_missing_env` checks a file without them.

A call that fails returns its error's type and message, as `run --json` prints them, such as `ConfigError` for a path outside the roots. [Serving tools to an agent](https://veridelta.github.io/veridelta/cli/#serving-tools-to-an-agent) lists the command's flags.

## Where row values appear

The summary holds counts and column names only. These outputs hold values from the data:

- the discrepancy files a run writes to `output_path`;
- the [HTML report](https://veridelta.github.io/veridelta/results/#html-report);
- a [Markdown summary](https://veridelta.github.io/veridelta/results/#markdown-summary) with `--markdown-max-rows` above zero;
- the proposals `veridelta crosswalk` prints;
- what `read_discrepancies`, `propose_value_maps`, and `suggest_rules` return, on an MCP server started with `--allow-row-values`.

A pair compared in place, such as two warehouse tables, brings back counts and primary keys only, unless it fetches a [row sample](https://veridelta.github.io/veridelta/pushdown/#row-samples).

## Writing a configuration

`veridelta schema` prints the JSON Schema of the configuration file. Check a draft against it, then run `veridelta validate`.

## Checking the output

`veridelta schema run` prints the JSON Schema of what `veridelta run --json` prints, and `validate`, `crosswalk`, `suggest`, `baseline`, and `error` name the others. The docs site serves the same files; see [Printing the schema](https://veridelta.github.io/veridelta/cli/#printing-the-schema). A script or an agent can check what it parses against them.

On a pull request, the comment the [GitHub Action](https://veridelta.github.io/veridelta/ci/#github-actions) keeps ends with the run's summary as JSON, in an HTML comment that readers never see. It holds counts and the column names the comment shows, never a value, and [Markdown summary](https://veridelta.github.io/veridelta/results/#markdown-summary) shows how to parse it.

## Fixing drift in a loop

An agent that changes a pipeline can check its own work against the data the pipeline produced before, and keep fixing until the two match:

1. Change the pipeline. Push the change to the pull request, or run the pipeline locally.
2. Let the comparison run: the [GitHub Action](https://veridelta.github.io/veridelta/ci/#github-actions) on the pull request, or `veridelta run --json` locally.
3. Read the result and its exit code. On the pull request, parse the JSON at the end of the Action's comment, as [Markdown summary](https://veridelta.github.io/veridelta/results/#markdown-summary) shows. Locally, read what `run --json` prints. Both hold `added_count`, `removed_count`, and `changed_count`. `column_mismatches` holds every drifting column in `run --json`, and in the comment the ones its table lists.
4. Find the cause in the change, not in the data. The columns that drift, and how many rows each one changes, point to the code that writes them. With the MCP server started with `--allow-row-values`, `read_discrepancies` returns the rows that differ.
5. Fix the change, and go back to step 1.

Stop when the run matches, and report the counts. Stop too when the drift that remains is what the user asked for, such as a new rounding, and say which columns it is in. Never add a rule, or raise `threshold`, to make a run pass unless the user agrees: a rule changes what counts as a match, for every later run too. When the user accepts the drift that remains, `veridelta run --save-baseline accepted.json` records it, and later runs pass `--baseline accepted.json`, so they fail only on new drift; see [Accepting drift](https://veridelta.github.io/veridelta/cli/#accepting-drift).

## Docs for language models

The site publishes two plain-text files for language models, as the [llms.txt proposal](https://llmstxt.org/) describes:

- [`llms.txt`](https://veridelta.github.io/veridelta/llms.txt) links every page, each with its opening sentence.
- [`llms-full.txt`](https://veridelta.github.io/veridelta/llms-full.txt) holds every page of prose in one file, this one included.

## Agent skill

The steps above are also an agent skill, in [`skills/veridelta/SKILL.md`](https://github.com/Veridelta/veridelta/blob/main/skills/veridelta/SKILL.md). An agent that follows the Agent Skills standard loads it from a skills folder in the project. This installs it where Claude Code reads skills; for another agent, use the folder its documentation names:

```bash
mkdir -p .claude/skills/veridelta
curl -fsSL -o .claude/skills/veridelta/SKILL.md https://raw.githubusercontent.com/Veridelta/veridelta/main/skills/veridelta/SKILL.md
```

To pin the skill to a release, put the release's tag in place of `main`.

Source: https://veridelta.github.io/veridelta/upgrading/

# Upgrading

Upgrading moves a project to a newer Veridelta release. Most releases ask nothing of you. This page lists each release that does, newest first, and what to change.

Veridelta is below 1.0, so a release that breaks something raises the minor version, such as 0.27.1 to 0.28.0. A patch release, such as 0.31.0 to 0.31.1, needs no change to your configuration or code, though a fix can change a verdict that was wrong. The [changelog](https://veridelta.github.io/veridelta/changelog/) lists every change in every release.

## How to upgrade

Read the section below for each release between yours and the new one. Then move the version in a uv project:

```bash
uv lock --upgrade-package veridelta
uv sync
```

Check each configuration file against the new release, which reports any setting it now refuses:

```bash
veridelta validate -c veridelta.yaml
```

Then run a comparison whose verdict you already know, and confirm the new release agrees. In CI, move the GitHub Action's `@v` ref, and the tag in the GitLab template's `remote` URL, to the new release, as [CI integrations](https://veridelta.github.io/veridelta/ci/) shows.

## 0.33.0

Pushdown refuses a compared column whose two sides hold different types, unless both are numeric. It raises `ConfigError` before reading a row, and names each such column with both of its types. Before, a database converted one side by its own rules, so pushdown could reach another verdict than a local run. The text `"007"` against the integer `7` matched in DuckDB, where a local run reports drift.

Give each named column a rule with `cast_to`, so both sides hold one type:

```yaml
rules:
  - column_names: ["code"]
    cast_to: "Int64"
```

Or set `strict_types: true` to fail every row of such a column instead. A local run compares as before.

## 0.32.0

`veridelta.config` no longer re-exports the seven source models. Importing `BigQueryConfig`, `DatabaseConfig`, `DatabricksConfig`, `DeltaLakeConfig`, `DuckDBConfig`, `IcebergConfig`, or `SnowflakeConfig` from it raises `ImportError`. Import them from the package root, or from `veridelta.models`:

```python
from veridelta import SnowflakeConfig
```

`veridelta.config` still exports `load_config`, `files_config`, `config_json_schema`, `referenced_variables`, `SCHEMA_URL`, and `SourceRef`.

## 0.28.0

The `compile_` methods of `SQLPushdownCompiler` that read both sides no longer take `source_alias` or `target_alias`. They are `compile_query`, `compile_column_predicate`, `compile_changed_sample_query`, `compile_missing_query`, `compile_added_query`, `compile_column_mismatch_query`, and `compile_value_map_query`. A call that passes either raises `TypeError`. Drop the argument: the aliases were always `src` and `tgt`.

Only code that calls the compiler itself is affected. Configuration files and the command line are not.

## 0.15.0

`DataIngestor` no longer exists. `DiffEngine.run_from_configs` loads, aligns, and compares both sides in its place:

```python
from veridelta import DiffEngine, load_config

diff, source, target = load_config("veridelta.yaml")
result = DiffEngine.run_from_configs(diff, source, target)
```

`fetch_schema()` no longer exists on a connector. Read a reader's schema with `lazyframe().collect_schema()`. Probe a warehouse table with `compile_schema_probe_query(table)`, run through `execute_pushdown(sql, query_type="schema")`.

A connector of your own now subclasses one of two bases. One the local engine reads subclasses `ReaderConnector`. One that runs compiled SQL subclasses `PushdownSession`, and defines `compiler` and `execute_pushdown()`. A reader has no `execute_pushdown`.

## 0.14.0

`run`, `validate`, and `crosswalk` exit with `3`, not `1`, when they cannot finish, such as when a file is missing. With `--json`, such a failure prints `{"error": {"type": ..., "message": ...}}` on stdout, where it printed nothing.

A CI step that read every nonzero exit as drift still fails, which is right. A script that tells drift from a failure reads `1` as drift and `3` as a comparison that did not run. The [command line](https://veridelta.github.io/veridelta/cli/#exit-codes) page lists every exit code.

## 0.12.0

Veridelta requires Python 3.11 or later.

## 0.11.0

Four comparisons became stricter. A run can report a difference that the release before it missed:

- `%f` in `datetime_format` reads `.5` as 500 milliseconds, not 5 nanoseconds.
- A local integer tolerance uses the true difference, so a pair whose difference overflowed into a match now mismatches.
- `strict_types: true` fails a warehouse column whose type differs after normalization.
- The crosswalk thresholds must be an `int` or a `float`, and not a boolean. `min_support` must be an `int`.

## 0.10.0

- A literal `${` inside `source` or `target` is written `$${`. A single `${` names an environment variable.
- Integer and float columns compare by value, so a pair that differs only in the fraction now mismatches in a local run.
- `load_nyc_taxi` raises `DatasetError`, which `except RuntimeError` no longer catches. Catch `DatasetError`, or `VerideltaError` for every error Veridelta raises.
- The tutorial notebooks were renamed, so their old URLs return 404.

## 0.7.0

`DiffEngine.run()` and `DiffEngine.run_from_configs()` return a `DiffResult`, where they returned a `DiffSummary`. The summary is on its `summary` attribute:

```python
summary = DiffEngine(config, source_frame, target_frame).run().summary
```

`SourceType` and `output_format` accept only the formats Veridelta reads and writes. A removed name, such as `sql` or `xml`, never worked, and now fails when the configuration loads.

## 0.6.0

`cast_to` accepts only `Int64`, `Float64`, `String`, `Boolean`, `Date`, and `Datetime`. Any other name fails when the configuration loads. Before, an unknown name left the column uncast, so a typo could hide a difference.

Source: https://veridelta.github.io/veridelta/roadmap/

# Roadmap

This page lists work that is not built yet. The current release is v0.35.1. Each item names the use case it serves, with a link to its card in [who Veridelta serves](https://github.com/Veridelta/veridelta/blob/main/product/USERS.md), or says that none asks for it yet. A test holds every item to that.

## Warehouses

- Pushdown for more SQL dialects, such as Redshift and Synapse. Serves [UC-05](https://github.com/Veridelta/veridelta/blob/main/product/USERS.md#uc-05-compare-two-tables-where-they-are-stored).

## Editor

These steps build on [Editor support](https://veridelta.github.io/veridelta/configuration/#editor-support), one at a time, tracked in [issue 127](https://github.com/Veridelta/veridelta/issues/127):

- A VS Code extension, in its own repository, whose first command runs `veridelta validate --json` on the open file and fills the Problems panel. Serves [UC-01](https://github.com/Veridelta/veridelta/blob/main/product/USERS.md#uc-01-a-first-verdict-on-two-files).
- A second command that runs the comparison, opens the HTML report in the editor, and opens the discrepancy files. Serves [UC-01](https://github.com/Veridelta/veridelta/blob/main/product/USERS.md#uc-01-a-first-verdict-on-two-files) and [UC-03](https://github.com/Veridelta/veridelta/blob/main/product/USERS.md#uc-03-sign-off-from-the-report-alone).
- Registration of `veridelta mcp` with the editor's agent mode. Serves [UC-04](https://github.com/Veridelta/veridelta/blob/main/product/USERS.md#uc-04-let-an-agent-run-the-comparison).

## AI workflows

These items build on [AI agents](https://veridelta.github.io/veridelta/agents/), in this order, tracked in [issue 128](https://github.com/Veridelta/veridelta/issues/128). A model never decides a verdict: it drives the command line, explains a run, or proposes a rule with its evidence, and only a rule the user declares changes what matches.

### Ambitious

These items may not come soon, and some may never ship. They are listed so the direction is written down. Each is opt-in, and none changes a verdict except through a rule the user declares.

- A rule that compares text by meaning, through embeddings from a provider the user chooses. Paraphrased text and model outputs match when their meaning matches. No use case yet: nobody has asked for matching by meaning, and the item stays as direction until someone does.
- A summary of a run written by a language model, with the user's own key or a local model. It sends totals only, unless the user allows row values, and is tested from recorded exchanges. The MCP server lets an agent explain a run, so this item may stay unbuilt. No use case yet: the reviewer reads the report, and nobody has asked for prose.