5. Validate and CI¶
In CI, a comparison runs on every pull request. veridelta validate checks its configuration without reading a row, so a broken file fails in seconds, before any database or warehouse is queried.
This tutorial compares a SQLite database with a Parquet export, checks the configuration with veridelta validate, writes what CI posts, and runs it all from the GitHub Action. Database sources need the database extra: uv add 'veridelta[database]'. CI integrations lists every input and output of the GitHub Action and the GitLab template.
Open this tutorial in Google Colab. Its first cell installs Veridelta there.
# A kernel without Veridelta, such as Colab's, installs the release this tutorial shows.
import importlib.util
import subprocess
import sys
if importlib.util.find_spec("veridelta") is None:
subprocess.run(
[sys.executable, "-m", "pip", "install", "--quiet", "veridelta[database]==0.35.1"],
check=True,
)
1. A legacy database and a modern export¶
SQLite stands in for the legacy system, because Python ships with it. Postgres, MySQL, SQL Server, and the other databases work the same way through a connection string. The export upper-cases status codes, one amount drifted by 50 cents, one cancelled order was dropped, and one new order arrived:
import sqlite3
from pathlib import Path
import polars as pl
Path("legacy.db").unlink(missing_ok=True)
db = sqlite3.connect("legacy.db")
db.execute("CREATE TABLE orders (order_id INTEGER PRIMARY KEY, status TEXT, amount REAL)")
db.executemany(
"INSERT INTO orders VALUES (?, ?, ?)",
[
(1001, "shipped", 25.00),
(1002, "pending", 40.50),
(1003, "shipped", 12.25),
(1004, "cancelled", 99.00),
],
)
db.commit()
db.close()
pl.DataFrame(
{
"order_id": [1001, 1002, 1003, 1005],
"status": ["SHIPPED", "PENDING", "SHIPPED", "PENDING"],
"amount": [25.00, 40.50, 12.75, 18.00],
}
).write_parquet("modern.parquet")
print("wrote legacy.db and modern.parquet")
# Output:
# wrote legacy.db and modern.parquet
2. Point the configuration at the database¶
type: database reads a table through ConnectorX into the engine that compares files. The database's path comes from an environment variable, so one reviewed file works on a laptop and in CI. The sqlite:// scheme stays in the file, which lets validate check it even where the variable is unset, as section 4 shows:
%%writefile veridelta.yaml
source:
type: database
uri: sqlite://${LEGACY_DB_PATH}
table: orders
target:
path: modern.parquet
primary_keys: ["order_id"]
rules:
- column_names: ["status"]
case_insensitive: true
# Output:
# Writing veridelta.yaml
import os
os.environ["LEGACY_DB_PATH"] = Path("legacy.db").resolve().as_posix()
print("LEGACY_DB_PATH points at legacy.db")
# Output:
# LEGACY_DB_PATH points at legacy.db
3. Validate before running¶
veridelta validate loads the file and checks every setting, pattern, and extra the run needs, without reading a row. Exit code 0 means the configuration can run, and exit code 1 lists what would stop it:
!veridelta validate -c veridelta.yaml; echo "exit=$?"
# Output:
# veridelta.yaml: valid.
# exit=0
A misspelled setting would otherwise surface only when the nightly job runs. Here it fails at once, with the path to the bad key:
Path("typo.yaml").write_text(
Path("veridelta.yaml").read_text().replace("case_insensitive", "case_insensitve")
)
!veridelta validate -c typo.yaml; echo "exit=$?"
# Output:
# error: Configuration Validation Failed:
# - [rules -> 0 -> case_insensitve]: Extra inputs are not permitted
# typo.yaml: 1 error, 0 warnings.
# exit=1
--schemas goes one step further: it reads each side's column names and types, still without reading rows. A database table gets a query that returns no rows, and a warehouse gets the column probes pushdown starts with. A key that does not exist passes the offline check, but fails here, before any comparison starts:
Path("wrong_key.yaml").write_text(
Path("veridelta.yaml").read_text().replace('["order_id"]', '["order_no"]')
)
!veridelta validate -c wrong_key.yaml; echo "exit=$?"
!veridelta validate -c wrong_key.yaml --schemas; echo "exit=$?"
# Output:
# wrong_key.yaml: valid.
# exit=0
# error: The primary key 'order_no' is not among the columns of the source, the `database` table `orders`. The columns read are: 'order_id', 'status', 'amount'.
# wrong_key.yaml: 1 error, 0 warnings.
# exit=1
4. Validate without secrets¶
The job that checks a pull request often has no credentials, and should not need them. --allow-missing-env reads each unset variable as its own name and warns, so the rest of the file is still checked:
!env -u LEGACY_DB_PATH veridelta validate -c veridelta.yaml --allow-missing-env; echo "exit=$?"
# Output:
# warning: Environment variable 'LEGACY_DB_PATH' is not set, so its references were checked as the text 'LEGACY_DB_PATH'.
# veridelta.yaml: valid, with 1 warning.
# exit=0
5. What CI reports¶
veridelta run exits 1 on drift. --markdown writes the summary that the GitHub Action and the GitLab template post on a pull request. It ends with its counts as JSON, in an HTML comment that readers never see and an agent can parse. --otel writes the run's OpenTelemetry metrics for a dashboard:
!veridelta run -c veridelta.yaml --quiet --markdown summary.md --otel otel-metrics.json; echo "exit=$?"
# Output:
#
# Veridelta Execution Summary
# ===========================
# Status: FAILED
# Match Rate: 25.0%
# Source Rows: 4
# Target Rows: 4
# Volume Shift: +0 rows
#
# Row-Level Discrepancies:
# ---------------------------
# Added: 1
# Removed: 1
# Changed: 1
# Total Issues: 3
#
# Top Column-Level Drifts:
# ---------------------------
# - amount: 1 mismatch
#
# exit=1
print(Path("summary.md").read_text())
# Output:
# ### Veridelta: FAILED
#
# | Metric | Value |
# | :--- | ---: |
# | Match rate | 25.0% |
# | Source rows | 4 |
# | Target rows | 4 |
# | Volume shift | +0 |
# | Added | 1 |
# | Removed | 1 |
# | Changed | 1 |
#
# #### Column-level drift
#
# | Column | Mismatches |
# | :--- | ---: |
# | `amount` | 1 |
#
# <!-- veridelta-summary https://veridelta.github.io/veridelta/schema/run.schema.json
# {"total_rows_source":4,"total_rows_target":4,"added_count":1,"removed_count":1,"changed_count":1,"column_mismatches":{"amount":1},"is_match":false,"accepted_count":0,"total_mismatches":3,"mismatch_ratio":0.75,"match_rate_percentage":25.0,"is_perfect_match":false,"volume_shift":0,"report_summary":"Veridelta Execution Summary\n===========================\nStatus: FAILED\nMatch Rate: 25.0%\nSource Rows: 4\nTarget Rows: 4\nVolume Shift: +0 rows\n\nRow-Level Discrepancies:\n---------------------------\nAdded: 1\nRemoved: 1\nChanged: 1\nTotal Issues: 3\n\nTop Column-Level Drifts:\n---------------------------\n- amount: 1 mismatch\n"}
# -->
The metrics file is one line of OTLP JSON, for an OpenTelemetry Collector or any OTLP/HTTP endpoint. Each metric is a gauge with one point per run, labeled by side, by kind of difference, or by column; see OpenTelemetry metrics. The cell below reads it back:
import json
(resource,) = json.loads(Path("otel-metrics.json").read_text())["resourceMetrics"]
attributes = {
item["key"]: item["value"]["stringValue"] for item in resource["resource"]["attributes"]
}
print(attributes["veridelta.source.type"], attributes["veridelta.source.name"])
for metric in resource["scopeMetrics"][0]["metrics"]:
for point in metric["gauge"]["dataPoints"]:
labels = [item["value"]["stringValue"] for item in point.get("attributes", [])]
label = "{" + ", ".join(labels) + "}" if labels else ""
print(f"{metric['name']}{label} = {point.get('asInt', point.get('asDouble'))}")
# Output:
# database orders
# veridelta.dataset.rows{source} = 4
# veridelta.dataset.rows{target} = 4
# veridelta.diff.rows{added} = 1
# veridelta.diff.rows{removed} = 1
# veridelta.diff.rows{changed} = 1
# veridelta.column.mismatched_rows{status} = 0
# veridelta.column.mismatched_rows{amount} = 1
# veridelta.diff.mismatch_ratio = 0.75
# veridelta.diff.match = 0
6. Run it on every pull request¶
In production, the source is a server rather than a file, and its password stays out of the configuration:
source:
type: database
uri: postgresql://analyst@legacy-db.internal:5432/sales
password: ${LEGACY_DB_PASSWORD}
table: orders
Save this workflow as .github/workflows/data-parity.yml. The validate job needs no secrets, so it runs on pull requests from forks too, and the comparison starts only once it passes:
name: Data parity
on: pull_request
permissions:
contents: read
pull-requests: write # for the summary comment
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: astral-sh/setup-uv@v10
- run: uvx --from 'veridelta[database]==0.35.1' veridelta validate -c veridelta.yaml --allow-missing-env
compare:
needs: validate
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: Veridelta/veridelta@v0.35.1
with:
config: veridelta.yaml
extras: database
env:
LEGACY_DB_PASSWORD: ${{ secrets.LEGACY_DB_PASSWORD }}
The action installs the release it is pinned to and runs the comparison. It keeps one summary comment per configuration on the pull request, and uploads summary.json, summary.md, report.html, and otel-metrics.json as an artifact. It fails the job on drift, unless fail-on-mismatch is false.
On GitLab, include the template from the same release:
include:
- remote: https://raw.githubusercontent.com/Veridelta/veridelta/v0.35.1/ci/gitlab/veridelta.yml
inputs:
config: veridelta.yaml
extras: database
CI integrations lists every input and output. Checking a configuration lists what validate checks, and Databases covers each database's connection string. The model evaluation tutorial compares two runs of a model's evaluation suite and fails on the regressions. The cell below removes the files this tutorial wrote:
for name in (
"legacy.db",
"modern.parquet",
"veridelta.yaml",
"typo.yaml",
"wrong_key.yaml",
"summary.md",
"otel-metrics.json",
):
Path(name).unlink(missing_ok=True)
os.environ.pop("LEGACY_DB_PATH", None)
print("cleaned")
# Output:
# cleaned