Skip to content

Diff

diff

Comparison of two dataset snapshots.

Takes two snapshot payloads -- the plain dicts SnapshotBuilder produces and write_snapshot stores -- and reports what changed between them: schema, dataset-level metrics, and per-column statistics, each classified by how much it should worry the reader.

Neither input has to come from a store. Missing fields degrade to "not recorded", and a payload with no recorded format version is compared as far as its fields allow. An unsupported recorded format version raises IncompatibleSnapshotError.

Layers

  • diff_snapshots is the whole public entry point.
  • SnapshotDiff.to_dict renders that result as plain JSON-serializable data, for a pipeline to consume.
  • DiffOptions carries the thresholds that decide unchanged from changed from significant; preset returns the named sensitivities the CLI offers.

Examples:

>>> from dstrack.diff import diff_snapshots
>>> old = {
...     "dataset_name": "customers",
...     "num_rows": 100,
...     "columns": [{"name": "age", "dtype": "int64", "nullable": True}],
...     "column_stats": {
...         "age": {"null_count": 0, "null_fraction": 0.0, "mean": 30.0, "std": 5.0}
...     },
... }
>>> new = {
...     "dataset_name": "customers",
...     "num_rows": 150,
...     "columns": [{"name": "age", "dtype": "float64", "nullable": True}],
...     "column_stats": {
...         "age": {"null_count": 15, "null_fraction": 0.1, "mean": 42.0, "std": 5.0}
...     },
... }
>>> result = diff_snapshots(old, new)
>>> result.severity.value
'critical'
>>> [change.name for change in result.schema.type_changed]
['age']
>>> result.summary.row_delta
50
>>> sorted(result.to_dict())[:3]
['columns', 'diff_format_version', 'identical']

Comparing two snapshots already in a store means reading their payloads first, with read_snapshot.

Classes:

Name Description
ColumnChangeKind

What happened to a column between the two schemas.

ColumnKind

Which statistics variant a column's column_stats entry holds.

ColumnSchemaChange

One column's schema entry, compared between snapshots.

ColumnStatDiff

One column's statistics, compared between snapshots.

DiffOptions

Thresholds deciding unchanged vs. changed vs. significant.

DiffSummary

Headline counts, for a one-line render and a report's summary block.

DistributionShift

How far a column's value distribution moved.

MetricDiff

One scalar quantity as it stood on each side.

SchemaDiff

Every difference between the two snapshots' column lists.

SetChange

A set-valued dataset field, e.g. constant_columns.

Severity

How much a single comparison should worry the reader.

ShiftMethod

How a column's distribution shift was measured.

SnapshotDiff

Everything that changed between two snapshots.

SnapshotRef

Identity of one side of the comparison, as its payload recorded it.

Functions:

Name Description
diff_snapshots

Compare two snapshot payloads.

preset

Return a named sensitivity preset.

worst

Return the highest severity in severities, or UNCHANGED if empty.

ColumnChangeKind

Bases: StrEnum


              flowchart TD
              dstrack.diff.ColumnChangeKind[ColumnChangeKind]

              

              click dstrack.diff.ColumnChangeKind href "" "dstrack.diff.ColumnChangeKind"
            

What happened to a column between the two schemas.

ColumnKind

Bases: StrEnum


              flowchart TD
              dstrack.diff.ColumnKind[ColumnKind]

              

              click dstrack.diff.ColumnKind href "" "dstrack.diff.ColumnKind"
            

Which statistics variant a column's column_stats entry holds.

ColumnSchemaChange(name, kind, old_dtype, new_dtype, old_nullable, new_nullable, severity) dataclass

One column's schema entry, compared between snapshots.

Attributes:

Name Type Description
name str

Column name.

kind ColumnChangeKind

How the column changed (added, removed, retyped, ...).

old_dtype str | None

Data type on the old side, or None if absent or not recorded.

new_dtype str | None

Data type on the new side, or None if absent or not recorded.

old_nullable bool | None

Nullability on the old side, or None if absent or not recorded.

new_nullable bool | None

Nullability on the new side, or None if absent or not recorded.

severity Severity

How the change classifies under the active options.

ColumnStatDiff(name, kind, old_kind, present_in_old, present_in_new, metrics, distribution_shift, severity) dataclass

One column's statistics, compared between snapshots.

Attributes:

Name Type Description
name str

Column name.

kind ColumnKind

The statistics variant on the new side, or on the old side when the column was removed.

old_kind ColumnKind | None

The variant on the old side, when it differs from kind. A column that changed variant has no comparable statistics beyond its null counts, which is why such a change is always CRITICAL.

present_in_old bool

Whether the column has statistics on the old side.

present_in_new bool

Whether the column has statistics on the new side.

metrics tuple[MetricDiff, ...]

Every scalar at least one side recorded, in a stable order.

distribution_shift DistributionShift | None

None for kinds that carry no distribution.

severity Severity

The worst severity among the column's metrics and shift.

DiffOptions(rel_tolerance=DEFAULT_REL_TOLERANCE, abs_tolerance=DEFAULT_ABS_TOLERANCE, warn_pct_change=DEFAULT_WARN_PCT_CHANGE, critical_pct_change=DEFAULT_CRITICAL_PCT_CHANGE, warn_fraction_shift=DEFAULT_WARN_FRACTION_SHIFT, critical_fraction_shift=DEFAULT_CRITICAL_FRACTION_SHIFT, warn_psi=DEFAULT_WARN_PSI, critical_psi=DEFAULT_CRITICAL_PSI, warn_tvd=DEFAULT_WARN_TVD, critical_tvd=DEFAULT_CRITICAL_TVD, warn_percentile_shift=DEFAULT_WARN_PERCENTILE_SHIFT, critical_percentile_shift=DEFAULT_CRITICAL_PERCENTILE_SHIFT) dataclass

Thresholds deciding unchanged vs. changed vs. significant.

Every threshold is a fraction, never a percentage.

Relative thresholds apply to counts and unbounded magnitudes. Absolute shift thresholds apply to quantities that are already fractions in [0, 1], where relative change is meaningless near zero: a null fraction going from 0.001 to 0.002 is a 100% relative change and nothing at all.

Attributes:

Name Type Description
rel_tolerance float

Relative difference below which two numbers count as equal. Not applied to a datetime column's min and max: as epoch seconds, it would hide shifts of several days.

abs_tolerance float

Absolute difference below which two numbers count as equal; covers comparisons against zero, where a relative tolerance is meaningless.

warn_pct_change float

Relative change in a count or magnitude at which it is flagged as a warning.

critical_pct_change float

Relative change in a count or magnitude at which it is flagged as critical.

warn_fraction_shift float

Absolute shift in a [0, 1] fraction (e.g. a null fraction) at which it is flagged as a warning.

critical_fraction_shift float

Absolute shift in a [0, 1] fraction at which it is flagged as critical.

warn_psi float

Population stability index (PSI) at which a numeric distribution shift is a warning.

critical_psi float

Population stability index at which a numeric distribution shift is critical.

warn_tvd float

Total variation distance at which a categorical distribution shift is a warning.

critical_tvd float

Total variation distance at which a categorical distribution shift is critical.

warn_percentile_shift float

Largest percentile move, as a fraction of the baseline's p5-p95 spread, at which a shift is a warning. Used when histograms are unusable for PSI.

critical_percentile_shift float

Same measure, at which a shift is critical.

DiffSummary(columns_added, columns_removed, columns_type_changed, columns_drifted, row_delta, severity) dataclass

Headline counts, for a one-line render and a report's summary block.

Attributes:

Name Type Description
columns_added int

Number of columns present only on the new side.

columns_removed int

Number of columns present only on the old side.

columns_type_changed int

Number of columns whose data type changed.

columns_drifted int

Columns whose own severity reached WARNING or above, counting statistical movement rather than schema edits.

row_delta int | None

Change in row count (new minus old), or None when either side did not record one.

severity Severity

The overall severity of the comparison.

DistributionShift(method, score, severity, added_categories=(), removed_categories=(), detail=None) dataclass

How far a column's value distribution moved.

Attributes:

Name Type Description
method ShiftMethod

Which measure score is on. Always read the two together.

score float | None

The measure's value, or None when method is NONE.

severity Severity

How the shift classifies under the active options.

added_categories tuple[str, ...]

Values the new top-values list carries that the old one did not. User-derived: renderers must build these with rich.text.Text, and the PDF must escape them.

removed_categories tuple[str, ...]

The mirror image.

detail str | None

A short note on how the score was arrived at, e.g. why a fallback measure was used.

MetricDiff(key, old, new, delta, pct_change, severity, old_display=None, new_display=None) dataclass

One scalar quantity as it stood on each side.

Attributes:

Name Type Description
key str

Stable machine name, e.g. "num_rows" or "percentiles.p50".

old float | None

Value on the left side, or None if it was not recorded.

new float | None

Value on the right side, or None if it was not recorded.

delta float | None

new - old, or None when either side is missing.

pct_change float | None

delta / abs(old), or None when old is zero or either side is missing. A fraction, not a percentage.

severity Severity

How the change classifies under the active options.

old_display str | None

How to show old when the number is not what a reader wants, as for a datetime bound held in epoch seconds. None means format the number.

new_display str | None

The same for new.

SchemaDiff(added=(), removed=(), type_changed=(), nullability_changed=(), reordered=(), schema_hash_equal=True, severity=Severity.UNCHANGED) dataclass

Every difference between the two snapshots' column lists.

Attributes:

Name Type Description
added tuple[ColumnSchemaChange, ...]

Columns present only on the new side.

removed tuple[ColumnSchemaChange, ...]

Columns present only on the old side.

type_changed tuple[ColumnSchemaChange, ...]

Columns on both sides whose data type differs.

nullability_changed tuple[ColumnSchemaChange, ...]

Columns on both sides whose nullability differs.

reordered tuple[str, ...]

Names present on both sides at a different position. schema_hash is computed over sorted pairs, so a pure reorder leaves the two hashes equal; this is the only field that shows it.

schema_hash_equal bool

Whether both sides recorded the same schema_hash. Display-only: the explicit column lists are what the other fields are derived from.

severity Severity

The worst severity among the schema changes.

changed property

Whether any column was added, removed, or redefined.

SetChange(key, added, removed, severity) dataclass

A set-valued dataset field, e.g. constant_columns.

Attributes:

Name Type Description
key str

Name of the field that changed.

added tuple[str, ...]

Members present only on the new side.

removed tuple[str, ...]

Members present only on the old side.

severity Severity

How the change classifies under the active options.

Severity

Bases: StrEnum


              flowchart TD
              dstrack.diff.Severity[Severity]

              

              click dstrack.diff.Severity href "" "dstrack.diff.Severity"
            

How much a single comparison should worry the reader.

ShiftMethod

Bases: StrEnum


              flowchart TD
              dstrack.diff.ShiftMethod[ShiftMethod]

              

              click dstrack.diff.ShiftMethod href "" "dstrack.diff.ShiftMethod"
            

How a column's distribution shift was measured.

Each measure has its own scale and its own thresholds, so a score is only meaningful alongside the method that produced it.

SnapshotDiff(old, new, identical, same_dataset, source_hash_equal, schema, metrics, sets, columns, summary, severity, options=DEFAULT_OPTIONS) dataclass

Everything that changed between two snapshots.

Attributes:

Name Type Description
identical bool

Whether every comparison came back UNCHANGED and no column was added, removed, or redefined. Not the same as source_hash_equal: two snapshots of one unchanged file still differ in created_at and snapshot_id, which are identity rather than content and are never compared.

same_dataset bool

Whether both sides belong to the same dataset. False is legitimate -- comparing two datasets is a supported use -- and is surfaced rather than raised on.

source_hash_equal bool

Whether both sides recorded the same non-empty source hash, i.e. the bytes behind them are known to be identical.

old SnapshotRef

The baseline snapshot.

new SnapshotRef

The snapshot compared against the baseline.

schema SchemaDiff

Column additions, removals and changes between the schemas.

metrics tuple[MetricDiff, ...]

Dataset-level scalar quantities (e.g. row count), compared.

sets tuple[SetChange, ...]

Changes to set-valued dataset fields, e.g. constant_columns.

columns tuple[ColumnStatDiff, ...]

Per-column statistics, compared.

summary DiffSummary

Counts of changes by kind and severity.

severity Severity

The highest severity across every comparison.

options DiffOptions

The thresholds the diff was classified with.

Methods:

Name Description
to_dict

Return the diff as plain JSON-serializable data.

to_dict()

Return the diff as plain JSON-serializable data.

Enum members are flattened to their string values and tuples become lists, so the result survives json.dumps with no custom encoder and compares equal to a literal in a test.

Returns:

Type Description
dict[str, Any]

A dict carrying diff_format_version plus every field of this

dict[str, Any]

object. Keys are stable within a

dict[str, Any]

[DIFF_FORMAT_VERSION][dstrack.diff._model.DIFF_FORMAT_VERSION]:

dict[str, Any]

new ones may appear, existing ones are not renamed or removed.

SnapshotRef(snapshot_id=None, dataset_id=None, parent_snapshot_id=None, dataset_name=None, dataset_path=None, created_at=None, created_by=None, source_type=None, source_hash=None, schema_hash=None, format_version=None, num_rows=None, num_columns=None) dataclass

Identity of one side of the comparison, as its payload recorded it.

Every field is optional. diff_snapshots accepts dicts that never touched a store, and a payload written by another version of dstrack may omit a field entirely; neither is a reason to refuse to compare.

Attributes:

Name Type Description
snapshot_id str | None

Unique identifier of the snapshot.

dataset_id str | None

Which dataset holds the snapshot. Never present in a payload -- it is a store concept -- so it is supplied by the caller or left unset.

parent_snapshot_id str | None

Identifier of the snapshot this one was derived from, if any.

dataset_name str | None

Human-readable name of the dataset.

dataset_path str | None

Path of the tracked data, relative to the root it was tracked against.

created_at str | None

Timestamp at which the snapshot was created.

created_by str | None

Who or what created the snapshot.

source_type str | None

Kind of source the snapshot was taken from (e.g. file).

source_hash str | None

Content hash of the source data.

schema_hash str | None

Hash of the dataset's schema.

format_version str | None

Version of the snapshot payload format.

num_rows int | None

Number of rows in the dataset.

num_columns int | None

Number of columns in the dataset.

diff_snapshots(old, new, *, options=None, old_dataset_id=None, new_dataset_id=None)

Compare two snapshot payloads.

Parameters:

Name Type Description Default
old Mapping[str, Any]

The earlier snapshot's payload, i.e. the baseline.

required
new Mapping[str, Any]

The later snapshot's payload.

required
options DiffOptions | None

Thresholds deciding what counts as a significant change. Defaults to [DEFAULT_OPTIONS][dstrack.diff._model.DEFAULT_OPTIONS].

None
old_dataset_id str | None

Dataset the baseline belongs to, if known. Never present in a payload, so it can only come from whoever read it.

None
new_dataset_id str | None

Dataset the later snapshot belongs to, if known.

None

Returns:

Type Description
SnapshotDiff

Everything that differs between them, each difference classified by

SnapshotDiff

severity.

Raises:

Type Description
IncompatibleSnapshotError

If either payload records a snapshot format version this dstrack cannot read. Any supported version is accepted, including two different ones, and a payload that records none is compared as far as its fields allow.

preset(name)

Return a named sensitivity preset.

Parameters:

Name Type Description Default
name str

One of the keys of [PRESETS][dstrack.diff._model.PRESETS].

required

Returns:

Type Description
DiffOptions

The thresholds that preset stands for.

Raises:

Type Description
KeyError

If no preset goes by that name.

worst(severities)

Return the highest severity in severities, or UNCHANGED if empty.

How a whole diff is classified: a group is exactly as serious as its most serious member, applied at every level from a column's metrics up to the diff itself.

Parameters:

Name Type Description Default
severities Iterable[Severity]

The severities to fold.

required

Returns:

Type Description
Severity

The most serious of them.