Diff
diff
¶
Comparison of two dataset snapshots.
Takes two snapshot payloads -- the plain dicts SnapshotBuilder produces and write_snapshot stores -- and reports what changed between them: schema, dataset-level metrics, and per-column statistics, each classified by how much it should worry the reader.
Neither input has to come from a store. Missing fields degrade to "not
recorded", and a payload with no recorded format version is compared as far as
its fields allow. An unsupported recorded format version raises
IncompatibleSnapshotError.
Layers¶
- diff_snapshots is the whole public entry point.
- SnapshotDiff.to_dict renders that result as plain JSON-serializable data, for a pipeline to consume.
- DiffOptions carries the thresholds that decide unchanged from changed from significant; preset returns the named sensitivities the CLI offers.
Examples:
>>> from dstrack.diff import diff_snapshots
>>> old = {
... "dataset_name": "customers",
... "num_rows": 100,
... "columns": [{"name": "age", "dtype": "int64", "nullable": True}],
... "column_stats": {
... "age": {"null_count": 0, "null_fraction": 0.0, "mean": 30.0, "std": 5.0}
... },
... }
>>> new = {
... "dataset_name": "customers",
... "num_rows": 150,
... "columns": [{"name": "age", "dtype": "float64", "nullable": True}],
... "column_stats": {
... "age": {"null_count": 15, "null_fraction": 0.1, "mean": 42.0, "std": 5.0}
... },
... }
>>> result = diff_snapshots(old, new)
>>> result.severity.value
'critical'
>>> [change.name for change in result.schema.type_changed]
['age']
>>> result.summary.row_delta
50
>>> sorted(result.to_dict())[:3]
['columns', 'diff_format_version', 'identical']
Comparing two snapshots already in a store means reading their payloads first, with read_snapshot.
Classes:
| Name | Description |
|---|---|
ColumnChangeKind |
What happened to a column between the two schemas. |
ColumnKind |
Which statistics variant a column's |
ColumnSchemaChange |
One column's schema entry, compared between snapshots. |
ColumnStatDiff |
One column's statistics, compared between snapshots. |
DiffOptions |
Thresholds deciding unchanged vs. changed vs. significant. |
DiffSummary |
Headline counts, for a one-line render and a report's summary block. |
DistributionShift |
How far a column's value distribution moved. |
MetricDiff |
One scalar quantity as it stood on each side. |
SchemaDiff |
Every difference between the two snapshots' column lists. |
SetChange |
A set-valued dataset field, e.g. |
Severity |
How much a single comparison should worry the reader. |
ShiftMethod |
How a column's distribution shift was measured. |
SnapshotDiff |
Everything that changed between two snapshots. |
SnapshotRef |
Identity of one side of the comparison, as its payload recorded it. |
Functions:
| Name | Description |
|---|---|
diff_snapshots |
Compare two snapshot payloads. |
preset |
Return a named sensitivity preset. |
worst |
Return the highest severity in |
ColumnChangeKind
¶
Bases: StrEnum
flowchart TD
dstrack.diff.ColumnChangeKind[ColumnChangeKind]
click dstrack.diff.ColumnChangeKind href "" "dstrack.diff.ColumnChangeKind"
What happened to a column between the two schemas.
ColumnKind
¶
Bases: StrEnum
flowchart TD
dstrack.diff.ColumnKind[ColumnKind]
click dstrack.diff.ColumnKind href "" "dstrack.diff.ColumnKind"
Which statistics variant a column's column_stats entry holds.
ColumnSchemaChange(name, kind, old_dtype, new_dtype, old_nullable, new_nullable, severity)
dataclass
¶
One column's schema entry, compared between snapshots.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Column name. |
kind |
ColumnChangeKind
|
How the column changed (added, removed, retyped, ...). |
old_dtype |
str | None
|
Data type on the old side, or |
new_dtype |
str | None
|
Data type on the new side, or |
old_nullable |
bool | None
|
Nullability on the old side, or |
new_nullable |
bool | None
|
Nullability on the new side, or |
severity |
Severity
|
How the change classifies under the active options. |
ColumnStatDiff(name, kind, old_kind, present_in_old, present_in_new, metrics, distribution_shift, severity)
dataclass
¶
One column's statistics, compared between snapshots.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Column name. |
kind |
ColumnKind
|
The statistics variant on the new side, or on the old side when the column was removed. |
old_kind |
ColumnKind | None
|
The variant on the old side, when it differs from |
present_in_old |
bool
|
Whether the column has statistics on the old side. |
present_in_new |
bool
|
Whether the column has statistics on the new side. |
metrics |
tuple[MetricDiff, ...]
|
Every scalar at least one side recorded, in a stable order. |
distribution_shift |
DistributionShift | None
|
|
severity |
Severity
|
The worst severity among the column's metrics and shift. |
DiffOptions(rel_tolerance=DEFAULT_REL_TOLERANCE, abs_tolerance=DEFAULT_ABS_TOLERANCE, warn_pct_change=DEFAULT_WARN_PCT_CHANGE, critical_pct_change=DEFAULT_CRITICAL_PCT_CHANGE, warn_fraction_shift=DEFAULT_WARN_FRACTION_SHIFT, critical_fraction_shift=DEFAULT_CRITICAL_FRACTION_SHIFT, warn_psi=DEFAULT_WARN_PSI, critical_psi=DEFAULT_CRITICAL_PSI, warn_tvd=DEFAULT_WARN_TVD, critical_tvd=DEFAULT_CRITICAL_TVD, warn_percentile_shift=DEFAULT_WARN_PERCENTILE_SHIFT, critical_percentile_shift=DEFAULT_CRITICAL_PERCENTILE_SHIFT)
dataclass
¶
Thresholds deciding unchanged vs. changed vs. significant.
Every threshold is a fraction, never a percentage.
Relative thresholds apply to counts and unbounded magnitudes. Absolute
shift thresholds apply to quantities that are already fractions in
[0, 1], where relative change is meaningless near zero: a null fraction
going from 0.001 to 0.002 is a 100% relative change and nothing at all.
Attributes:
| Name | Type | Description |
|---|---|---|
rel_tolerance |
float
|
Relative difference below which two numbers count as
equal. Not applied to a datetime column's |
abs_tolerance |
float
|
Absolute difference below which two numbers count as equal; covers comparisons against zero, where a relative tolerance is meaningless. |
warn_pct_change |
float
|
Relative change in a count or magnitude at which it is flagged as a warning. |
critical_pct_change |
float
|
Relative change in a count or magnitude at which it is flagged as critical. |
warn_fraction_shift |
float
|
Absolute shift in a |
critical_fraction_shift |
float
|
Absolute shift in a |
warn_psi |
float
|
Population stability index (PSI) at which a numeric distribution shift is a warning. |
critical_psi |
float
|
Population stability index at which a numeric distribution shift is critical. |
warn_tvd |
float
|
Total variation distance at which a categorical distribution shift is a warning. |
critical_tvd |
float
|
Total variation distance at which a categorical distribution shift is critical. |
warn_percentile_shift |
float
|
Largest percentile move, as a fraction of the baseline's p5-p95 spread, at which a shift is a warning. Used when histograms are unusable for PSI. |
critical_percentile_shift |
float
|
Same measure, at which a shift is critical. |
DiffSummary(columns_added, columns_removed, columns_type_changed, columns_drifted, row_delta, severity)
dataclass
¶
Headline counts, for a one-line render and a report's summary block.
Attributes:
| Name | Type | Description |
|---|---|---|
columns_added |
int
|
Number of columns present only on the new side. |
columns_removed |
int
|
Number of columns present only on the old side. |
columns_type_changed |
int
|
Number of columns whose data type changed. |
columns_drifted |
int
|
Columns whose own severity reached |
row_delta |
int | None
|
Change in row count (new minus old), or |
severity |
Severity
|
The overall severity of the comparison. |
DistributionShift(method, score, severity, added_categories=(), removed_categories=(), detail=None)
dataclass
¶
How far a column's value distribution moved.
Attributes:
| Name | Type | Description |
|---|---|---|
method |
ShiftMethod
|
Which measure |
score |
float | None
|
The measure's value, or |
severity |
Severity
|
How the shift classifies under the active options. |
added_categories |
tuple[str, ...]
|
Values the new top-values list carries that the old
one did not. User-derived: renderers must build these with
|
removed_categories |
tuple[str, ...]
|
The mirror image. |
detail |
str | None
|
A short note on how the score was arrived at, e.g. why a fallback measure was used. |
MetricDiff(key, old, new, delta, pct_change, severity, old_display=None, new_display=None)
dataclass
¶
One scalar quantity as it stood on each side.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
str
|
Stable machine name, e.g. |
old |
float | None
|
Value on the left side, or |
new |
float | None
|
Value on the right side, or |
delta |
float | None
|
|
pct_change |
float | None
|
|
severity |
Severity
|
How the change classifies under the active options. |
old_display |
str | None
|
How to show |
new_display |
str | None
|
The same for |
SchemaDiff(added=(), removed=(), type_changed=(), nullability_changed=(), reordered=(), schema_hash_equal=True, severity=Severity.UNCHANGED)
dataclass
¶
Every difference between the two snapshots' column lists.
Attributes:
| Name | Type | Description |
|---|---|---|
added |
tuple[ColumnSchemaChange, ...]
|
Columns present only on the new side. |
removed |
tuple[ColumnSchemaChange, ...]
|
Columns present only on the old side. |
type_changed |
tuple[ColumnSchemaChange, ...]
|
Columns on both sides whose data type differs. |
nullability_changed |
tuple[ColumnSchemaChange, ...]
|
Columns on both sides whose nullability differs. |
reordered |
tuple[str, ...]
|
Names present on both sides at a different position.
|
schema_hash_equal |
bool
|
Whether both sides recorded the same
|
severity |
Severity
|
The worst severity among the schema changes. |
changed
property
¶
Whether any column was added, removed, or redefined.
SetChange(key, added, removed, severity)
dataclass
¶
A set-valued dataset field, e.g. constant_columns.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
str
|
Name of the field that changed. |
added |
tuple[str, ...]
|
Members present only on the new side. |
removed |
tuple[str, ...]
|
Members present only on the old side. |
severity |
Severity
|
How the change classifies under the active options. |
Severity
¶
Bases: StrEnum
flowchart TD
dstrack.diff.Severity[Severity]
click dstrack.diff.Severity href "" "dstrack.diff.Severity"
How much a single comparison should worry the reader.
ShiftMethod
¶
Bases: StrEnum
flowchart TD
dstrack.diff.ShiftMethod[ShiftMethod]
click dstrack.diff.ShiftMethod href "" "dstrack.diff.ShiftMethod"
How a column's distribution shift was measured.
Each measure has its own scale and its own thresholds, so a score is only meaningful alongside the method that produced it.
SnapshotDiff(old, new, identical, same_dataset, source_hash_equal, schema, metrics, sets, columns, summary, severity, options=DEFAULT_OPTIONS)
dataclass
¶
Everything that changed between two snapshots.
Attributes:
| Name | Type | Description |
|---|---|---|
identical |
bool
|
Whether every comparison came back |
same_dataset |
bool
|
Whether both sides belong to the same dataset. |
source_hash_equal |
bool
|
Whether both sides recorded the same non-empty source hash, i.e. the bytes behind them are known to be identical. |
old |
SnapshotRef
|
The baseline snapshot. |
new |
SnapshotRef
|
The snapshot compared against the baseline. |
schema |
SchemaDiff
|
Column additions, removals and changes between the schemas. |
metrics |
tuple[MetricDiff, ...]
|
Dataset-level scalar quantities (e.g. row count), compared. |
sets |
tuple[SetChange, ...]
|
Changes to set-valued dataset fields, e.g. |
columns |
tuple[ColumnStatDiff, ...]
|
Per-column statistics, compared. |
summary |
DiffSummary
|
Counts of changes by kind and severity. |
severity |
Severity
|
The highest severity across every comparison. |
options |
DiffOptions
|
The thresholds the diff was classified with. |
Methods:
| Name | Description |
|---|---|
to_dict |
Return the diff as plain JSON-serializable data. |
to_dict()
¶
Return the diff as plain JSON-serializable data.
Enum members are flattened to their string values and tuples become
lists, so the result survives json.dumps with no custom encoder and
compares equal to a literal in a test.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
A dict carrying |
dict[str, Any]
|
object. Keys are stable within a |
dict[str, Any]
|
[DIFF_FORMAT_VERSION][dstrack.diff._model.DIFF_FORMAT_VERSION]: |
dict[str, Any]
|
new ones may appear, existing ones are not renamed or removed. |
SnapshotRef(snapshot_id=None, dataset_id=None, parent_snapshot_id=None, dataset_name=None, dataset_path=None, created_at=None, created_by=None, source_type=None, source_hash=None, schema_hash=None, format_version=None, num_rows=None, num_columns=None)
dataclass
¶
Identity of one side of the comparison, as its payload recorded it.
Every field is optional. diff_snapshots accepts dicts that never touched
a store, and a payload written by another version of dstrack may omit a
field entirely; neither is a reason to refuse to compare.
Attributes:
| Name | Type | Description |
|---|---|---|
snapshot_id |
str | None
|
Unique identifier of the snapshot. |
dataset_id |
str | None
|
Which dataset holds the snapshot. Never present in a payload -- it is a store concept -- so it is supplied by the caller or left unset. |
parent_snapshot_id |
str | None
|
Identifier of the snapshot this one was derived from, if any. |
dataset_name |
str | None
|
Human-readable name of the dataset. |
dataset_path |
str | None
|
Path of the tracked data, relative to the root it was tracked against. |
created_at |
str | None
|
Timestamp at which the snapshot was created. |
created_by |
str | None
|
Who or what created the snapshot. |
source_type |
str | None
|
Kind of source the snapshot was taken from (e.g. |
source_hash |
str | None
|
Content hash of the source data. |
schema_hash |
str | None
|
Hash of the dataset's schema. |
format_version |
str | None
|
Version of the snapshot payload format. |
num_rows |
int | None
|
Number of rows in the dataset. |
num_columns |
int | None
|
Number of columns in the dataset. |
diff_snapshots(old, new, *, options=None, old_dataset_id=None, new_dataset_id=None)
¶
Compare two snapshot payloads.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
old
|
Mapping[str, Any]
|
The earlier snapshot's payload, i.e. the baseline. |
required |
new
|
Mapping[str, Any]
|
The later snapshot's payload. |
required |
options
|
DiffOptions | None
|
Thresholds deciding what counts as a significant change. Defaults to [DEFAULT_OPTIONS][dstrack.diff._model.DEFAULT_OPTIONS]. |
None
|
old_dataset_id
|
str | None
|
Dataset the baseline belongs to, if known. Never present in a payload, so it can only come from whoever read it. |
None
|
new_dataset_id
|
str | None
|
Dataset the later snapshot belongs to, if known. |
None
|
Returns:
| Type | Description |
|---|---|
SnapshotDiff
|
Everything that differs between them, each difference classified by |
SnapshotDiff
|
severity. |
Raises:
| Type | Description |
|---|---|
IncompatibleSnapshotError
|
If either payload records a snapshot format version this dstrack cannot read. Any supported version is accepted, including two different ones, and a payload that records none is compared as far as its fields allow. |
preset(name)
¶
Return a named sensitivity preset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
One of the keys of [PRESETS][dstrack.diff._model.PRESETS]. |
required |
Returns:
| Type | Description |
|---|---|
DiffOptions
|
The thresholds that preset stands for. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If no preset goes by that name. |
worst(severities)
¶
Return the highest severity in severities, or UNCHANGED if empty.
How a whole diff is classified: a group is exactly as serious as its most serious member, applied at every level from a column's metrics up to the diff itself.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
severities
|
Iterable[Severity]
|
The severities to fold. |
required |
Returns:
| Type | Description |
|---|---|
Severity
|
The most serious of them. |