Skip to content

Snapshot

snapshot

Snapshot computation for dstrack.

Classes

SnapshotBuilder Builds a complete snapshot from a single reader, combining the two below.

MetadataBuilder Builds identity and structural snapshot fields from a reader's schema.

StatsComputer Computes per-column and dataset-level statistics in a single data pass.

Result types

SnapshotMetadata, DatasetStats

Classes:

Name Description
DatasetStats

Per-column and dataset-level statistics produced by a data pass.
DatetimeColumnStats

Statistics for a datetime64 column; min/max are ISO strings.
HistogramStats

Equal-width histogram of a numeric column's values.
MetadataBuilder

Builds identity and structural metadata without reading data rows.
NumericColumnStats

Statistics for an int*/float*/bool column.
OtherColumnStats

Null-only statistics for a column whose dtype is unrecognized.
PercentileStats

p5-p99 percentiles of a numeric column, linearly interpolated.
SnapshotBuilder

Builds a complete dataset snapshot from a single reader.
SnapshotMetadata

Identity and structural fields for a snapshot.
StatsComputer

Computes per-column and dataset-level statistics in a single data pass.
StringColumnStats

Statistics for a string column.

Functions:

Name Description
build_snapshot_dict

Merge metadata and statistics into one JSON-serializable snapshot dict.

DatasetStats(num_rows, column_stats, duplicate_row_fraction, constant_columns, high_null_columns) dataclass

Per-column and dataset-level statistics produced by a data pass.

Covers num_rows, column_stats, duplicate_row_fraction, constant_columns, and high_null_columns. Sketch-based fields (near_duplicate_estimate, row_minhash, row_hyperloglog) are handled by a separate builder.

DatetimeColumnStats(null_count=0, null_fraction=0.0, min='', max='', range_days=0.0) dataclass

Statistics for a datetime64 column; min/max are ISO strings.

HistogramStats(bin_edges=list(), counts=list()) dataclass

Equal-width histogram of a numeric column's values.

MetadataBuilder

Builds identity and structural metadata without reading data rows.

Computes snapshot_id, created_at, created_by, dataset_name, dataset_path, source_type, source_hash, num_columns, columns, and schema_hash from the reader's column descriptors alone.

Note

schema_hash is order-independent: reordering a dataset's columns without changing their names or dtypes produces the same hash.

Methods:

Name Description
build

Build metadata for a snapshot.

build(reader, *, dataset_name, dataset_path, source_type, created_by, source=None, source_hash=None)

Build metadata for a snapshot.

dataset_path is what gets recorded; source is what gets read. They differ whenever the recorded path is relative to a path root that is not the current working directory, as it is for snapshots written by the CLI.

Parameters:

Name Type Description Default
reader TabularReader

Any TabularReader; only columns() is called.

 required
dataset_name str

Human-readable dataset name stored in the snapshot.

 required
dataset_path str | PurePath

Source path or URI at snapshot time, recorded in the snapshot as a forward-slash string. Never opened.

 required
source_type str

Origin kind ("file", "directory", etc.).

 required
created_by str

User or process identifier.

 required
source str | Path | None

Location the data actually lives at, used only to compute source_hash. Defaults to dataset_path, which is correct when the recorded path is one the process can open.

 None
source_hash str | None

Pre-computed source hash. When None and the source resolves to a regular file, a SHA-256 of the file bytes is computed automatically.

 None

Returns:

Type Description
SnapshotMetadata

A populated SnapshotMetadata instance.

NumericColumnStats(null_count=0, null_fraction=0.0, mean=0.0, std=0.0, min=0.0, max=0.0, percentiles=PercentileStats(), histogram=HistogramStats(), num_unique=0) dataclass

Statistics for an int*/float*/bool column.

OtherColumnStats(null_count=0, null_fraction=0.0) dataclass

Null-only statistics for a column whose dtype is unrecognized.

PercentileStats(p5=0.0, p25=0.0, p50=0.0, p75=0.0, p95=0.0, p99=0.0) dataclass

p5-p99 percentiles of a numeric column, linearly interpolated.

SnapshotBuilder(*, metadata_builder=None, stats_computer=None)

Builds a complete dataset snapshot from a single reader.

Combines MetadataBuilder (identity and schema) and StatsComputer (a data pass over the rows) into one JSON-ready snapshot dict. Import it to build snapshots from Python without going through the CLI:

Examples:

>>> from dstrack.readers import CsvReader
>>> from dstrack.snapshot import SnapshotBuilder
>>> reader = CsvReader("data.csv")
>>> snapshot = SnapshotBuilder().build(
...     reader,
...     dataset_name="customers",
...     dataset_path="data.csv",
...     created_by="alice",
... )

Parameters:

Name Type Description Default
metadata_builder MetadataBuilder | None

Builder used for identity and schema fields. Defaults to a fresh MetadataBuilder.

 None
stats_computer StatsComputer | None

Computer used for the per-column and dataset-level statistics. Defaults to a fresh StatsComputer. Pass a pre-configured StatsComputer(max_rows=...) here to change the in-memory row limit the statistics pass enforces.

 None

Methods:

Name Description
build

Build a JSON-serializable snapshot from a reader.

build(reader, *, dataset_name, dataset_path, created_by, source_type='file', source=None, source_hash=None)

Build a JSON-serializable snapshot from a reader.

The reader is consumed once for schema inference and once for the statistics data pass.

Parameters:

Name Type Description Default
reader TabularReader

Any TabularReader.

 required
dataset_name str

Human-readable dataset name stored in the snapshot.

 required
dataset_path str | PurePath

Source path or URI recorded in the snapshot as a forward-slash string. Never opened; pass source when the data lives elsewhere.

 required
created_by str

User or process identifier.

 required
source_type str

Origin kind ("file", "directory", etc.).

 'file'
source str | Path | None

Location the data actually lives at, used only to compute source_hash. Defaults to dataset_path.

 None
source_hash str | None

Pre-computed source hash. When None and the source resolves to a regular file, a SHA-256 of the file bytes is computed automatically by the metadata builder.

 None

Returns:

Type Description
dict[str, Any]

A dict merging every metadata and statistics field, ready to be
dict[str, Any]

serialized with json.dumps.

SnapshotMetadata(format_version, snapshot_id, created_at, created_by, dataset_name, dataset_path, source_type, source_hash, num_columns, columns, schema_hash) dataclass

Identity and structural fields for a snapshot.

Covers every required top-level field that does not require a data pass: versioning identifiers, authorship, schema shape, and the source hash.

StatsComputer(*, max_rows=_DEFAULT_MAX_ROWS)

Computes per-column and dataset-level statistics in a single data pass.

Handles int*, float*, and bool dtypes as numeric, string columns, and datetime64 columns. Unknown dtypes produce only null counts.

The pass is exact but in-memory: per-column values, the distinct-row set, and per-string value counts are all retained until it completes, so worst-case memory grows with the row count. max_rows bounds that growth by rejecting datasets larger than the limit rather than risking memory exhaustion.

Parameters:

Name Type Description Default
max_rows int | None

Maximum number of rows the pass will accept. When the reader yields more than this, an [InputTooLargeError][dstrack.errors.InputTooLargeError] is raised. Defaults to [_DEFAULT_MAX_ROWS][dstrack.snapshot._stats._DEFAULT_MAX_ROWS]. Pass None to disable the limit when enough memory is available.

 _DEFAULT_MAX_ROWS

Methods:

Name Description
compute

Run a full data pass and return aggregated statistics.

compute(reader)

Run a full data pass and return aggregated statistics.

Parameters:

Name Type Description Default
reader TabularReader

Any TabularReader whose batches will be consumed once.

 required

Returns:

Type Description
DatasetStats

A populated DatasetStats instance.

Raises:

Type Description
InputTooLargeError

If the reader yields more than max_rows rows.

StringColumnStats(null_count=0, null_fraction=0.0, num_unique=0, top_values=dict(), top_values_coverage=0.0, avg_char_length=0.0, min_char_length=0.0, max_char_length=0.0, avg_token_count=0.0, min_token_count=0.0, max_token_count=0.0) dataclass

Statistics for a string column.

build_snapshot_dict(metadata, stats)

Merge metadata and statistics into one JSON-serializable snapshot dict.

The two inputs contribute disjoint sets of keys, so the merge never drops a field.

Parameters:

Name Type Description Default
metadata SnapshotMetadata

Identity and schema fields from MetadataBuilder.

 required
stats DatasetStats

Volume, per-column, and quality fields from StatsComputer.

 required

Returns:

Type Description
dict[str, Any]

A dict combining both, with nested dataclasses expanded to dicts.