Skip to content

Store

store

Persistence of snapshots into the local store, and reads back out of it.

A snapshot is written under datasets/<dataset_id>/ as its full JSON payload, a one-line append to log.jsonl, and an updated HEAD. The three are written in that order so a crash never leaves HEAD pointing at a snapshot that was not fully written. Re-tracking a source whose recorded path matches an existing dataset's latest snapshot continues that dataset's lineage rather than creating a new one.

The three writes are not a single atomic transaction, so two safeguards keep readers consistent. A store-wide advisory lock serializes the whole resolve-parent-then-write sequence, so concurrent writers cannot read the same HEAD as their parent nor mint two datasets for one path. And HEAD -- written last -- is the single source of truth for what is committed: recovery and path matching read the log entry that HEAD names rather than trusting the final line, so a log.jsonl left one entry ahead by a crash between the append and the HEAD write is ignored.

Reading history back is therefore reachability-based, not positional. read_log_entries returns every line a dataset's log holds, including entries no HEAD ever named; walk_lineage then follows parent_snapshot_id links back from HEAD to select the ones that are actually committed. Neither "read every line" nor "read every line up to HEAD" is correct: a crash between the append and the HEAD write leaves an uncommitted entry behind, and the next successful write appends after it, stranding it in the middle of the file rather than at the end. Only reachability from HEAD tells the two apart.

Classes:

Name Description
DatasetSummary

A dataset in the store, described by its latest committed snapshot.
LogEntry

One snapshot as recorded in a dataset's log.jsonl.
SnapshotWriteResult

Outcome of writing one snapshot into the store.

Functions:

Name Description
dataset_exists

Return whether the store holds a dataset with this id.
dataset_log_path

Return the path of a dataset's log.jsonl. Need not exist.
list_dataset_ids

Return the id of every dataset in the store, sorted.
read_head

Return the id of the snapshot a dataset's HEAD names.
read_log_entries

Return every entry in a dataset's log.jsonl, in file order.
walk_lineage

Select the snapshots reachable from head, newest first.
write_snapshot

Write snapshot into the store and update the dataset's history.

DatasetSummary(dataset_id, head) dataclass

A dataset in the store, described by its latest committed snapshot.

Attributes:

Name Type Description
dataset_id str

The dataset's directory name under datasets/.
head LogEntry | None

The log entry the dataset's HEAD names, or None if the dataset has no committed snapshot yet.

LogEntry(snapshot_id, parent_snapshot_id, created_at, created_by, dataset_name, dataset_path, num_rows, num_columns) dataclass

One snapshot as recorded in a dataset's log.jsonl.

The lightweight per-snapshot record, mirroring log fields rather than the full snapshots/<snapshot_id>.json payload.

Every field except snapshot_id is optional. Log lines are written with payload.get(key) (see write_snapshot), so a snapshot built without a field records it as JSON null, and a line written by a different version of dstrack may omit it entirely.

Attributes:

Name Type Description
snapshot_id str

Identifier of the snapshot. Always present: an entry is only ever reached via HEAD or a parent_snapshot_id link, both of which name it.
parent_snapshot_id str | None

Snapshot this one succeeds, or None for the dataset's first snapshot.
created_at str | None

ISO-8601 UTC timestamp, kept as the raw recorded string. Deliberately not parsed here: an unparsable value is a display concern, not a reason to fail a read.
created_by str | None

Recorded author, or None if not recorded.
dataset_name str | None

Human-readable name at snapshot time, or None.
dataset_path str | None

POSIX path relative to the path root in force at snapshot time, or None.
num_rows int | None

Row count, or None if not recorded.
num_columns int | None

Column count, or None if not recorded.

SnapshotWriteResult(dataset_id, snapshot_id, snapshot_path, parent_snapshot_id, is_new_dataset) dataclass

Outcome of writing one snapshot into the store.

Attributes:

Name Type Description
dataset_id str

Dataset whose lineage the snapshot joined, whether it was passed in, matched by path, or minted by this write.
snapshot_id str

Identifier of the snapshot that was written, copied from the snapshot payload.
snapshot_path Path

Path of the written snapshots/<snapshot_id>.json.
parent_snapshot_id str | None

Snapshot this one succeeds, i.e. the dataset's HEAD before this write, or None for a dataset's first snapshot.
is_new_dataset bool

Whether this write minted a new dataset_id, as opposed to continuing an existing dataset's lineage.

dataset_exists(dataset_id, *, store_root)

Return whether the store holds a dataset with this id.

Parameters:

Name Type Description Default
dataset_id str

The dataset id, which may come from the user.

 required
store_root Path

Path to the .dstrack/ directory.

 required

Returns:

Type Description
bool

True if the dataset has a directory in the store, whether or not
bool

it has been snapshotted yet.

Raises:

Type Description
ValueError

If dataset_id resolves to a path outside the store.

dataset_log_path(dataset_id, *, store_root)

Return the path of a dataset's log.jsonl. Need not exist.

Parameters:

Name Type Description Default
dataset_id str

Dataset whose log to locate.

 required
store_root Path

Path to the .dstrack/ directory.

 required

Returns:

Type Description
Path

The path of the dataset's append-only log.

Raises:

Type Description
ValueError

If dataset_id resolves to a path outside the store.

list_dataset_ids(*, store_root)

Return the id of every dataset in the store, sorted.

Parameters:

Name Type Description Default
store_root Path

Path to the .dstrack/ directory, e.g. from resolve_store_root.

 required

Returns:

Type Description
list[str]

Each dataset's id, i.e. its directory name under datasets/. Empty
list[str]

if the store holds no datasets yet.

read_head(dataset_id, *, store_root)

Return the id of the snapshot a dataset's HEAD names.

HEAD is the single source of truth for what a dataset has committed, so callers that derive history from it should read it before reading log.jsonl: a snapshot's log line is appended before the HEAD write that names it, so any HEAD observed at a given moment is guaranteed to have its whole ancestry already on disk. Reading them the other way round can observe a HEAD naming a snapshot whose line was appended after the log was read, which looks indistinguishable from corruption.

Parameters:

Name Type Description Default
dataset_id str

Dataset whose HEAD to read.

 required
store_root Path

Path to the .dstrack/ directory.

 required

Returns:

Type Description
str | None

The latest committed snapshot's id, or None if the dataset does
str | None

not exist or has never been snapshotted.

Raises:

Type Description
ValueError

If dataset_id resolves to a path outside the store.

read_log_entries(dataset_id, *, store_root)

Return every entry in a dataset's log.jsonl, in file order.

Includes entries that no HEAD ever named, so the result is what the log says, not what the dataset has committed. Pass the result through walk_lineage to select the committed lineage.

A trailing line that is not valid JSON and is not newline-terminated is treated as an interrupted append and dropped, not reported as corruption: the log is appended to before HEAD moves, so a torn final line is a snapshot that was never committed and is expected after a crash. A malformed line anywhere else was written completely and is corruption.

Parameters:

Name Type Description Default
dataset_id str

Dataset whose log to read.

 required
store_root Path

Path to the .dstrack/ directory.

 required

Returns:

Type Description
list[LogEntry]

One entry per log line, oldest first. Empty if the dataset does not
list[LogEntry]

exist or has no log yet.

Raises:

Type Description
ValueError

If dataset_id resolves to a path outside the store.
StoreCorruptionError

If a complete log line is not a valid JSON object, or records a snapshot_id or parent_snapshot_id that is not a string.
OSError

If the log cannot be read.

walk_lineage(index, head)

Select the snapshots reachable from head, newest first.

Follows parent_snapshot_id links back from head, which is what makes a history committed rather than merely recorded. Entries the walk does not reach are ignored: a crash between a log append and the HEAD write (see write_snapshot) leaves an entry the store never committed, and a later successful write appends after it, stranding it mid-file rather than at the end. Reachability is the only thing that tells such an entry apart from a real snapshot.

Parameters:

Name Type Description Default
index Mapping[str, LogEntry]

Every entry the dataset's log holds, keyed by snapshot_id, e.g. built from read_log_entries.

 required
head str

Id of the snapshot to walk back from, i.e. the dataset's HEAD.

 required

Returns:

Type Description
list[LogEntry]

The lineage head names, ordered newest first and ending at the
list[LogEntry]

snapshot with no parent.

Raises:

Type Description
StoreCorruptionError

If the lineage names a snapshot index has no entry for, or if the parent_snapshot_id links form a cycle.

write_snapshot(snapshot, *, store_root, dataset_id=None)

Write snapshot into the store and update the dataset's history.

Resolves which dataset the snapshot belongs to, then writes the full payload, appends the lightweight log.jsonl line, and moves HEAD onto the new snapshot.

Parameters:

Name Type Description Default
snapshot dict[str, Any]

A snapshot dict as produced by SnapshotBuilder. Must contain snapshot_id and dataset_path; parent_snapshot_id is filled in here and overwritten if already present.

 required
store_root Path

Path to the .dstrack/ directory, e.g. from resolve_store_root.

 required
dataset_id str | None

Continue this dataset's lineage explicitly. The dataset must already exist. When None, the dataset is matched by dataset_path against each existing dataset's latest snapshot; a new dataset_id is minted if none matches.

 None

Returns:

Type Description
SnapshotWriteResult

A SnapshotWriteResult describing
SnapshotWriteResult

where the snapshot landed and which lineage it joined.

Raises:

Type Description
KeyError

If snapshot lacks snapshot_id or dataset_path.
ValueError

If snapshot_id or an explicit dataset_id resolves to a path outside the store.
DatasetNotFoundError

If dataset_id is given but names no dataset in the store.
StoreCorruptionError

If an existing dataset's log.jsonl ends in a line that is not valid JSON.
OSError

If the store cannot be written to.