Data Engineering

Dataset Versioning

Tracking, versioning, and reproducing AI datasets across their full lifecycle — from raw collection through cleaning, labelling, and splits — for reproducibility and rollback.

Tools
DVC, Hugging Face Hub, lakeFS
Metadata tracked
schema + stats + lineage
Key benefit
reproducibility

Table of Contents

SECTION 01

Why Version Datasets?

Without versioning, you can't answer: what data did model v3.1 train on? Why did performance drop after last week's data pipeline run? Can we reproduce experiment #47 exactly? Dataset versioning gives every training run a pointer to the exact dataset state — enabling reproducibility, debugging, and safe rollback when a data update degrades model quality.

SECTION 02

DVC for Data Versioning

DVC (Data Version Control) stores large files in remote storage (S3, GCS, Azure) " "and tracks them in git as small pointer files (.dvc). " "Every git commit can include a pointer to the exact dataset version used.

# Initialize DVC in a git repo
# dvc init
# dvc remote add -d myremote s3://my-bucket/dvc-store
import subprocess
def version_dataset(dataset_path: str, message: str):
    # Add dataset to DVC tracking
    subprocess.run(["dvc", "add", dataset_path], check=True)
    # Stage DVC pointer file + data dir for git
    subprocess.run(["git", "add", f"{dataset_path}.dvc", ".gitignore"], check=True)
    # Push data to remote
    subprocess.run(["dvc", "push"], check=True)
    # Commit git pointer
    subprocess.run(["git", "commit", "-m", f"Dataset: {message}"], check=True)
    print(f"Dataset versioned: {message}")
SECTION 03

Hugging Face Hub

The Hugging Face Hub is the standard for sharing and versioning NLP/LLM datasets. Every dataset repo is a git-LFS repository — push new versions as commits, load any version by git revision hash. Dataset cards (README.md) document provenance, splits, and intended use. The `datasets` library caches and loads any Hub dataset by name and revision.

SECTION 04

Dataset Cards

A dataset card is the metadata document for a dataset. Minimum contents: dataset description and intended use, data source and collection methodology, preprocessing steps applied, train/val/test split sizes and strategy, label distribution statistics, known biases and limitations, and licence + citation information. Without a dataset card, the dataset is effectively undocumented.

SECTION 05

Split Reproducibility

Use a fixed random seed and deterministic split logic so the same split " "is produced every time. Store split indices alongside the dataset.

import hashlib
from sklearn.model_selection import train_test_split
def reproducible_split(ids: list, train_ratio=0.8, val_ratio=0.1, seed=42):
    # Sort deterministically before splitting
    ids_sorted = sorted(ids)
    train, temp = train_test_split(ids_sorted, test_size=1-train_ratio, random_state=seed)
    val_size = val_ratio / (1 - train_ratio)
    val, test = train_test_split(temp, test_size=1-val_size, random_state=seed)
    return {"train": train, "val": val, "test": test}
SECTION 06

Data Lineage Tracking

Track the full provenance of every dataset version: which source documents were included, which filters were applied and with what parameters, which annotation job produced which labels, and which model generated synthetic examples. Store this as a lineage graph in a metadata store (e.g. Postgres). When a model trained on dataset v2.3 shows unexpected behaviour, lineage lets you trace it back to the exact data change that caused it.

SECTION 07

Version Control Systems for Data

Traditional git is designed for code, not large binary files. Tools like DVC (Data Version Control), Pachyderm, and LakeFS add dataset versioning on top of object storage (S3, GCS). They track data lineage, enable branching (experiment on a copy), and support reproducibility (recreate the exact dataset used for a model).

# DVC dataset versioning
# dvc.yaml: Declare your dataset
datasets:
  raw_data:
    path: s3://my-bucket/data/raw/
    version: "1.2.3"  # Immutable version tag

# dvc.lock: Lock file tracks exact versions used in training
# If you train model A on dataset 1.2.3 and later update to 1.3.0,
# you can always recreate model A by checking out dvc.lock

# Usage
dvc get s3://my-bucket/dvc-repo raw_data -v 1.2.3
# Downloads exact dataset version 1.2.3, reproducibly

Branch-Based Experimentation

Just like git branches for code, data versions enable branching for ML experiments. Experiment on a "feature branch" of your dataset, A/B test, and merge back to main only if it improves metrics. Non-merged experiments are discarded without cluttering production data.

# DVC branching for ML experiments
git checkout -b experiment/new-preprocessing
# Modify raw_data to add new features
dvc repro  # Recompute dataset and model
# Train model on the new version
python train.py --dataset-version=$(dvc dag --output)
# If model improves, merge back
git merge experiment/new-preprocessing
dvc push  # Push dataset version to storage
SECTION 08

Dataset Storage & Cost Optimization

Storing every version of every dataset indefinitely is prohibitively expensive. Implement retention policies: keep "main" branch versions permanently, but garbage collect old experiment branches after 30 days. Use compression and deduplication to reduce storage footprint.

Storage Strategy Cost/month Retention Access Speed Use Case
S3 Standard (hot) $23/TB Unlimited Immediate Active datasets, current experiments
S3-IA (warm) $12.50/TB 30+ days 1–2 min retrieval Older versions, occasional access
Glacier (cold) $4/TB 6+ months 1–12 hours Archive, compliance holds
Dedup + Compression -40% to -60% Same tier Same tier + decompress latency Large datasets with overlaps

Reproducibility Best Practice: Always lock dataset versions in your experiment config or dvc.lock file. Committing only the git version without the data version is incomplete; weeks later, someone pulls your code but the default dataset has been updated, silently breaking reproducibility. Use semantic versioning for datasets (MAJOR.MINOR.PATCH) to signal breaking changes (schema changes = MAJOR).

For production models, enforce that training datasets are immutable. Create a "golden dataset" snapshot used for all future training; any update requires a new version and retraining. This prevents the "data drift" bug where model training uses different data than what it was evaluated on.

Dataset Versioning in CI/CD Pipelines: Integrate dataset versioning into your CI/CD workflows. A pull request that modifies training data should trigger automatic retraining and evaluation on the new dataset version. If performance regresses, block the merge until the issue is fixed. This prevents accidental data corruption or bad data from reaching production models. DVC integrates with GitHub Actions and GitLab CI natively; define a workflow that runs: validate data schema → train model → evaluate on test set → require approval if metrics degrade.

Implement dataset branching that mirrors code branches: when you branch code to develop a feature, branch the dataset too. Train on the branched dataset, evaluate, and merge both back to main if approved. This ensures code and data stay synchronized and can be checked out together for reproducibility.

Monitoring and observability are essential for production systems. Set up comprehensive logging at every layer: API requests, model predictions, database queries, cache hits/misses. Use structured logging (JSON) to enable filtering and aggregation across thousands of servers. For production deployments, track not just errors but also latency percentiles (p50, p95, p99); if p99 latency suddenly doubles, something is wrong even if error rates are normal. Set up alerting based on SLO violations: if a service is supposed to have 99.9% availability and it drops to 99.5%, alert immediately. Use distributed tracing (Jaeger, Lightstep) to track requests across multiple services; a slow end-to-end latency might be hidden in one deep service call, invisible in aggregate metrics.

For long-running ML jobs (training, batch inference), implement checkpoint recovery and graceful degradation. If a training job crashes after 2 weeks, you want to resume from the last checkpoint, not restart from scratch. Implement job orchestration with Kubernetes or Airflow to handle retries, resource allocation, and dependency management. Use feature flags for safe deployment: deploy new model versions behind a flag that's off by default, gradually roll out to 1% of users, 10%, then 100%, monitoring metrics at each step. If something goes wrong, flip the flag back instantly. This approach reduces risk and enables fast rollback.

Finally, build a culture of incident response and post-mortems. When something breaks (and it will), document the incident: timeline, root cause, mitigation steps, and preventive measures. Use incidents as learning opportunities; blameless post-mortems focus on systems, not people. Share findings across teams to prevent repeat incidents. A well-documented incident history is an organization's institutional knowledge about system failures and how to avoid them.

The rapid evolution of AI infrastructure requires continuous learning and adaptation. Teams should establish regular tech talks and knowledge-sharing sessions where engineers present lessons learned from production deployments, performance optimization work, and incident postmortems. Create internal wiki pages documenting best practices specific to your organization: how to debug common failure modes, performance tuning guides for your hardware, and checklists for safe deployments. This prevents repeating mistakes and accelerates onboarding of new team members.

Build relationships with vendors and open-source communities. If you encounter bugs in frameworks (PyTorch, JAX), file detailed reports. If you have questions, ask on forums; community members often have encountered similar issues. For mission-critical infrastructure, consider purchasing support contracts with vendors (PyTorch, HuggingFace, cloud providers). Support gives you direct access to engineers who understand your system and can prioritize fixes. This is insurance against production outages caused by third-party software bugs.

Finally, remember that optimization is a journey, not a destination. Today's cutting-edge technique becomes tomorrow's baseline. Allocate 10-15% of engineering time to exploration and experimentation. Some experiments will fail, but successful ones compound into significant efficiency gains. Foster a culture of continuous improvement: measure, analyze, iterate, and share results. The teams that stay ahead are those that invest in understanding their systems deeply and adapting proactively to new technologies and changing demands.

Key Takeaway: Success in GenAI infrastructure depends on mastering fundamentals: understand your hardware constraints, profile your workloads, measure everything, and iterate. The most sophisticated techniques (dynamic batching, mixed precision, distributed training) build on solid foundations of clear thinking and empirical validation. Avoid cargo-cult engineering: if you don't understand why a technique helps your specific use case, it probably won't. Invest time in understanding root causes, not just applying trendy solutions. Over time, this rigor will compound into significant competitive advantage.

Related concepts

Data Contracts Data Lineage Metadata Design ETL for AI Data Quality