Data Engineering & MLOps
Mission. Ship data and models to production, and keep them healthy.
Scope
- Pipelines with Airflow or Dagster
- Spark, warehouses, and lakes
- Experiment tracking and reproducibility
- Model serving, monitoring, and rollback
- Drift detection, data and model versioning
Learning objectives
- Build a reproducible pipeline from ingestion to feature table
- Serve a model behind an API with monitoring and rollback
- Detect and respond to data drift and model degradation
Lessons
This track is scaffolded and waiting for content. See the curriculum dashboard for the roadmap and progress across all tracks.
Planned lessons (20)
Wave 3 — depth (provisional roadmap) — 20 lessons
foundations
- The Pipeline Mental Model Stages, contracts, and failure isolation for data in motion.
- Databases, Warehouses, Lakes What each store is for and when data belongs in it.
- ETL vs ELT Where transformation happens, and why the industry moved it.
orchestration
- Airflow: DAGs and Tasks Scheduling, dependencies, and retries without cron archaeology.
- Idempotent Pipelines Rerun anything, anytime, without fear.
- Data Quality Tests Great Expectations-style checks at every stage boundary.
transformations
- dbt-Style Transformations Versioned, tested, documented SQL as the pipeline's middle.
distributed
- Spark: When and Why Where a single machine stops being enough.
- Spark DataFrames Pandas-like API, distributed execution, different rules.
- Streaming: Batch vs Event Time Windows, watermarks, and what late means.
mlops
- Experiment Tracking MLflow-style records: what ran, with what, producing what.
- Reproducibility Environments, seeds, and runs that can be replayed.
- Data Versioning DVC and lakeFS: versioning inputs, not just code.
- Model Serving Patterns Batch scoring, online endpoints, and feature reuse.
- Monitoring Models in Production The metrics that say a model is quietly failing.
- Drift: Detection and Response Data drift, concept drift, and the playbook for both.
- Feature Stores, Gently Training-serving consistency without the platform team.
- CI for Data Projects Testing pipelines on every commit, not every crisis.
reference
- The MLOps Maturity Ladder Levels of production discipline, and the next rung for your team.
projects
- Case Study: A Production Pipeline from Scratch Ingestion to dashboard, scheduled, tested, monitored.