Data Engineering
Refresh the pipeline and analytics fundamentals that fade fastest: batch vs streaming, data modeling, file layouts, warehouse SQL, quality, lineage, and operations.
40 concepts249 live tasks
Start practicing freeWhat's inside
Batch, Streaming & Orchestration
Batch and streaming differ in latency, not importance
Event time and processing time answer different clocks
Watermarks are guesses about completeness
Exactly-once means exactly-once effects, not magic delivery
Orchestration is dependency-aware scheduling
Backfills are normal pipeline work, not emergencies
Idempotent pipelines make retries boring
DAG shape determines blast radius
Data Modeling & Warehouses
A fact table's grain is the first design decision
Star schemas optimize analytics around facts and dimensions
Slowly changing dimensions preserve changing context
Analytics models denormalize for query shape
Wide and tall event models trade schema stability for query ease
Surrogate keys are stable joins; natural keys are business facts
A semantic layer centralizes metric definitions
Data marts serve a specific audience or workflow
Storage, File Formats & Lakehouse
Columnar storage reads analytics-shaped data efficiently
Parquet row groups are the unit of pruning and parallelism
Partitioning helps only when queries filter on the partition key
Many tiny files can dominate query planning and metadata costs
Compression trades CPU for IO and storage
Schema evolution must preserve old data and old readers
Object storage changes filesystem assumptions
Table formats add transactions and metadata over files
Analytical SQL & Performance
Window functions keep row detail while adding context
Approximate distinct counts trade tiny error for huge savings
Join performance depends on data size and skew
A CTE may be syntax, optimization boundary, or materialization
Incremental aggregates avoid recomputing all history
Planner statistics decide whether the engine believes your query
SELECT * is expensive in columnar analytics
Historical analysis needs the dimension version from that time
Quality, Governance & Operations
Data contracts make producers responsible for breaking changes
Freshness, completeness, validity, and uniqueness catch different failures
Data tests are assertions about trust
Lineage answers what breaks if this changes
PII handling starts with classification
Data observability watches behavior, not just job status
Schema drift is a compatibility problem, not just a parser problem
A data incident needs correction and communication