Python for Data Engineering

Python Data Structures for DE — Lists, Dicts, Sets

Reading

File I/O — CSV, JSON, Parquet with pandas

Reading

Working with APIs and Pagination

Reading

Regular Expressions for Data Cleaning

Reading

Logging, Error Handling and Retry Patterns

Reading

Practice — Ingest and Clean a REST API Response

Exercise

DataFrame Operations — Merge, Reshape, Explode

Reading

Handling Large CSVs with Chunking and dtypes

Reading

Time-Series Resampling and Rolling Windows

Reading

Polars — Fast DataFrame Library Overview

Reading

Memory Profiling and Optimisation

Reading

Practice — ETL Pipeline with pandas and Parquet

Exercise

SQLAlchemy Core and ORM for DE Tasks

Reading

psycopg2 and asyncpg for PostgreSQL

Reading

pymongo for MongoDB Operations

Reading

Bulk Inserts and Upserts — Performance Patterns

Reading

Connection Pooling with SQLAlchemy

Reading

Practice — Load Transformed Data into Postgres

Exercise

ETL vs ELT — When and Why

Reading

Incremental vs Full-Load Strategies

Reading

Change Data Capture (CDC) Concepts

Reading

Idempotency and Exactly-Once Semantics

Reading

Data Quality Checks in Pipeline Code

Reading

Practice — Build an Incremental ELT Loader

Exercise

Unit Testing Pipeline Functions with pytest

Reading

Mocking External Services in Tests

Reading

Great Expectations for Data Validation

Reading

Packaging a DE Project with pyproject.toml

Reading

Docker Containers for DE Scripts

Reading

Practice — Add Tests and Docker to an ETL Project

Exercise

Project Brief — Multi-Source Data Integration Pipeline

Reading

Ingest from REST API and Database

Exercise

Transform, Validate and Write to Parquet

Exercise

Test Suite and Docker Packaging

Exercise

Capstone — Submit Python DE Pipeline Project

Project