Python Data Engineering & Automation
Build the pipelines and automation that power real systems with Python: scripting and CLI tools, web scraping and data acquisition, data engineering (ETL, validation, warehousing), big data and orche…
Build the pipelines and automation that power real systems with Python: scripting and CLI tools, web scraping and data acquisition, data engineering (ETL, validation, warehousing), big data and orchestration (PySpark, Airflow, dbt, streaming), and the cloud, containerization, and reliability practices that ship pipelines to production. Prepares for Data Engineer, Automation, and platform-adjacent roles.
5 phases · 43 projects · v1.1.0
Target roles: Data Engineer, Automation Engineer, Analytics Engineer, Platform/Backend Engineer (Data)
Stack: Python 3.12, pandas, BeautifulSoup, Scrapy, Playwright, SQLAlchemy, PySpark, Airflow, dbt, Docker, boto3
PHASE 1 · 8 projects
Automation & Scripting
Automate real tasks with robust Python scripts and polished CLI tools.
01 · featured
Command-Line Tools with argparse
Build a real CLI tool with argparse: positional and optional arguments, subcommands, help text, and proper exit codes — the stdlib way to make scripts usable.
Python 3.12 · argparse
02
Modern CLIs with Typer
Build polished CLIs with Typer (built on Click): type-hint-driven commands, options with help, prompts, and colored output — far less boilerplate than argparse.
Python 3.12 · Typer · Click
03
Filesystem Automation
Automate bulk file operations with pathlib and shutil: organize, rename, deduplicate, and process files safely — with dry-run support and idempotency.
Python 3.12 · pathlib · shutil
04
Scheduling Automation
Run tasks on a schedule: cron for system-level scheduling and APScheduler for in-process jobs — and learn idempotency, overlap, and failure handling for scheduled work.
Python 3.12 · APScheduler · cron
05
API Data Collection
Collect data from REST APIs robustly: pagination, rate limiting, retries with backoff, authentication, and saving results — the foundation of API-based data ingestion.
Python 3.12 · httpx · requests
06
Spreadsheet Automation
Automate spreadsheet and CSV work: read, transform, and generate spreadsheets with pandas and openpyxl — formatted reports, multiple sheets, and formulas — replacing manual spreadsheet drudgery.
Python 3.12 · pandas · openpyxl
07
Notifications & Alerts
Send automated notifications — email (SMTP) and chat (incoming webhooks) — so your scripts and pipelines tell you about successes, failures, and results.
Python 3.12 · smtplib · httpx
08 · featured
Robust Automation Scripts
Make automation production-grade: structured logging, retries with backoff, clear exit codes, configuration, and idempotency — turning a fragile script into a reliable, observable job.
Python 3.12 · logging · tenacity
PHASE 2 · 8 projects
Web Scraping & Data Acquisition
Acquire data from the web responsibly — static and dynamic sites, APIs, and documents.
01 · featured
Web Scraping with BeautifulSoup
Fetch and parse static HTML to extract structured data: navigate the DOM with BeautifulSoup, select with CSS selectors, and turn a web page into clean rows.
Python 3.12 · httpx · beautifulsoup4
02
Forms, Sessions & Pagination
Scrape sites that need state: keep cookies across requests with a session, log in by posting a form, and follow pagination to collect every page of results.
Python 3.12 · httpx · beautifulsoup4
03 · featured
Ethical Scraping: robots.txt, Rate Limiting & Politeness
Scrape responsibly: read and respect robots.txt, identify yourself, rate-limit and back off, cache to avoid re-fetching, and understand the legal/ethical lines so your scraper is a good citizen.
Python 3.12 · httpx · urllib.robotparser
04 · featured
Large-Scale Crawling with Scrapy
Graduate from scripts to a framework: build a Scrapy spider that crawls, follows links, yields structured items, and exports data — with built-in concurrency, throttling, and pipelines.
Python 3.12 · Scrapy
05 · featured
Scraping Dynamic Sites with Playwright
Scrape JavaScript-rendered pages a plain HTTP fetch can't see: drive a real browser with Playwright, wait for content to load, interact (click/scroll/fill), and extract from the live DOM.
Python 3.12 · Playwright
06
Parsing HTML, XML & Structured Documents
Go beyond CSS selectors: parse XML/HTML with lxml and XPath, consume RSS/Atom feeds and sitemaps, and handle namespaces and malformed markup robustly.
Python 3.12 · lxml · feedparser
07 · featured
Extracting Data from PDFs & Documents
Liberate data trapped in documents: extract text and tables from PDFs with pdfplumber, read .docx/.xlsx files, and handle scanned pages with OCR — turning unstructured documents into structured data.
Python 3.12 · pdfplumber · python-docx · pytesseract
08 · featured
Storing Scraped Data: Pipelines, Dedup & Incremental Saves
Close the loop on acquisition: store scraped records durably (CSV/JSONL/SQLite), deduplicate with stable keys, and scrape incrementally so re-runs only fetch what's new — a complete, resumable acquisition pipeline.
Python 3.12 · sqlite3 · pandas
PHASE 3 · 9 projects
Data Engineering Core
Build reliable ETL: clean, validate, model, and load data into warehouses.
01 · featured
Efficient pandas at Scale
Make pandas fast and memory-efficient on large data: pick the right dtypes, vectorize instead of looping, process in chunks, and measure memory — so your transforms scale beyond toy datasets.
Python 3.12 · pandas
02 · featured
Data Cleaning & Transformation
Turn messy real-world data into a clean dataset: handle missing values, fix types and inconsistent categories, remove duplicates, parse dates, and reshape — the unglamorous work that makes data usable.
Python 3.12 · pandas
03 · featured
Data Validation with Pandera
Stop bad data at the gate: declare schemas with Pandera that enforce types, ranges, nullability, uniqueness, and custom rules — failing loudly when input violates expectations instead of silently corrupting downstream data.
Python 3.12 · pandas · pandera
04 · featured
Designing ETL Pipelines
Structure data pipelines properly: separate Extract, Transform, and Load into composable stages, understand ETL vs ELT, and build a pipeline that's modular, testable, idempotent, and observable.
Python 3.12 · pandas
05 · featured
SQL Databases with SQLAlchemy
Move data in and out of relational databases efficiently: connect with SQLAlchemy Core, run parameterized queries safely, bulk-load with pandas, and use transactions — the load layer of real pipelines.
Python 3.12 · SQLAlchemy · pandas · psycopg
06 · featured
Data Warehousing & Dimensional Modeling
Model data for analytics: understand warehouses vs OLTP databases, design star schemas with fact and dimension tables, handle slowly changing dimensions, and build a dimensional model from raw data.
Python 3.12 · pandas · SQL
07 · featured
Columnar Storage with Parquet
Store analytical data efficiently: use Parquet's columnar format for fast, compressed, typed reads, partition datasets for query pruning, and understand why columnar beats CSV/row formats for analytics.
Python 3.12 · pandas · pyarrow
08 · featured
Incremental & Idempotent Loads
Load only what changed, safely: track high-water marks, detect changes, implement upserts/merge, design idempotent loads that survive retries and backfills without duplicating data — the heart of production pipelines.
Python 3.12 · SQLAlchemy · pandas
09 · featured
Data Quality Checks & Monitoring
Keep data trustworthy over time: define quality dimensions and checks (completeness, freshness, uniqueness, distribution), run them as automated gates, monitor for anomalies, and alert when data quality degrades.
Python 3.12 · pandas · pandera
PHASE 4 · 9 projects
Big Data & Orchestration
Scale and schedule pipelines with PySpark, Airflow, dbt, and streaming.
01 · featured
PySpark Fundamentals
Process data that doesn't fit on one machine: understand distributed computing, create a SparkSession, load and explore data as a Spark DataFrame, and grasp how Spark partitions and parallelizes work across a cluster.
Python 3.12 · PySpark
02 · featured
Spark Transformations & Actions
Understand Spark's execution model: lazy transformations build a plan, actions trigger it, and wide transformations cause shuffles. Read the DAG, know what's expensive, and reason about how Spark runs your code.
Python 3.12 · PySpark
03 · featured
Spark SQL, Joins & Aggregations
Transform big data with Spark SQL: write SQL against DataFrames, join large datasets, aggregate and window at scale, and understand join strategies (shuffle vs broadcast) that determine performance.
Python 3.12 · PySpark · Spark SQL
04 · featured
Spark Performance Tuning
Make Spark jobs fast: minimize shuffles, cache reused data, control partitioning, fix data skew, and read the Spark UI to find bottlenecks — the optimization skills that turn hours into minutes.
Python 3.12 · PySpark
05 · featured
Workflow Orchestration with Airflow
Schedule and orchestrate pipelines: model a workflow as an Airflow DAG of tasks with dependencies, schedule it, and understand operators, the scheduler, and how Airflow runs and tracks data pipelines.
Python 3.12 · Apache Airflow
06 · featured
Production Airflow: Retries, Sensors, Backfills & Idempotency
Make orchestrated pipelines production-grade: idempotent tasks parameterized by execution date, retries and alerting, sensors that wait for data, and safe backfills of historical windows.
Python 3.12 · Apache Airflow
07 · featured
Analytics Engineering with dbt
Transform data in the warehouse the modern way: build dbt models as SQL SELECTs, layer staging→marts, add tests and documentation, and use refs/lineage — the ELT 'T' that analytics teams run on.
Python 3.12 · dbt
08 · featured
Streaming Data with Kafka
Process data as it arrives, not in nightly batches: understand event streaming, produce and consume messages with Kafka, grasp topics/partitions/offsets/consumer groups, and reason about batch vs streaming.
Python 3.12 · Kafka · confluent-kafka
09 · featured
Stream Processing with Spark Structured Streaming
Compute on streams with the same DataFrame API as batch: read a stream, apply transformations and windowed aggregations, handle event-time and late data with watermarks, and write results with checkpointing for fault tolerance.
Python 3.12 · PySpark · Structured Streaming
PHASE 5 · 9 projects
Cloud, Testing & Reliability
Ship pipelines: cloud SDKs, containers, CI/CD, testing, logging, and observability.
01 · featured
Cloud Storage with boto3 (the cloud provider SDK)
Work with cloud infrastructure from Python: use boto3 to read/write object storage (the data lake), handle credentials securely, stream large objects, and understand object storage as the foundation of cloud data platforms.
Python 3.12 · boto3 · object storage-compatible object storage
02 · featured
Containerizing Data Jobs with Docker
Package a pipeline so it runs identically everywhere: write a Dockerfile, build a lean image, pass config via environment, handle dependencies, and run your job as a reproducible container.
Python 3.12 · Docker
03 · featured
Testing Data Pipelines with pytest
Test pipelines like software: unit-test pure transforms, use fixtures and sample data, test edge cases, mock external systems, and add data-quality assertions — so changes don't silently break your data.
Python 3.12 · pytest · pandas
04 · featured
CI/CD for Data Pipelines with GitHub Actions
Automate quality and deployment: build a GitHub Actions workflow that lints, tests, and builds your pipeline on every push, then deploys (e.g. pushes an image) — so changes are validated and shipped automatically.
Python 3.12 · GitHub Actions · Docker
05
Configuration & Secrets Management
Manage config and secrets the right way: separate config from code (12-factor), layer env/files/defaults, validate config with Pydantic Settings, and pull secrets from a secrets manager instead of env or code.
Python 3.12 · pydantic-settings · the cloud provider Secrets Manager
06 · featured
Logging, Monitoring & Observability
See what your pipelines are doing: emit structured logs, expose metrics, trace runs, and build the observability that lets you answer 'is it healthy?' and debug failures fast — the difference between flying blind and operating with confidence.
Python 3.12 · logging · Prometheus
07 · featured
Reliability Patterns for Pipelines
Design pipelines that survive failure: retries with backoff, idempotency, dead-letter queues for poison records, circuit breakers and timeouts for flaky dependencies, and graceful degradation — so partial failures don't become outages or corrupt data.
Python 3.12 · tenacity
08 · featured
Serverless & Event-Driven Data Processing
Process data without managing servers: write a function (a serverless function runtime) triggered by events (a file landing in object storage, a queue message), understand the serverless model and its tradeoffs, and design event-driven pipelines.
Python 3.12 · a serverless function runtime · object storage · SQS
09 · featured
Capstone — Production Data Pipeline
Build a complete, production-grade data pipeline end to end: ingest from a source, validate and transform, load to a warehouse with incremental/idempotent loads and quality gates, orchestrated/scheduled, tested, containerized, with logging and alerting — integrating every skill in the track.
Python 3.12 · pandas · SQLAlchemy · Airflow · Docker · pytest
Next step
Request Python Data Engineering & Automation
An advisor reviews the fit and confirms terms in writing. A request is not an acceptance and nothing is charged until you agree.