Parallel Computing & Performance (CPU to GPU)
Make code fast by making it parallel.
Make code fast by making it parallel. You start with the laws of parallelism (Amdahl, Gustafson, scalability) and CPU parallelism done right — thread pools, the C++17 parallel STL, and load balancing — then go deep on data parallelism with SIMD/AVX, and finally cross to the GPU: CUDA fundamentals (the SIMT model, threads/blocks/warps, the memory hierarchy), and GPU performance engineering (memory coalescing, shared-memory tiling, occupancy, reductions). The track culminates in optimizing a real compute kernel — matrix multiply — from a naive loop all the way to a tiled CUDA kernel, measured against hardware peak and vendor libraries. This is the performance-and-parallelism track of the Zero-to-Low-Level-Systems-Elite program and the direct on-ramp to the AI-chip / ML-systems branch, where GPUs run the math behind modern AI.
6 phases · 48 projects · v1.1.0
Target roles: AI-Chip / GPU Kernel Engineer (foundation), HPC / Performance Engineer, Low-Latency Engineer, ML-Systems Engineer (foundation)
Stack: C++17/20, CUDA, OpenMP, AVX / SIMD, Nsight
PHASE 1 · 8 projects
Foundations of Parallelism
Reason about parallel speedup: concurrency vs parallelism, Amdahl's and Gustafson's laws, scalability and efficiency, and task vs data parallelism.
01 · featured
Concurrency vs Parallelism
Build and time programs that demonstrate the concrete difference between concurrency (interleaving on one thread) and parallelism (simultaneous execution on multiple cores), then survey the three hardware parallelism levels — multi-core, SIMD, and GPU.
C++20 · std::thread · chrono
02 · featured
Amdahl's Law
Implement Amdahl's Law formula, tabulate predicted speedup curves for serial fractions 1–50%, parallelize a prefix-sum workload, measure its scaling from 1–8 threads, and fit the formula to recover the program's empirical serial fraction.
C++20 · std::thread · chrono
03 · featured
Gustafson's Law & Weak Scaling
Implement Gustafson's Law formula alongside Amdahl's, run a weak scaling experiment where problem size grows proportionally with thread count, measure weak scaling efficiency, and decide which scaling model applies to real-world system designs.
C++20 · std::thread · chrono
04 · featured
Speedup, Efficiency & Scalability
Build a professional scaling study: measure parallel dot product and Mandelbrot speedup from 1 to N threads, compute parallel efficiency at each point, isolate four overhead sources with micro-benchmarks, and diagnose bottleneck type from the efficiency curve shape.
C++20 · std::thread · std::atomic · chrono
05 · featured
Task vs Data Parallelism
Implement data parallelism (image blur split across rows) and task parallelism (3-stage producer-consumer pipeline) for the same processing workload, measure their performance profiles, demonstrate Monte Carlo as an embarrassingly parallel benchmark, and build a decision framework for choosing the right decomposition.
C++20 · std::thread · std::barrier · chrono
06 · featured
Decomposing a Problem
Apply block, cyclic, and dynamic partitioning strategies to an irregular workload, measure load imbalance for each, then run a 2D domain decomposition experiment demonstrating how row-major vs column-major partition alignment affects cache performance by more than 3×.
C++20 · std::thread · std::atomic · chrono
07 · featured
The Costs of Going Parallel
Build five targeted overhead micro-experiments — thread creation, uncontended vs contended mutex, false sharing, and memory bandwidth saturation — then combine all overhead sources into a deliberately bad parallel design that is measurably slower than serial with every thread count.
C++20 · std::thread · std::mutex · std::atomic · chrono
08 · featured
Scaling Study Lab
Complete a professional parallel scaling study: parallelize a 2D Gaussian convolution with correct decomposition, false-sharing avoidance, and median timing; run from 1 to N threads; compute speedup and efficiency; fit Amdahl's Law to the data; and write a structured performance report with bottleneck identification in journal/.
C++20 · std::thread · chrono
PHASE 2 · 8 projects
CPU Parallelism at Scale
Use all your cores well: thread pools, work distribution and load balancing, the C++17 parallel STL (std::execution), and parallel reduction patterns.
01 · featured
Parallelizing a Loop with Threads
Split a 500 M-element reduction loop across std::thread workers, combine partial results, and measure speedup vs thread count — the manual baseline before higher-level abstractions.
C++20 · std::thread · std::chrono · pthreads
02 · featured
Building a Thread Pool
Implement a ThreadPool class with a mutex-protected task queue, std::packaged_task for future-returning submit(), and graceful shutdown — then benchmark it against per-task thread creation.
C++20 · std::thread · std::mutex · std::condition_variable · std::packaged_task · std::future
03 · featured
Load Balancing & Work Stealing
Contrast static vs dynamic scheduling on imbalanced tasks using an atomic counter, measure the speedup from dynamic dispatch, and study the per-thread-deque work-stealing design used by TBB and Rayon.
C++20 · std::thread · std::atomic · std::chrono
04 · featured
The C++17 Parallel STL
Apply std::execution::par and par_unseq to for_each, transform, reduce, and sort; measure par vs seq speedup across data sizes; understand the TBB backend requirement and floating-point non-determinism of parallel reduce.
C++20 · std::execution · TBB · std::reduce · std::sort · std::transform
05 · featured
Parallel Reduction & Scan
Implement tree reduction with per-thread partial sums, demonstrate why shared-atomic accumulation serializes cores, use std::reduce with its associativity caveat, and implement parallel prefix-sum scan.
C++20 · std::thread · std::atomic · std::reduce · std::inclusive_scan · TBB
06 · featured
Parallel Patterns: Map, Reduce, Pipeline
Implement and benchmark the five core parallel skeletons — map, reduce, map-reduce (word count), pipeline (bounded queue), and fork-join (recursive async) — and learn to match each pattern to the right problem.
C++20 · std::thread · std::async · std::future · std::condition_variable
07 · featured
Concurrent Data Structures & Sharing
Measure false sharing on parallel histogram counters, fix it with alignas(64) padding, then implement per-thread aggregation to achieve 40x speedup over shared atomics — the universal high-throughput parallel counting pattern.
C++20 · std::thread · std::atomic · alignas(64) · std::array
08 · featured
Multicore Optimization Lab
Apply thread pool dispatch, per-thread unordered_map aggregation, and parallel-STL sort to a 64-document word-count corpus; run a full scaling study from 1 to 8 threads reporting speedup and parallel efficiency.
C++20 · std::thread · std::unordered_map · std::execution::par · std::sort · TBB · std::chrono
PHASE 3 · 8 projects
SIMD & Data Parallelism
Do many operations per instruction: vectorization, AVX intrinsics, alignment and data layout, and writing SIMD algorithms that the compiler can't auto-generate.
01 · featured
What is SIMD?
Explore the SIMD execution model by probing CPU vector ISA support, measuring scalar vs AVX2 array-add throughput, and understanding register widths across the SSE/AVX/AVX-512 family.
C++17 · GCC/Clang · AVX2 · immintrin.h
02 · featured
Auto-Vectorization
Write loops the compiler can auto-vectorize, read -fopt-info-vec reports, and systematically remove vectorization blockers including aliasing, loop-carried dependencies, and non-unit stride access.
C++17 · GCC/Clang · AVX2 · -march=native · -fopt-info-vec
03 · featured
AVX Intrinsics Basics
Hand-write a SAXPY kernel using __m256, _mm256_loadu_ps, and _mm256_fmadd_ps, then compare its throughput against scalar and auto-vectorized versions to understand when intrinsics add value.
C++17 · AVX2 · FMA3 · immintrin.h · GCC/Clang
04 · featured
Alignment & Data Layout for SIMD
Benchmark a particle physics update in Array-of-Structs vs Struct-of-Arrays layout, measure the SIMD throughput difference, and learn to use alignas(32) and std::aligned_alloc correctly.
C++17 · AVX2 · immintrin.h · alignas · std::aligned_alloc
05 · featured
SIMD Reductions & Horizontal Ops
Implement vectorized array sum and dot product using multi-lane FMA accumulators and a hadd/extractf128 horizontal reduction, measuring throughput and examining the floating-point associativity implications.
C++17 · AVX2 · FMA3 · immintrin.h · GCC/Clang
06 · featured
SIMD with Branches: Masking & Blend
Replace per-lane conditional branches with compare-and-blendv patterns to implement threshold filtering and ReLU activation using AVX2, measuring throughput and understanding the GPU warp divergence analogy.
C++17 · AVX2 · immintrin.h · GCC/Clang
07 · featured
Practical SIMD Algorithms
Implement a complete RGB-to-grayscale + threshold image processing pipeline in scalar and AVX2, measure end-to-end speedup, and evaluate the portability tradeoff between raw intrinsics and cross-platform SIMD libraries.
C++17 · AVX2 · FMA3 · immintrin.h · GCC/Clang
08 · featured
SIMD Optimization Lab
Vectorize a batched L2 normalization kernel using the full SIMD toolkit — auto-vectorization, manual AVX2+FMA with 4-accumulator reduction, and OpenMP multithreading — then produce a speedup breakdown table and efficiency analysis vs theoretical peak.
C++17 · AVX2 · FMA3 · OpenMP · immintrin.h · GCC/Clang
PHASE 4 · 8 projects
GPU Programming I — CUDA Fundamentals
Program the GPU: the SIMT model, threads/blocks/grids/warps, the device memory hierarchy, writing and launching your first kernels, and host/device data movement.
01 · featured
The GPU & the SIMT Model
Set up a CUDA environment and write your first device kernel, learning the CPU-vs-GPU design tradeoff, the SIMT execution model, warps, and the Streaming Multiprocessor hierarchy.
CUDA · C++ · nvcc · nvidia-smi
02 · featured
Your First Kernel — Vector Add
Implement GPU vector addition end-to-end: cudaMalloc, H2D copy, a __global__ kernel using the global index formula, D2H copy, verification, and a grid-stride loop for arbitrary array sizes.
CUDA · C++ · nvcc
03 · featured
GPU Memory Management
Benchmark and compare pageable host memory, pinned (page-locked) host memory, and Unified Memory for GPU data transfer, quantifying bandwidth differences and learning when each approach is appropriate.
CUDA · C++ · nvcc
04 · featured
Threads, Blocks & Grids in Depth
Map 2D problems (images, matrices) to 2D CUDA grids using dim3, apply boundary checks for non-multiple dimensions, study how block size affects occupancy, and use a 2D grid-stride loop for arbitrary-size problems.
CUDA · C++ · nvcc
05 · featured
Errors, Synchronization & Timing
Write a CUDA_CHECK macro for robust error handling, demonstrate why CPU-side timing is wrong for asynchronous kernels, and build a correct cudaEvent_t timing harness — the professional foundation for all future kernel benchmarking.
CUDA · C++ · nvcc · compute-sanitizer
06 · featured
Warps & Warp Divergence
Measure the cost of intra-warp branch divergence, write branchless alternatives using CUDA math intrinsics, and restructure a reduction kernel to be warp-uniform — learning the pattern that prevents the most common GPU performance pitfall.
CUDA · C++ · nvcc
07 · featured
2D Kernels — Image & Matrix Processing
Write a complete 2D box-blur image filter and a matrix transpose kernel, applying dim3 2D launches, clamped boundary handling, CUDA_CHECK error checking, and cudaEvent timing — and observe why naive transpose is bandwidth-limited by strided writes.
CUDA · C++ · nvcc
08 · featured
CUDA Fundamentals Lab
Build a complete GPU image processing pipeline (5x5 Gaussian blur + Sobel edge detection) integrating pinned memory, __constant__ memory, CUDA_CHECK error handling, GPU event timing, and a CPU-vs-GPU speedup comparison — the phase capstone.
CUDA · C++ · nvcc
PHASE 5 · 8 projects
GPU Programming II — Performance
Make GPU kernels fast: memory coalescing, shared-memory tiling, occupancy, warp divergence, efficient reductions, and profiling with Nsight.
01 · featured
Memory Coalescing
Benchmark coalesced vs strided global memory access patterns to quantify the bandwidth penalty of uncoalesced loads and writes on a real GPU.
CUDA · C++ · nvcc · cudaEvent_t
02 · featured
Shared Memory
Implement the staging pattern using __shared__ memory to build a latency microbenchmark and a 1D stencil kernel that reduces global memory traffic by exploiting data reuse.
CUDA · C++ · nvcc · __shared__ · __syncthreads__
03 · featured
Tiled Matrix Multiply
Implement the canonical tiled GEMM kernel using shared memory to reduce global memory traffic by a factor of TILE and measure the resulting GFLOP/s speedup over the naive kernel.
CUDA · C++ · nvcc · __shared__ · __syncthreads__ · GEMM
04 · featured
Shared Memory Bank Conflicts
Measure the performance penalty of 32-way shared memory bank conflicts in a matrix transpose, then eliminate them with +1 padding and verify the speedup and bank assignment analytically.
CUDA · C++ · nvcc · __shared__ · bank conflicts
05 · featured
Occupancy & Launch Configuration
Measure how block size, register count, and shared memory allocation affect SM occupancy, then observe the relationship between occupancy and throughput for both memory-bound and compute-bound kernels.
CUDA · C++ · nvcc · cudaOccupancyMaxActiveBlocksPerMultiprocessor · cudaEvent_t
06 · featured
Efficient Parallel Reductions
Implement the 5-step reduction optimization progression — from naive interleaved to warp-shuffle __shfl_down_sync — measuring the effective bandwidth at each step and explaining each bottleneck fixed.
CUDA · C++ · nvcc · __shfl_down_sync · __syncthreads__ · cudaEvent_t
07 · featured
Profiling with Nsight
Use Nsight Compute to profile naive and tiled GEMM kernels, read the key metrics (SM throughput, stall reasons, memory bandwidth %, occupancy), and follow the optimize-measure loop to connect profiler output to code fixes.
CUDA · C++ · nvcc · Nsight Compute · ncu · -lineinfo
08 · featured
GPU Optimization Lab
Apply the complete Phase 5 optimization stack (coalescing, shared memory staging, bank conflict padding, occupancy tuning) to a matrix transpose kernel, measuring bandwidth and profiling with Nsight at each step from 8% to 83%+ of peak.
CUDA · C++ · nvcc · Nsight Compute · ncu · __shared__ · cudaEvent_t
PHASE 6 · 8 projects
HPC, Heterogeneous Computing & the Capstone
Tie it together: OpenMP and NUMA-aware scaling, the CPU-vs-GPU tradeoff, and a capstone optimizing matrix multiply from naive to tiled CUDA against hardware peak.
01 · featured
OpenMP: Parallelism with Pragmas
Parallelize loops with #pragma omp parallel for, write correct parallel reductions, compare scheduling strategies, and benchmark OpenMP against parallel STL on a shared-memory machine.
C++17 · OpenMP · GCC/Clang -fopenmp
02 · featured
OpenMP Advanced: Tasks, Collapse, and False Sharing
Use OpenMP tasks for recursive parallelism, collapse(2) for nested loops, atomic/critical/reduction for shared state, and diagnose and fix false sharing between OpenMP threads.
C++17 · OpenMP 4.5+ · GCC/Clang -fopenmp
03 · featured
NUMA & Memory Affinity
Measure local vs remote NUMA memory access bandwidth, fix NUMA-unaware allocation using first-touch initialization, and control thread/memory placement with numactl and OMP_PROC_BIND.
C++17 · OpenMP · numactl · Linux /proc/sys
04 · featured
Choosing CPU vs GPU: The Throughput/Latency/Transfer Tradeoff
Measure PCIe transfer bandwidth, find the CPU vs GPU break-even problem size for elementwise ops, apply the roofline model to classify operations as bandwidth or compute bound, and build a decision framework.
CUDA · C++17 · nvcc · cuBLAS
05 · featured
CUDA Streams & Overlap: Hiding Transfer Latency
Use cudaMemcpyAsync and multiple CUDA streams to overlap PCIe data transfers with GPU kernel execution, measure the speedup from 2-stream and 4-stream pipelines, and understand multi-GPU dispatch with cudaSetDevice.
CUDA · C++17 · nvcc · cudaStream_t · cudaMallocHost
06 · featured
Standing on Giants: BLAS, cuBLAS, and Thrust
Call cuBLAS sgemm to perform GEMM and compare GFLOP/s against a naive kernel and the theoretical peak; use Thrust device algorithms without writing kernels; and build a decision framework for library vs custom kernel.
CUDA · cuBLAS · Thrust · C++17 · nvcc · OpenBLAS
07 · featured
Performance Methodology End-to-End
Apply the measure → roofline → bottleneck → fix → repeat optimization loop to a 2D convolution: use perf stat to identify cache bottlenecks, apply tiling and OpenMP, and report results honestly with GFLOP/s, % of peak, speedup, and machine context.
C++17 · OpenMP · perf stat · GCC -O3 -march=native
08 · featured
GEMM Optimization Lab (Track Capstone)
Optimize dense matrix multiply through the full stack: naive → cache-blocked → OpenMP → AVX2 → OpenBLAS on CPU, and naive CUDA → shared-memory tiled → cuBLAS on GPU. Report GFLOP/s and % of peak at every stage; package with CMake + GoogleTest; explain every speedup with roofline reasoning and document two tradeoffs.
C++17 · OpenMP · AVX2 · CUDA · cuBLAS · OpenBLAS · CMake · GoogleTest
Next step
Request Parallel Computing & Performance (CPU to GPU)
An advisor reviews the fit and confirms terms in writing. A request is not an acceptance and nothing is charged until you agree.