Reinforcement Learning & Policy Optimization

Welcome to the definitive guide to Reinforcement Learning and Deep Policy Optimization. Sequential decision making under uncertainty is grounded in dynamic programming, stochastic approximation, and the mathematical contraction mappings of the Bellman operators.

This curriculum derives every fundamental reinforcement learning theorem and algorithmic update from mathematical first principles and implements each agent from scratch in both pure Python and pure Rust without gym or external dependencies.

Explore lessons from the sidebar to inspect step-by-step mathematical proofs, zero-dependency implementations, interactive zoomable diagrams, and knowledge verification checkpoints.

Curriculum Architecture at a Glance#

This course provides comprehensive mathematical foundations and production-grade implementations across the reinforcement learning lifecycle:

1. Markov Decision Processes & Bellman Optimality: Formal 5-tuple MDP definitions $(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$, state-value $V^\pi(s)$ and action-value $Q^\pi(s, a)$ formulations, Bellman expectation equations, and Banach fixed-point contraction proofs for Bellman optimality operators ($\|T^* u - T^* v\|_\infty \le \gamma \|u - v\|_\infty$).
2. Temporal Difference & Tabular Control: Monte Carlo high-variance returns, Temporal Difference TD(0) bootstrapping error formulations, off-policy Q-Learning asymptotic convergence guarantees via Robbins-Monro conditions, and $\varepsilon$-greedy exploration schedules.
3. Deep Q-Networks (DQN) & Double DQN: Deadly triad stability analysis (function approximation, bootstrapping, off-policy training), experience replay buffer decorrelation, target network Polyak averaging, and Double DQN decoupled action selection and evaluation ($Y_t^{\text{DoubleQ}} = R_{t+1} + \gamma Q(S_{t+1}, \text{argmax}_a Q(S_{t+1}, a; \theta_t); \theta_t^-)$).
4. Policy Gradients & Proximal Policy Optimization (PPO): The Policy Gradient Theorem derivation ($\nabla_\theta J(\theta) = \mathbb{E}[\nabla_\theta \log \pi_\theta(a|s) Q^\pi(s, a)]$), baseline variance reduction, Generalized Advantage Estimation (GAE-$\lambda$), and PPO clipped surrogate objective functions ($L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t [\min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1-\varepsilon, 1+\varepsilon)\hat{A}_t)]$).

Core Pedagogical Standard#

• Step-by-Step Algebraic Derivations: Every Bellman operator, contraction proof, advantage estimation, and clipped objective is derived algebraically step by step.
• Zero-Dependency Implementations: Every RL agent and environment includes complete, runnable implementations in pure Python (standard library only) and pure Rust (std only).
• Interactive Visual Architecture: Explore interactive flowcharts, mathematical formulas rendered with KaTeX, expandable diagrams, and knowledge verification checkpoints.

The complete course source code and implementations are open on the Source Repository.