Cognixa Labs
AI Model Development

Advanced Reinforcement Learning: From Theory to Production Systems

Master advanced reinforcement learning techniques including deep Q-learning, policy gradients, and actor-critic methods with practical implementations for real-world applications.

By Cognixa Labs Research Team January 25, 2024 25 min read

Reinforcement Learning (RL) has emerged as one of the most powerful paradigms in artificial intelligence, enabling agents to learn optimal behaviors through interaction with their environment. Recent breakthroughs in deep RL have led to superhuman performance in games, robotics, and autonomous systems, with applications showing 40-60% improvement over traditional control methods.

Understanding Reinforcement Learning

Core Concepts

RL is built on the foundation of the Markov Decision Process (MDP):

  • State (S): Current situation of the agent
  • Action (A): Choices available to the agent
  • Reward (R): Feedback from the environment
  • Policy (π): Strategy for selecting actions
  • Value Function (V): Expected cumulative reward

Advanced RL Framework

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import numpy as np
import random
from collections import deque, namedtuple
from typing import Dict, List, Tuple, Optional
import gym
from dataclasses import dataclass
import matplotlib.pyplot as plt

@dataclass
class RLConfig:
    """Configuration for RL algorithms"""
    state_dim: int
    action_dim: int
    hidden_dim: int = 256
    learning_rate: float = 3e-4
    gamma: float = 0.99
    tau: float = 0.005
    buffer_size: int = 1000000
    batch_size: int = 256
    exploration_noise: float = 0.1
    policy_noise: float = 0.2
    noise_clip: float = 0.5
    policy_freq: int = 2

class ReplayBuffer:
    """Experience replay buffer for off-policy learning"""
    
    def __init__(self, capacity: int, state_dim: int, action_dim: int):
        self.capacity = capacity
        self.buffer = deque(maxlen=capacity)
        self.experience = namedtuple('Experience', 
                                   ['state', 'action', 'reward', 'next_state', 'done'])
    
    def push(self, state: np.ndarray, action: np.ndarray, reward: float, 
             next_state: np.ndarray, done: bool) -> None:
        """Add experience to buffer"""
        experience = self.experience(state, action, reward, next_state, done)
        self.buffer.append(experience)
    
    def sample(self, batch_size: int) -> Tuple[torch.Tensor, ...]:
        """Sample batch of experiences"""
        batch = random.sample(self.buffer, batch_size)
        
        states = torch.FloatTensor([e.state for e in batch])
        actions = torch.FloatTensor([e.action for e in batch])
        rewards = torch.FloatTensor([e.reward for e in batch]).unsqueeze(1)
        next_states = torch.FloatTensor([e.next_state for e in batch])
        dones = torch.BoolTensor([e.done for e in batch]).unsqueeze(1)
        
        return states, actions, rewards, next_states, dones
    
    def __len__(self) -> int:
        return len(self.buffer)

class PrioritizedReplayBuffer:
    """Prioritized Experience Replay buffer"""
    
    def __init__(self, capacity: int, alpha: float = 0.6):
        self.capacity = capacity
        self.alpha = alpha
        self.buffer = deque(maxlen=capacity)
        self.priorities = deque(maxlen=capacity)
        self.experience = namedtuple('Experience', 
                                   ['state', 'action', 'reward', 'next_state', 'done'])
    
    def push(self, state: np.ndarray, action: np.ndarray, reward: float, 
             next_state: np.ndarray, done: bool, priority: float = None) -> None:
        """Add experience with priority"""
        experience = self.experience(state, action, reward, next_state, done)
        
        if priority is None:
            priority = max(self.priorities) if self.priorities else 1.0
        
        self.buffer.append(experience)
        self.priorities.append(priority)
    
    def sample(self, batch_size: int, beta: float = 0.4) -> Tuple[torch.Tensor, ...]:
        """Sample batch with importance sampling"""
        priorities = np.array(self.priorities)
        probabilities = priorities ** self.alpha
        probabilities /= probabilities.sum()
        
        indices = np.random.choice(len(self.buffer), batch_size, p=probabilities)
        
        # Importance sampling weights
        weights = (len(self.buffer) * probabilities[indices]) ** (-beta)
        weights /= weights.max()
        
        batch = [self.buffer[idx] for idx in indices]
        
        states = torch.FloatTensor([e.state for e in batch])
        actions = torch.FloatTensor([e.action for e in batch])
        rewards = torch.FloatTensor([e.reward for e in batch]).unsqueeze(1)
        next_states = torch.FloatTensor([e.next_state for e in batch])
        dones = torch.BoolTensor([e.done for e in batch]).unsqueeze(1)
        weights = torch.FloatTensor(weights).unsqueeze(1)
        
        return states, actions, rewards, next_states, dones, weights, indices
    
    def update_priorities(self, indices: List[int], priorities: List[float]) -> None:
        """Update priorities for sampled experiences"""
        for idx, priority in zip(indices, priorities):
            self.priorities[idx] = priority

Deep Q-Network (DQN) Implementation

class DQNNetwork(nn.Module):
    """Deep Q-Network with dueling architecture"""
    
    def __init__(self, state_dim: int, action_dim: int, hidden_dim: int = 256):
        super().__init__()
        
        # Shared feature layers
        self.feature_layers = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU()
        )
        
        # Dueling architecture
        self.value_stream = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim // 2),
            nn.ReLU(),
            nn.Linear(hidden_dim // 2, 1)
        )
        
        self.advantage_stream = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim // 2),
            nn.ReLU(),
            nn.Linear(hidden_dim // 2, action_dim)
        )
    
    def forward(self, state: torch.Tensor) -> torch.Tensor:
        features = self.feature_layers(state)
        
        value = self.value_stream(features)
        advantage = self.advantage_stream(features)
        
        # Combine value and advantage
        q_values = value + advantage - advantage.mean(dim=1, keepdim=True)
        return q_values

class DQNAgent:
    """Deep Q-Network agent with Double DQN and Prioritized Replay"""
    
    def __init__(self, config: RLConfig, use_prioritized_replay: bool = True):
        self.config = config
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        
        # Networks
        self.q_network = DQNNetwork(config.state_dim, config.action_dim, config.hidden_dim).to(self.device)
        self.target_network = DQNNetwork(config.state_dim, config.action_dim, config.hidden_dim).to(self.device)
        
        # Copy weights to target network
        self.target_network.load_state_dict(self.q_network.state_dict())
        
        # Optimizer
        self.optimizer = optim.Adam(self.q_network.parameters(), lr=config.learning_rate)
        
        # Replay buffer
        if use_prioritized_replay:
            self.memory = PrioritizedReplayBuffer(config.buffer_size)
        else:
            self.memory = ReplayBuffer(config.buffer_size, config.state_dim, config.action_dim)
        
        self.use_prioritized_replay = use_prioritized_replay
        
        # Exploration
        self.epsilon = 1.0
        self.epsilon_min = 0.01
        self.epsilon_decay = 0.995
        
        # Training statistics
        self.training_step = 0
    
    def select_action(self, state: np.ndarray, training: bool = True) -> int:
        """Select action using epsilon-greedy policy"""
        if training and random.random() < self.epsilon:
            return random.randrange(self.config.action_dim)
        
        state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
        q_values = self.q_network(state_tensor)
        return q_values.argmax().item()
    
    def store_experience(self, state: np.ndarray, action: int, reward: float, 
                        next_state: np.ndarray, done: bool) -> None:
        """Store experience in replay buffer"""
        action_one_hot = np.zeros(self.config.action_dim)
        action_one_hot[action] = 1.0
        
        if self.use_prioritized_replay:
            self.memory.push(state, action_one_hot, reward, next_state, done)
        else:
            self.memory.push(state, action_one_hot, reward, next_state, done)
    
    def train(self) -> Dict[str, float]:
        """Train the DQN agent"""
        if len(self.memory) < self.config.batch_size:
            return {}
        
        if self.use_prioritized_replay:
            states, actions, rewards, next_states, dones, weights, indices = self.memory.sample(
                self.config.batch_size, beta=min(1.0, 0.4 + self.training_step * 0.0001)
            )
        else:
            states, actions, rewards, next_states, dones = self.memory.sample(self.config.batch_size)
            weights = torch.ones(self.config.batch_size, 1)
        
        states = states.to(self.device)
        actions = actions.to(self.device)
        rewards = rewards.to(self.device)
        next_states = next_states.to(self.device)
        dones = dones.to(self.device)
        weights = weights.to(self.device)
        
        # Current Q values
        current_q_values = self.q_network(states)
        current_q_values = (current_q_values * actions).sum(dim=1, keepdim=True)
        
        # Double DQN: use main network to select actions, target network to evaluate
        with torch.no_grad():
            next_actions = self.q_network(next_states).argmax(dim=1, keepdim=True)
            next_q_values = self.target_network(next_states).gather(1, next_actions)
            target_q_values = rewards + (self.config.gamma * next_q_values * ~dones)
        
        # Compute loss
        td_errors = target_q_values - current_q_values
        loss = (weights * td_errors.pow(2)).mean()
        
        # Update priorities
        if self.use_prioritized_replay:
            priorities = td_errors.abs().detach().cpu().numpy().flatten()
            self.memory.update_priorities(indices, priorities)
        
        # Optimize
        self.optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(self.q_network.parameters(), 1.0)
        self.optimizer.step()
        
        # Update target network
        if self.training_step % 1000 == 0:
            self.target_network.load_state_dict(self.q_network.state_dict())
        
        # Decay epsilon
        self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)
        
        self.training_step += 1
        
        return {
            'loss': loss.item(),
            'epsilon': self.epsilon,
            'q_value_mean': current_q_values.mean().item()
        }

Actor-Critic Methods (A3C/PPO)

class ActorCriticNetwork(nn.Module):
    """Actor-Critic network for continuous control"""
    
    def __init__(self, state_dim: int, action_dim: int, hidden_dim: int = 256):
        super().__init__()
        
        # Shared feature extractor
        self.shared_layers = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU()
        )
        
        # Actor (policy) head
        self.actor_mean = nn.Linear(hidden_dim, action_dim)
        self.actor_std = nn.Linear(hidden_dim, action_dim)
        
        # Critic (value) head
        self.critic = nn.Linear(hidden_dim, 1)
        
        # Initialize weights
        self.apply(self._init_weights)
    
    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            torch.nn.init.orthogonal_(module.weight, 0.01)
            module.bias.data.zero_()
    
    def forward(self, state: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
        features = self.shared_layers(state)
        
        # Actor outputs
        action_mean = torch.tanh(self.actor_mean(features))
        action_std = F.softplus(self.actor_std(features)) + 1e-5
        
        # Critic output
        value = self.critic(features)
        
        return action_mean, action_std, value
    
    def get_action_and_value(self, state: torch.Tensor, action: torch.Tensor = None):
        action_mean, action_std, value = self.forward(state)
        
        # Create distribution
        dist = torch.distributions.Normal(action_mean, action_std)
        
        if action is None:
            action = dist.sample()
        
        log_prob = dist.log_prob(action).sum(dim=-1)
        entropy = dist.entropy().sum(dim=-1)
        
        return action, log_prob, entropy, value

class PPOAgent:
    """Proximal Policy Optimization agent"""
    
    def __init__(self, config: RLConfig, clip_epsilon: float = 0.2, 
                 value_coef: float = 0.5, entropy_coef: float = 0.01):
        self.config = config
        self.clip_epsilon = clip_epsilon
        self.value_coef = value_coef
        self.entropy_coef = entropy_coef
        
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        
        # Network
        self.network = ActorCriticNetwork(
            config.state_dim, config.action_dim, config.hidden_dim
        ).to(self.device)
        
        # Optimizer
        self.optimizer = optim.Adam(self.network.parameters(), lr=config.learning_rate)
        
        # Storage for rollouts
        self.rollout_buffer = []
    
    def select_action(self, state: np.ndarray) -> Tuple[np.ndarray, float, float]:
        """Select action using current policy"""
        state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
        
        with torch.no_grad():
            action, log_prob, _, value = self.network.get_action_and_value(state_tensor)
        
        return action.cpu().numpy()[0], log_prob.item(), value.item()
    
    def store_experience(self, state: np.ndarray, action: np.ndarray, reward: float,
                        log_prob: float, value: float, done: bool) -> None:
        """Store experience for batch update"""
        self.rollout_buffer.append({
            'state': state,
            'action': action,
            'reward': reward,
            'log_prob': log_prob,
            'value': value,
            'done': done
        })
    
    def compute_gae(self, next_value: float = 0.0, gae_lambda: float = 0.95) -> None:
        """Compute Generalized Advantage Estimation"""
        rewards = [exp['reward'] for exp in self.rollout_buffer]
        values = [exp['value'] for exp in self.rollout_buffer] + [next_value]
        dones = [exp['done'] for exp in self.rollout_buffer]
        
        advantages = []
        gae = 0
        
        for i in reversed(range(len(rewards))):
            delta = rewards[i] + self.config.gamma * values[i + 1] * (1 - dones[i]) - values[i]
            gae = delta + self.config.gamma * gae_lambda * (1 - dones[i]) * gae
            advantages.insert(0, gae)
        
        returns = [adv + val for adv, val in zip(advantages, values[:-1])]
        
        # Store advantages and returns
        for i, exp in enumerate(self.rollout_buffer):
            exp['advantage'] = advantages[i]
            exp['return'] = returns[i]
    
    def train(self, epochs: int = 4) -> Dict[str, float]:
        """Train using PPO algorithm"""
        if not self.rollout_buffer:
            return {}
        
        # Convert to tensors
        states = torch.FloatTensor([exp['state'] for exp in self.rollout_buffer]).to(self.device)
        actions = torch.FloatTensor([exp['action'] for exp in self.rollout_buffer]).to(self.device)
        old_log_probs = torch.FloatTensor([exp['log_prob'] for exp in self.rollout_buffer]).to(self.device)
        advantages = torch.FloatTensor([exp['advantage'] for exp in self.rollout_buffer]).to(self.device)
        returns = torch.FloatTensor([exp['return'] for exp in self.rollout_buffer]).to(self.device)
        
        # Normalize advantages
        advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)
        
        total_policy_loss = 0
        total_value_loss = 0
        total_entropy_loss = 0
        
        for epoch in range(epochs):
            # Get current policy outputs
            _, log_probs, entropy, values = self.network.get_action_and_value(states, actions)
            
            # Policy loss with clipping
            ratio = torch.exp(log_probs - old_log_probs)
            surr1 = ratio * advantages
            surr2 = torch.clamp(ratio, 1 - self.clip_epsilon, 1 + self.clip_epsilon) * advantages
            policy_loss = -torch.min(surr1, surr2).mean()
            
            # Value loss
            value_loss = F.mse_loss(values.squeeze(), returns)
            
            # Entropy loss
            entropy_loss = -entropy.mean()
            
            # Total loss
            total_loss = policy_loss + self.value_coef * value_loss + self.entropy_coef * entropy_loss
            
            # Optimize
            self.optimizer.zero_grad()
            total_loss.backward()
            torch.nn.utils.clip_grad_norm_(self.network.parameters(), 0.5)
            self.optimizer.step()
            
            total_policy_loss += policy_loss.item()
            total_value_loss += value_loss.item()
            total_entropy_loss += entropy_loss.item()
        
        # Clear buffer
        self.rollout_buffer.clear()
        
        return {
            'policy_loss': total_policy_loss / epochs,
            'value_loss': total_value_loss / epochs,
            'entropy_loss': total_entropy_loss / epochs
        }

Twin Delayed Deep Deterministic Policy Gradient (TD3)

class TD3Agent:
    """Twin Delayed DDPG for continuous control"""
    
    def __init__(self, config: RLConfig):
        self.config = config
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        
        # Actor networks
        self.actor = self._build_actor().to(self.device)
        self.actor_target = self._build_actor().to(self.device)
        self.actor_target.load_state_dict(self.actor.state_dict())
        
        # Twin critic networks
        self.critic1 = self._build_critic().to(self.device)
        self.critic1_target = self._build_critic().to(self.device)
        self.critic1_target.load_state_dict(self.critic1.state_dict())
        
        self.critic2 = self._build_critic().to(self.device)
        self.critic2_target = self._build_critic().to(self.device)
        self.critic2_target.load_state_dict(self.critic2.state_dict())
        
        # Optimizers
        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=config.learning_rate)
        self.critic1_optimizer = optim.Adam(self.critic1.parameters(), lr=config.learning_rate)
        self.critic2_optimizer = optim.Adam(self.critic2.parameters(), lr=config.learning_rate)
        
        # Replay buffer
        self.memory = ReplayBuffer(config.buffer_size, config.state_dim, config.action_dim)
        
        # Training step counter
        self.training_step = 0
    
    def _build_actor(self) -> nn.Module:
        """Build actor network"""
        return nn.Sequential(
            nn.Linear(self.config.state_dim, self.config.hidden_dim),
            nn.ReLU(),
            nn.Linear(self.config.hidden_dim, self.config.hidden_dim),
            nn.ReLU(),
            nn.Linear(self.config.hidden_dim, self.config.action_dim),
            nn.Tanh()
        )
    
    def _build_critic(self) -> nn.Module:
        """Build critic network"""
        return nn.Sequential(
            nn.Linear(self.config.state_dim + self.config.action_dim, self.config.hidden_dim),
            nn.ReLU(),
            nn.Linear(self.config.hidden_dim, self.config.hidden_dim),
            nn.ReLU(),
            nn.Linear(self.config.hidden_dim, 1)
        )
    
    def select_action(self, state: np.ndarray, add_noise: bool = True) -> np.ndarray:
        """Select action using current policy"""
        state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
        action = self.actor(state_tensor).cpu().data.numpy().flatten()
        
        if add_noise:
            noise = np.random.normal(0, self.config.exploration_noise, size=self.config.action_dim)
            action = np.clip(action + noise, -1, 1)
        
        return action
    
    def store_experience(self, state: np.ndarray, action: np.ndarray, reward: float,
                        next_state: np.ndarray, done: bool) -> None:
        """Store experience in replay buffer"""
        self.memory.push(state, action, reward, next_state, done)
    
    def train(self) -> Dict[str, float]:
        """Train the TD3 agent"""
        if len(self.memory) < self.config.batch_size:
            return {}
        
        states, actions, rewards, next_states, dones = self.memory.sample(self.config.batch_size)
        
        states = states.to(self.device)
        actions = actions.to(self.device)
        rewards = rewards.to(self.device)
        next_states = next_states.to(self.device)
        dones = dones.to(self.device)
        
        with torch.no_grad():
            # Target policy smoothing
            noise = (torch.randn_like(actions) * self.config.policy_noise).clamp(
                -self.config.noise_clip, self.config.noise_clip
            )
            next_actions = (self.actor_target(next_states) + noise).clamp(-1, 1)
            
            # Twin Q-values
            target_q1 = self.critic1_target(torch.cat([next_states, next_actions], dim=1))
            target_q2 = self.critic2_target(torch.cat([next_states, next_actions], dim=1))
            target_q = torch.min(target_q1, target_q2)
            target_q = rewards + (self.config.gamma * target_q * ~dones)
        
        # Current Q-values
        current_q1 = self.critic1(torch.cat([states, actions], dim=1))
        current_q2 = self.critic2(torch.cat([states, actions], dim=1))
        
        # Critic losses
        critic1_loss = F.mse_loss(current_q1, target_q)
        critic2_loss = F.mse_loss(current_q2, target_q)
        
        # Update critics
        self.critic1_optimizer.zero_grad()
        critic1_loss.backward()
        self.critic1_optimizer.step()
        
        self.critic2_optimizer.zero_grad()
        critic2_loss.backward()
        self.critic2_optimizer.step()
        
        # Delayed policy updates
        actor_loss = 0
        if self.training_step % self.config.policy_freq == 0:
            # Actor loss
            actor_loss = -self.critic1(torch.cat([states, self.actor(states)], dim=1)).mean()
            
            # Update actor
            self.actor_optimizer.zero_grad()
            actor_loss.backward()
            self.actor_optimizer.step()
            
            # Soft update target networks
            self._soft_update(self.actor_target, self.actor)
            self._soft_update(self.critic1_target, self.critic1)
            self._soft_update(self.critic2_target, self.critic2)
        
        self.training_step += 1
        
        return {
            'critic1_loss': critic1_loss.item(),
            'critic2_loss': critic2_loss.item(),
            'actor_loss': actor_loss.item() if isinstance(actor_loss, torch.Tensor) else actor_loss
        }
    
    def _soft_update(self, target: nn.Module, source: nn.Module) -> None:
        """Soft update target network"""
        for target_param, param in zip(target.parameters(), source.parameters()):
            target_param.data.copy_(self.config.tau * param.data + (1 - self.config.tau) * target_param.data)

Multi-Agent Reinforcement Learning

class MultiAgentEnvironment:
    """Multi-agent environment wrapper"""
    
    def __init__(self, num_agents: int, state_dim: int, action_dim: int):
        self.num_agents = num_agents
        self.state_dim = state_dim
        self.action_dim = action_dim
        self.agents = []
    
    def reset(self) -> List[np.ndarray]:
        """Reset environment and return initial states"""
        return [np.random.randn(self.state_dim) for _ in range(self.num_agents)]
    
    def step(self, actions: List[np.ndarray]) -> Tuple[List[np.ndarray], List[float], List[bool]]:
        """Execute actions and return next states, rewards, and done flags"""
        # Simplified multi-agent dynamics
        next_states = [np.random.randn(self.state_dim) for _ in range(self.num_agents)]
        rewards = [np.random.randn() for _ in range(self.num_agents)]
        dones = [False for _ in range(self.num_agents)]
        
        return next_states, rewards, dones

class MADDPGAgent:
    """Multi-Agent Deep Deterministic Policy Gradient"""
    
    def __init__(self, agent_id: int, num_agents: int, config: RLConfig):
        self.agent_id = agent_id
        self.num_agents = num_agents
        self.config = config
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        
        # Individual actor
        self.actor = self._build_actor().to(self.device)
        self.actor_target = self._build_actor().to(self.device)
        self.actor_target.load_state_dict(self.actor.state_dict())
        
        # Centralized critic (observes all agents)
        total_state_dim = config.state_dim * num_agents
        total_action_dim = config.action_dim * num_agents
        
        self.critic = self._build_critic(total_state_dim, total_action_dim).to(self.device)
        self.critic_target = self._build_critic(total_state_dim, total_action_dim).to(self.device)
        self.critic_target.load_state_dict(self.critic.state_dict())
        
        # Optimizers
        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=config.learning_rate)
        self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=config.learning_rate)
        
        # Shared replay buffer
        self.memory = None  # Will be set externally
    
    def _build_actor(self) -> nn.Module:
        """Build individual actor network"""
        return nn.Sequential(
            nn.Linear(self.config.state_dim, self.config.hidden_dim),
            nn.ReLU(),
            nn.Linear(self.config.hidden_dim, self.config.hidden_dim),
            nn.ReLU(),
            nn.Linear(self.config.hidden_dim, self.config.action_dim),
            nn.Tanh()
        )
    
    def _build_critic(self, state_dim: int, action_dim: int) -> nn.Module:
        """Build centralized critic network"""
        return nn.Sequential(
            nn.Linear(state_dim + action_dim, self.config.hidden_dim),
            nn.ReLU(),
            nn.Linear(self.config.hidden_dim, self.config.hidden_dim),
            nn.ReLU(),
            nn.Linear(self.config.hidden_dim, 1)
        )
    
    def select_action(self, state: np.ndarray, add_noise: bool = True) -> np.ndarray:
        """Select action using individual policy"""
        state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
        action = self.actor(state_tensor).cpu().data.numpy().flatten()
        
        if add_noise:
            noise = np.random.normal(0, self.config.exploration_noise, size=self.config.action_dim)
            action = np.clip(action + noise, -1, 1)
        
        return action

Advanced Techniques and Optimizations

Curiosity-Driven Learning

class CuriosityModule(nn.Module):
    """Intrinsic Curiosity Module for exploration"""
    
    def __init__(self, state_dim: int, action_dim: int, hidden_dim: int = 256):
        super().__init__()
        
        # Forward model: predicts next state given current state and action
        self.forward_model = nn.Sequential(
            nn.Linear(state_dim + action_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, state_dim)
        )
        
        # Inverse model: predicts action given current and next state
        self.inverse_model = nn.Sequential(
            nn.Linear(state_dim * 2, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )
        
        # Feature encoder
        self.feature_encoder = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim // 2)
        )
    
    def forward(self, state: torch.Tensor, action: torch.Tensor, 
                next_state: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
        # Encode states
        state_features = self.feature_encoder(state)
        next_state_features = self.feature_encoder(next_state)
        
        # Forward model prediction
        predicted_next_features = self.forward_model(torch.cat([state_features, action], dim=1))
        
        # Inverse model prediction
        predicted_action = self.inverse_model(torch.cat([state_features, next_state_features], dim=1))
        
        # Intrinsic reward (prediction error)
        intrinsic_reward = F.mse_loss(predicted_next_features, next_state_features, reduction='none').mean(dim=1)
        
        return predicted_next_features, predicted_action, intrinsic_reward

Key Benefits and Applications

Performance Advantages

  • Autonomous Decision Making: Optimal policies without human intervention
  • Adaptability: Learning from environmental changes
  • Sample Efficiency: Modern algorithms require 50-80% less data
  • Generalization: Transfer learning across similar environments

Real-world Applications

  1. Robotics: Manipulation, navigation, and control
  2. Autonomous Vehicles: Path planning and decision making
  3. Game AI: Strategic gameplay and opponent modeling
  4. Finance: Algorithmic trading and portfolio optimization
  5. Healthcare: Treatment optimization and drug discovery

Implementation Best Practices

Algorithm Selection

  • Discrete Actions: DQN, Rainbow DQN, A3C
  • Continuous Control: PPO, TD3, SAC
  • Multi-Agent: MADDPG, QMIX, MAPPO
  • Sparse Rewards: Curiosity-driven methods, HER

Training Optimization

  • Hyperparameter Tuning: Use automated search methods
  • Curriculum Learning: Gradually increase task difficulty
  • Regularization: Prevent overfitting with dropout and weight decay
  • Distributed Training: Scale across multiple workers

Future Directions

Emerging Techniques

  • Model-Based RL: Learning environment dynamics
  • Meta-Learning: Learning to learn new tasks quickly
  • Hierarchical RL: Decomposing complex tasks
  • Safe RL: Ensuring safety constraints during learning

Research Frontiers

  • Quantum RL: Leveraging quantum computing advantages
  • Neuromorphic RL: Brain-inspired computing architectures
  • Federated RL: Distributed learning across devices
  • Explainable RL: Understanding agent decision processes

Conclusion

Reinforcement Learning has evolved from theoretical foundations to practical solutions that power some of today's most advanced AI systems. By mastering advanced techniques like deep Q-learning, policy gradients, and multi-agent systems, practitioners can develop agents capable of solving complex real-world problems with superhuman performance.

The future of RL lies in developing more sample-efficient algorithms, ensuring safety and robustness, and creating agents that can rapidly adapt to new environments and tasks.

References

  1. Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
  2. Mnih, V., et al. (2015). Human-level control through deep reinforcement learning. nature, 518(7540), 529-533.
  3. Schulman, J., et al. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
  4. Fujimoto, S., et al. (2018). Addressing function approximation error in actor-critic methods. International conference on machine learning.
  5. Lowe, R., et al. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems.
Reinforcement LearningDeep Q-LearningPolicy GradientsMulti-Agent SystemsPPO
CX

Cognixa Labs Research Team

Independent AI research and benchmarking

The collective byline for benchmark releases, methodology notes, and research writing from Cognixa Labs.

Keep riding

Related from the archive