Skip to content

Report.md

Latest commit

 

History

History
117 lines (87 loc) · 5.2 KB

Report.md

File metadata and controls

117 lines (87 loc) · 5.2 KB
Raw

Collaboration and Competition - Report

Implementation

This project implements a Deep Deterministic Policy Gradient (DDPG) agent to solve the Unity ML-Agents Tennis environment. The environment is considered solved when the agents achieve an average score of +0.5 over 100 consecutive episodes (taking the maximum score across both agents per episode).

Multi-Agent Strategy

A shared agent approach was adopted where both tennis players use the same actor and critic networks. This design choice offers several advantages:

  • Faster learning: Both agents contribute experiences to a shared replay buffer, effectively doubling the learning samples
  • Simpler implementation: Single set of networks to train and maintain
  • Cooperative learning: Both agents learn the same optimal policy for keeping the ball in play
  • Independent exploration: Each agent explores independently using distinct Gaussian noise

This approach works well for the Tennis environment since it's primarily a cooperative task where both agents have the same goal (keeping the ball in play) despite being in opposite positions.

Learning Algorithm

DDPG (Deep Deterministic Policy Gradient)

DDPG is an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. It combines ideas from DPG (Deterministic Policy Gradient) and DQN (Deep Q-Network).

Key Components:

  1. Actor Network: Approximates the optimal policy deterministically, mapping states to actions

    • Input: State (24 dimensions)
    • Output: Action (2 dimensions, continuous)

  2. Critic Network: Learns to evaluate the optimal action-value function using the actor's best believed action

    • Input: State (24 dimensions) + Action (2 dimensions)
    • Output: Q-value (scalar)

  3. Target Networks: Slow-moving copies of actor and critic networks for stable learning

    • Soft updates with τ = 0.001

  4. Experience Replay: Stores transitions (s, a, r, s', done) in a replay buffer

    • Buffer size: 300,000 transitions
    • Random sampling breaks correlation between consecutive samples

  5. Ornstein-Uhlenbeck Noise: Adds temporally correlated exploration noise

    • For Tennis, Gaussian noise was used for independent agent exploration

Network Architecture

Actor Network:

State (24) → FC1 (128) → ReLU → FC2 (64) → ReLU → FC3 (2) → Tanh
  • Input layer: 24 units (state size)
  • Hidden layer 1: 128 units with ReLU activation
  • Hidden layer 2: 64 units with ReLU activation
  • Output layer: 2 units with Tanh activation (action size, bounded [-1, 1])

Critic Network:

State (24) → FC1 (128) → ReLU → Concat(FC1_output, Action(2)) → FC2 (64) → ReLU → FC3 (1)
  • Input layer: 24 units (state size)
  • Hidden layer 1: 128 units with ReLU activation
  • Action concatenation at layer 2: 130 units (128 + 2)
  • Hidden layer 2: 64 units with ReLU activation
  • Output layer: 1 unit (Q-value)

Weight Initialization:

  • Hidden layers: Uniform distribution based on fan-in ([-1/√f, 1/√f])
  • Output layers: Uniform distribution ([-3×10⁻³, 3×10⁻³])

Hyperparameters

Parameter Value Description
BUFFER_SIZE 300,000 Replay buffer size
BATCH_SIZE 128 Minibatch size for training
GAMMA (γ) 0.99 Discount factor for future rewards
TAU (τ) 0.001 Soft update parameter for target networks
LR_ACTOR 0.0001 Learning rate for actor network
LR_CRITIC 0.001 Learning rate for critic network
WEIGHT_DECAY 0.00001 L2 weight decay for critic optimizer
UPDATE_EVERY 1 How often to perform learning updates (timesteps)
NUM_UPDATES 2 Number of learning updates per timestep
WARMUP_STEPS 1000 Random actions before learning starts
NOISE_SIGMA 0.3 → 0.05 Exploration noise standard deviation
NOISE_DECAY 0.995 Multiplicative factor for noise decay per episode

Training-specific parameters:

  • Maximum episodes: 2000
  • Maximum timesteps per episode: 1000
  • Noise minimum threshold: 0.05

Training Results

The agent successfully solved the environment in 690 episodes, achieving an average score of 0.5158 over 100 consecutive episodes.

Training Progress

Training Progress

Training Characteristics

  • Warmup phase: Collected 1000 transitions with random actions to initialize the replay buffer
  • Exploration strategy: Independent Gaussian noise per agent, decaying from σ=0.3 to σ=0.05
  • Learning frequency: 2 gradient updates per environment timestep (4 total per step for both agents)
  • Episode termination: When ball hits ground for either agent (using np.any(dones))
  • Scoring: Maximum score across both agents per episode (Tennis competition rule)

Ideas for Future Work

1. Alternative Algorithms

Compare DDPG performance with:

  • PPO (Proximal Policy Optimization): More stable, better for multi-agent
  • A3C/A2C: Asynchronous advantage actor-critic methods

2. Competitive Training (Self-Play)

Extend to fully competitive scenario:

  • Train agents with opposing objectives
  • Implement self-play where agents compete against past versions
  • Could lead to more robust and sophisticated strategies