dreamerv3-torch

Pytorch implementation of Mastering Diverse Domains through World Models. DreamerV3 is a scalable algorithm that outperforms previous approaches across various domains with fixed hyperparameters. This repository builds on NM512/dreamerv3-torch and extends it with first-class Meta-World simulator training support.

Instructions

Method 1: Manual

Get dependencies with python 3.11:

pip install -r requirements.txt

Run training on DMC Vision:

python3 dreamer.py --configs dmc_vision --task dmc_walker_walk --logdir ./logdir/dmc_walker_walk

Run training on Metaworld:

python3 dreamer.py --configs metaworld_vision --task metaworld_reach-v3 --logdir ./logdir/metaworld_reach

See METAWORLD_GUIDE.md for detailed Metaworld integration documentation.

Monitor results:

tensorboard --logdir ./logdir

To set up Atari or Minecraft environments, please check the scripts located in env/setup_scripts.

Meta-World Support

Meta-World support from fangzr/DreamerV3-Meta-world has been upstreamed into this codebase so you can train the PyTorch DreamerV3 agent directly inside the Meta-World simulator without leaving this repository.

Quickstart

1. Activate the environment (e.g. conda activate dreamerv3) and move into the repo root.
2. Launch training with the Meta-World configs:

   python dreamer.py \
     --configs metaworld_vision \
     --task metaworld_reach-v3 \
     --logdir ./logdir/metaworld_reach

3. All task names must start with the metaworld_ prefix:

Incorrect	Correct
--task reach-v3	--task metaworld_reach-v3
--task basketball	--task metaworld_basketball-v3

4. Monitor training with TensorBoard:

   tensorboard --logdir ./logdir --port 6006

   Scalars show performance, while train_openl and eval_openl streams display imagined rollouts. Run multiple boards on different ports to compare experiments, or use watch -n 1 nvidia-smi for real-time GPU stats.

Available Configurations

Config name	GPU usage	Speed	Description
metaworld_vision	~80%	Standard	Default 64×64 visual encoder + video logging
metaworld_vision_fast	~90%	Fast	Larger batches/env count for high-end GPUs
metaworld_proprio	~80%	Very fast	State-only training (no image encoder)

Use the fast profile when you have spare GPU memory, or the proprio-only profile for rapid debugging without rendering.

Command Template and Common Flags

python dreamer.py \
  --configs <metaworld_vision|metaworld_vision_fast|metaworld_proprio> \
  --task metaworld_<task name>-v3 \
  --logdir ./logdir/<experiment> \
  [--envs 8] \
  [--batch_size 32] \
  [--steps 1000000] \
  [--eval_every 10000] \
  [--device cuda:0] \
  [--seed 0]

Tips:

• Increase --envs (8–16) and --batch_size (32–48) to improve GPU utilization.
• If you hit memory limits, reduce the batch size or choose metaworld_proprio.
• Training resumes automatically from checkpoints stored under --logdir, so rerun the same command to continue interrupted jobs.

Example Commands

# Entry-level reach task with vision encoder
python dreamer.py --configs metaworld_vision --task metaworld_reach-v3 --logdir ./logdir/reach

# High-performance push training
python dreamer.py --configs metaworld_vision_fast --task metaworld_push-v3 --logdir ./logdir/push --envs 16 --batch_size 48

# Door open with custom budget
python dreamer.py --configs metaworld_vision --task metaworld_door-open-v3 --logdir ./logdir/door_open --steps 2000000

# State-only pick-and-place
python dreamer.py --configs metaworld_proprio --task metaworld_pick-place-v3 --logdir ./logdir/pick_place_state

Task Recommendations

• Warm-up: reach-v3, push-v3, button-press-v3.
• Intermediate: pick-place-v3, door-open-v3, drawer-open-v3.
• Advanced: assembly-v3, basketball-v3, peg-insert-side-v3.

When specifying a task on the CLI, always prepend metaworld_ (for example --task metaworld_push-v3). A complete list of the 50 supported tasks is available in METAWORLD_GUIDE.md, covering manipulation skills such as drawers, doors, buttons, and peg insertions.

Troubleshooting

• Low GPU usage? Switch to metaworld_vision_fast and raise --envs/--batch_size.
• Training slow? Use the proprio config or shorten --steps for quick validation runs.
• No rollout videos? Ensure video_pred_log: true in the config (default) and wait for training to pass 10k steps before checking TensorBoard.

See METAWORLD_GUIDE.md for the full integration guide, including environment-specific tuning tips and a categorized task list.

Method 2: Docker

Please refer to the Dockerfile for the instructions, as they are included within.

Benchmarks

So far, the following benchmarks can be used for testing.

Environment	Observation	Action	Budget	Description
DMC Proprio	State	Continuous	500K	DeepMind Control Suite with low-dimensional inputs.
DMC Vision	Image	Continuous	1M	DeepMind Control Suite with high-dimensional images inputs.
Metaworld	State or Image	Continuous	500K-1M	50 robotic manipulation tasks for multi-task RL.
Atari 100k	Image	Discrete	400K	26 Atari games.
Crafter	Image	Discrete	1M	Survival environment to evaluates diverse agent abilities.
Minecraft	Image and State	Discrete	100M	Vast 3D open world.
Memory Maze	Image	Discrete	100M	3D mazes to evaluate RL agents' long-term memory.

Results

DMC Proprio

DMC Vision

Atari 100k

Crafter

Acknowledgments

This code is heavily inspired by the following works:

• danijar's Dreamer-v3 jax implementation: https://github.com/danijar/dreamerv3
• danijar's Dreamer-v2 tensorflow implementation: https://github.com/danijar/dreamerv2
• jsikyoon's Dreamer-v2 pytorch implementation: https://github.com/jsikyoon/dreamer-torch
• RajGhugare19's Dreamer-v2 pytorch implementation: https://github.com/RajGhugare19/dreamerv2
• denisyarats's DrQ-v2 original implementation: https://github.com/facebookresearch/drqv2