Add envpool support

[araffin]

Description

Motivation and Context

• I have raised an issue to propose this change (required for new features and bug fixes)

closes #241

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

[qgallouedec]

For DeepMind Control Suite, here are the hyperparam I understand from the paper, and that could be default:

AcrobotSwingup-v1: &dmcs-defaults
  policy: 'MlpPolicy'
  n_timesteps: !!float 1e8
  batch_size: 64
  policy_kwargs: "dict(net_arch=dict(pi=[300, 200], qf=[400, 300]))" 
  learning_rate: !!float 1e-4
  gamma: 0.99
  noise_type: 'ornstein-uhlenbeck'
  noise_std: 0.3

[araffin]

mmh 1e8 sounds a bit too much for me (i would prefer to stick to 1e6), and i would try mujoco default first.
Let me try to find more recent paper later.

[qgallouedec]

I agree, most tasks can be solved with much less interaction.

[qgallouedec]

	Hyperparam	Train steps
Fujimoto et al. (2018)	Tuned (but not specifically for DMCS), see Sup. Mat	1e6
Zhang et al. (2019)	Lillicrap et al. (2016)	1e6
Bhatt et al. (2019)	"we used the well tuned “OurDDPG" and TD3 code published by (Fujimoto et al., 2018)"	1e6
Kuznetsov et al. (2022)	Fujimoto et al. (2018)	1e6

Fujimoto et al. (2018) is actually the &mujoco-defaults (hyperparams/ddpg.yml)

[araffin]

Side note: mypy now hangs with python 3.7 for some reasons... (doesn't happen with other versions) and I cannot reproduce locally...

[araffin]

@qgallouedec same as openai/gym#3176, no?

pip3 install setuptools==65.5.0 should fix it

[araffin]

Closing in favor of #355