[Feature Request] Plug in feature extractors for Actor / Critic

[walid-abouir]

Required prerequisites

• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

Motivation

Thank you for the Safe RL library, and the test environments, it is very useful. However, OmniSafe currently hard-codes the observation encoder inside each Actor / Critic.
As a consequence, users cannot easily plug sequence encoders such as CNN or LSTM without forking the code.

Stable-Baselines3 solved this with a generic BaseFeaturesEncoder API that feeds its output into the policy / value network.

I would like to offer the same flexibility in Omnisafe while preserving the compatibility with what's been done so far.

Thank you for your time and consideration. I look forward to your feedback.

Solution

Create a generic feature extractor, so any extractor returns a fixed-size vector that flows into the existing actor/critic head.

Alternatives

No response

Additional context

I will be offline from May 27th until June 3rd but will resume immediately afterwards.