[New Model] Reverso Foundation Model

[daidahao]

This is a sub-issue of #2933.

Is your feature request related to a current problem? Please describe.
Reverso is a new super-lightweight forecasting foundation model that combines convolutions, MLP, and linear RNN. With only 3M parameters, it matches the accuracy of foundation models 100x of its size.

Supporting Reverso in Darts could bring those accuracy and efficiency benefits to Darts users. I've discussed the possibility shinfxh/reverso#1 of reference implementation in Darts from Reverso team (@shinfxh). They have kindly accepted and even quickly (!) provided torch-native implementation to address concerns on hardware constraints and platform compatibility.

I am starting the issue here to align on API design between Reverso team (@shinfxh) and Darts team (@dennisbader, but not in office this week) before Reverso team would contribute a PR. We also host discussions here for any concerns, clarification, testing needed.

Describe proposed solution
A PyTorch-native port of Reverso in Darts using FoundationModel base class and HuggingFaceConnector.

Describe potential alternatives
A clear and concise description of any alternative solutions or existing features that might solve it.

Additional context

• Reverso code and weights are released under MIT license, compatible with Darts' Apache 2.0. We should include and display the license accordingly for the ported portion.
• Fine-tuning foundation models is on the horizon Enhance fine-tuning capabilities for foundation models #3003 (beautiful work by @Kurokabe) and we could consider supporting it for Reverso too, given how lightweight the model is.

[daidahao]

@dennisbader
Any thoughts on this?

[dennisbader]

Thanks a lot @daidahao for driving this. Very nice also to see from the reverso team that they implemented a torch native solution. For me it sounds good to add reverso as a new foundation model to Darts :)

Regarding MIT license, that should be compatible with our Apache 2, as long as we include the license in the model file.

I do have some questions that I'd like to clarify before starting:

• Are all three model variations available on HuggingFace? I can only find the checkpoint for the "small" version (see here)
• Would the model integration look similar to TiRex where we add reverso as an optional dependency and rely on their implementation?
• Which of the following does the model support?
  • Uni / multivariate target
  • Past / future / static covariates
  • Quantile forecasts

[daidahao]

Hi @dennisbader,

To answer your questions (while we await response from @shinfxh):

• Are all three model variations available on HuggingFace? I can only find the checkpoint for the "small" version (see here)

Only Reverso small (550K) is available as of now. Reverso (2.6M) and Reverso nano (200K) are marked as "available soon" on the repo.

• Would the model integration look similar to TiRex where we add reverso as an optional dependency and rely on their implementation?

Because Reverso is not available on PyPI and conda-forge and has dependencies on flash-linear-attention, I would strongly recommend porting only the torch-native impl. reverso_torch to Darts. This is similar to what we did for Chronos-2 and TimesFM 2.5--no optional dependencies on PyPI packages, only torch-native impl.

Which of the following does the model support?

• Uni / multivariate target
• Past / future / static covariates
• Quantile forecasts

Based on my understanding of Reverso and codebase, the model only supports univariate target and no covariates. Quantile forecasts are not supported. However, for multivariate target, I suggest we can assume channel (component) independence, and fold the component dimension into batch dimension. That way, multivariate forecasting is being supported. For context, all torch models in Darts support multivariate forecasting.

@shinfxh
Finally, because I have limited bandwidth at the moment, I would really appreciate it if you could contribute the PR directly. I am happy to provide code edits and/or review once you have an initial draft. BTW, this is also what we are doing now to help bring TiRex to the finish line. What do you think?

[shinfxh]

This sounds good! Just some quick replies

• Reverso-Nano and Reverso will be added within the next 1-2 weeks!
• We can go with the toch-native implementation first, but note that inference is slower without additional dependencies.
• Reverso currently only support univariate point forecasting.

Thanks @daidahao for organizing this, will update again when the PR is ready.

[daidahao]

Hi, @shinfxh It's been a while! Is there anything we can do to help with the PR?

[shinfxh]

Thanks! Was just running some more extensive benchmarking earlier this week. Also just added support for Reverso and Reverso-Nano. Should just a a few days before I update with the PR :)

[shinfxh]

Hi, the PR is created. The chunked parallel form on GPU is implemented which should give some speedup. Please let me know if anything needs fixing! @daidahao @dennisbader