Replaced manual gating/blending with Gating/BlendingActivation classes.#192
Replaced manual gating/blending with Gating/BlendingActivation classes.#192jfsantos wants to merge 2 commits intosdatkinson:mainfrom
Conversation
There was a problem hiding this comment.
Few things I might prefer done a little differently.
Mainly:
- Leave backward compatibility only at the factory, remove it from the constructors so there's "only one way to do it."
- Remove default "Sigmoid secondary activations, requiring them to be specified as well.
Honestly, these are a bit "nit"ty but I think that removing the old code paths now will maybe be worth it with a bit more clarity, focusing the backward compatibility into one spot.
I'll take care of 'em and get this merged here
| const std::string blending_activation; | ||
|
|
||
| // Backward compatibility constructor with boolean gated parameter | ||
| LayerArrayParams(const int input_size_, const int condition_size_, const int head_size_, const int channels_, |
There was a problem hiding this comment.
I'm gonna refactor this so that all of the backward compatibility is in the factory, then all the constructors below that can be nice & fresh.
| // Use the GatingActivation class | ||
| // Extract the blocks first to avoid temporary reference issues | ||
| auto input_block = this->_z.leftCols(num_frames); | ||
| auto output_block = this->_z.topRows(bottleneck).leftCols(num_frames); |
There was a problem hiding this comment.
This is a bit tricky to me...so the bottom rows are for the gate/blend op, and the top are both for the main activation's input and the gating activation's output.
I'll make a note of this.
|
Continuing this on #193 |
2 participants
This PR replaces the manual gating implementation in wavenet.cpp by either a GatingActivation or a BlendingActivation instance and makes those configurable.
I implemented backwards compatibility with older config files that expect gated to be boolean, and also implemented backwards-compatible constructors that can take gated as a boolean instead of the enum I created. We can remove those if not needed, but they are here for completeness.
Developed with support and sponsorship from TONE3000