backward() for ActivationFunction#123
Merged
grantbruer merged 2 commits intoslimgroup:masterfrom Oct 1, 2025
Merged
Conversation
Contributor
grantbruer
commented
Sep 21, 2025
- Added backward() for ActivationFunction. It calls the user-defined backward function using the output for invertible functions and the input for non-invertible functions.
- Fixed ExpClampLayer activation. The forward computation didn't match the inverse and gradient.
- Updated some failing gradient tests to use sigmoid so that they don't rely on the gradient of relu at x = 0.
… doesn't need to know if the function is invertible. Fixed ExpClampLayer activation.
Contributor
Author
|
The previous limitations were as follows:
With this PR, each layer should be able to use arbitrary activation functions, with the stipulation that the coupling layers need positive activations. In this PR, I didn't add tests to make sure these work correctly, although I have run more tests in a different PR to make sure these changes are correct. If someone else wants to add commits with unit tests for these behaviors, please do. I don't think I'll have time for a couple months. |
Open
…better-conditioned gradient than sigmoid.
e3a764a to
ead64d9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.