Skip to content

backward() for ActivationFunction#123

Merged
grantbruer merged 2 commits intoslimgroup:masterfrom
gbruer-forks:gbruer/activation-backward
Oct 1, 2025
Merged

backward() for ActivationFunction#123
grantbruer merged 2 commits intoslimgroup:masterfrom
gbruer-forks:gbruer/activation-backward

Conversation

@grantbruer
Copy link
Copy Markdown
Contributor

  1. Added backward() for ActivationFunction. It calls the user-defined backward function using the output for invertible functions and the input for non-invertible functions.
  2. Fixed ExpClampLayer activation. The forward computation didn't match the inverse and gradient.
  3. Updated some failing gradient tests to use sigmoid so that they don't rely on the gradient of relu at x = 0.

… doesn't need to know if the function is invertible. Fixed ExpClampLayer activation.
@grantbruer
Copy link
Copy Markdown
Contributor Author

The previous limitations were as follows:

  • ResidualBlock requires non-invertible activations.
  • CouplingLayerBasic requires invertible activation.
  • CouplingLayerGlow requires invertible activation.
  • ConditionalLayerGlow requires invertible activation.

With this PR, each layer should be able to use arbitrary activation functions, with the stipulation that the coupling layers need positive activations.

In this PR, I didn't add tests to make sure these work correctly, although I have run more tests in a different PR to make sure these changes are correct. If someone else wants to add commits with unit tests for these behaviors, please do. I don't think I'll have time for a couple months.

@grantbruer grantbruer mentioned this pull request Sep 21, 2025
@grantbruer grantbruer force-pushed the gbruer/activation-backward branch from e3a764a to ead64d9 Compare September 30, 2025 22:00
@grantbruer grantbruer merged commit f3c9b02 into slimgroup:master Oct 1, 2025
49 of 54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant