alt Hacker NewsIt's funny that we've finally returned to tanh activation functions, time is a circle.
Replies
pinkmuffinere • today at 4:44 PM
Wow this is fascinating enough that I’m actually going to read tfa lol
It's funny that we've finally returned to tanh activation functions, time is a circle.
Wow this is fascinating enough that I’m actually going to read tfa lol
From the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but:
Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).