Skip to main content

Smooth gates: softplus, GELU and SiLU

Softplus is a ReLU with the kink rounded off, and its derivative is the sigmoid. SiLU and GELU multiply z by a soft gate instead of a hard one, dip just below zero, and are the activations of modern transformers. Each derived, minimised and coded.
DL01.3.3L2 Intermediate25 min
In review

The maths and code on this page are checked by automated tests on every change, but an independent reader has not signed it off yet. Spot a mistake or a confusing step? Tell us.

Before this lesson: DL01.3.2 ReLU and its relatives

What you will be able to do
  • Derive that the derivative of softplus is the sigmoid, and bound how far softplus lies above ReLU.
  • Write ReLU, SiLU and GELU as z times a gate, and derive the derivatives of SiLU and GELU.
  • Locate the minima of SiLU and GELU numerically and explain why both functions are not monotone.
  • Implement the three activations from scratch with scipy.special.erf, and measure the error of the tanh approximation of GELU.

The question

BERT, one of the first large transformer language models, replaced ReLU by a function called GELU, which is ReLU with its corner sanded smooth and a small dip below zero. What is that function, where exactly is its dip, and what does a hidden unit gain from it on the backward pass?

From the last lesson

1. What value do libraries use for the derivative of ReLU at exactly \({z = 0}\)?

Show answer

0, the subgradient of smallest size; PyTorch’s rule for convex functions picks it.

2. Why does a first-layer ReLU unit that is negative on every training input never recover?

Show answer

Its slope is 0 on every example, so the gradients for all its weights and its bias are exactly zero, and gradient descent never changes them.

Intuition

ReLU’s corner and its shut gate are its two weak points. Three smooth activations keep ReLU’s shape far from zero and change it only near zero.

Two panels. Left, titled Softplus: a dashed grey ReLU hinge and a smooth vermillion curve ln of 1 plus e to the z that runs just above it, joining it far from zero; at z equals 0 a black segment of height ln 2 marks how far softplus sits above the kink. Right, titled SiLU and GELU: the dashed ReLU, a solid vermillion SiLU curve and a dashed vermillion GELU curve. Both dip below zero for negative z before rising; a solid dot marks the SiLU minimum, minus 0.278, near z equals minus 1.28, and a hollow dot marks the GELU minimum, minus 0.170, near z equals minus 0.75. For large positive z both follow the line a equals z.
DL01.3.3-F1

Softplus rounds off the kink and stays above ReLU. SiLU and GELU also round it off, but dip a little below zero before rising.

Softplus is the smooth curve closest in spirit to ReLU: always positive, always rising. SiLU (also called Swish) and GELU take another route. They keep the idea of a gate, but let it open gradually. The price is a small dip below zero; the gain is that a slightly negative input still sends a gradient back.

Build it up

Step 1. ReLU as a gate. Write \({\operatorname{relu}(z) = z \cdot 1[z > 0]}\), where \({1[z > 0]}\) is 1 when the condition holds and 0 otherwise. The input \(z\) passes, multiplied by a gate that is either open or shut.

Step 2. Soften the gate. Replace the step by a smooth curve that rises from 0 to 1. With the sigmoid as gate, \({z\,\sigma(z)}\) is SiLU. With the standard normal cumulative distribution function \(\Phi\) as gate, \({z\,\Phi(z)}\) is GELU. \(\Phi(z)\) is the probability that a standard normal variable is at most \(z\); it rises from 0 to 1 like the sigmoid, a little more steeply.

A plot titled The gate that multiplies z, for z from minus 4 to 4. A solid vermillion step at 0 for negative z and 1 for positive z is labelled step: ReLU equals z times the indicator of z greater than 0. A dashed vermillion S-curve is labelled sigma of z: SiLU equals z sigma of z. A dotted vermillion S-curve, slightly steeper, is labelled Phi of z: GELU equals z Phi of z. All three gates pass through one half at z equals 0, except the step, which jumps there.
DL01.3.3-F2

Three gates. ReLU’s snaps shut at 0; SiLU’s and GELU’s close smoothly, GELU’s a little faster.

Step 3. Why a dip. For a negative \(z\) the soft gate is small but not zero, so \({z \times \text{gate}}\) is a small negative number. Far to the left the gate dies away faster than \(z\) grows, and the product returns to 0. In between it must reach a lowest point.

Step 4. Softplus differently. Softplus is not a gate times \(z\). Its derivative is a gate: softplus is the function whose slope is \({\sigma(z)}\), so it climbs from slope 0 to slope 1 without ever taking a corner.

The formal statement

NoteSoftplus, SiLU and GELU

\[ \begin{aligned} \operatorname{softplus}(z) &= \ln(1 + e^{z}), \\ \operatorname{silu}(z) &= z\,\sigma(z), \\ \operatorname{gelu}(z) &= z\,\Phi(z), \end{aligned} \]

where \({\Phi(z) = \tfrac12\big(1 + \operatorname{erf}(z/\sqrt2)\big)}\), with derivative \({\varphi(z) = e^{-z^2/2}/\sqrt{2\pi}}\), the standard normal density.

Softplus. By the chain rule,

\[ \begin{aligned} \frac{d}{dz}\ln(1 + e^{z}) &= \frac{e^{z}}{1 + e^{z}} \\ &= \frac{1}{e^{-z} + 1} = \sigma(z) . \end{aligned} \]

The middle step divides top and bottom by \(e^{z}\). To compare with ReLU, write \({1 + e^{z} = e^{\max(0, z)}\,(1 + e^{-\abs{z}})}\), which holds for either sign of \(z\). Taking logs,

\[ \begin{aligned} \operatorname{softplus}(z) &= \operatorname{relu}(z) \\ &\quad + \ln\big(1 + e^{-\abs{z}}\big) . \end{aligned} \]

The gap \({\ln(1 + e^{-\abs{z}})}\) is largest, \({\ln 2 \approx 0.693}\), at \({z = 0}\), and it shrinks like \({e^{-\abs{z}}}\) on both sides. This form is also the stable way to compute softplus.

SiLU’s derivative and minimum. By the product rule and \({\sigma' = \sigma(1 - \sigma)}\),

\[ \begin{aligned} \operatorname{silu}'(z) &= \sigma(z) + z\,\sigma(z)\big(1 - \sigma(z)\big) \\ &= \sigma(z)\,\big(1 + z\,(1 - \sigma(z))\big) . \end{aligned} \]

Since \({\sigma(z) > 0}\), the slope is zero exactly where \({1 + z(1 - \sigma(z)) = 0}\). That needs \({z < 0}\), and has no solution in elementary functions; solving it numerically gives \({z^\ast \approx -1.278}\). At that point \({\sigma(z^\ast) = 1 + 1/z^\ast}\), so the lowest value is

\[ \begin{aligned} \operatorname{silu}(z^\ast) &= z^\ast\big(1 + 1/z^\ast\big) \\ &= z^\ast + 1 \approx -0.278 . \end{aligned} \]

For \({z < z^\ast}\) the slope is negative and the function falls; after \(z^\ast\) it rises. So SiLU is not monotone, unlike every activation of the two lessons before.

GELU’s derivative and minimum. Since \({\Phi' = \varphi}\),

\[ \operatorname{gelu}'(z) = \Phi(z) + z\,\varphi(z) . \]

It is zero where \({\Phi(z) = -z\,\varphi(z)}\), which again needs \({z < 0}\); numerically \({z^\ast \approx -0.752}\), and there

\[ \begin{aligned} \operatorname{gelu}(z^\ast) &= z^\ast\,\Phi(z^\ast) \\ &= -(z^\ast)^2\,\varphi(z^\ast) \approx -0.170 . \end{aligned} \]

GELU’s dip is shallower than SiLU’s and closer to zero.

One identity, both functions. Each gate \(g\) satisfies \({g(z) + g(-z) = 1}\) (for the sigmoid, \({\sigma(-z) = 1 - \sigma(z)}\); for \(\Phi\), by the symmetry of the normal density). Then for \({f(z) = z\,g(z)}\),

\[ \begin{aligned} f(z) - f(-z) &= z\,g(z) + z\,g(-z) \\ &= z . \end{aligned} \]

Differentiating gives \({f'(z) + f'(-z) = 1}\): the slope at \(-z\) is 1 minus the slope at \(z\). So the negative dip of each slope below zero mirrors an overshoot above 1. GELU’s slope is lowest at \({z = -\sqrt2}\), where its own derivative \({\varphi(z)\,(2 - z^2)}\) is zero, and equals \({\Phi(-\sqrt2) - \sqrt2\,\varphi(\sqrt2) \approx -0.129}\); it peaks at \({1.129}\) at \({z = \sqrt2}\).

A plot titled Slopes of the smooth gates, for z from minus 5 to 4. A dashed grey step at 0 and 1 is labelled ReLU prime. A dotted green S-curve rising from 0 to 1 is labelled softplus prime equals sigma. A solid green curve, SiLU prime, and a dashed green curve, GELU prime, both pass through one half at z equals 0, dip slightly below zero for negative z, and rise above 1 for positive z before settling back to 1. A dot marks the lowest point of GELU prime, labelled GELU prime min minus 0.129 at minus root 2.
DL01.3.3-F5

The slopes. Softplus’s slope is the sigmoid, between 0 and 1. The slopes of SiLU and GELU go a little below 0 on the left and above 1 on the right, by the same amounts.

A faster GELU. Hendrycks and Gimpel give two approximations that avoid \(\operatorname{erf}\):

\[ \begin{aligned} \operatorname{gelu}(z) &\approx \tfrac12 z\big(1 + \tanh u\big), \\ u &= \sqrt{2/\pi}\,\big(z + 0.044715\,z^3\big), \end{aligned} \]

and \({z\,\sigma(1.702\,z)}\), a SiLU with a steeper gate. Measured on a fine grid, the tanh form is never more than about \({4.7 \times 10^{-4}}\) from the exact GELU, the sigmoid form about \({0.020}\).

Two panels of error curves against z from minus 6 to 6. Left, Tanh approximation, error in units of 10 to the minus 4: a symmetric reddish-purple curve with two peaks of about 4.7 at z equals plus and minus 2.70 and smaller negative lobes near z equals plus and minus 1.3, labelled largest 0.00047 at z equals plus or minus 2.70. Right, Sigmoid approximation, in units of 10 to the minus 2: a symmetric curve with two troughs of about minus 2 at z equals plus and minus 2.27 and small positive bumps near zero, labelled largest 0.0203 at z equals plus or minus 2.27.
DL01.3.3-F6

The error of each approximation. Mind the scales: the tanh form is about 40 times closer.

Why transformers use them. The case is empirical, not a theorem. Hendrycks and Gimpel found GELU ahead of ReLU and ELU on every vision, language and speech task they tried; BERT adopted it, following GPT, and it became the default in transformer blocks, which DL05.4 builds. The smooth slope and the small gradient for slightly negative inputs are the usual explanations offered, stated here without proof.

Worked example

A unit at \({z = -1}\). What does each activation output, and what slope does it pass back?

Two bar charts side by side with rows ReLU, softplus, SiLU and GELU. Left, vermillion bars titled Value at z equals minus 1: ReLU 0, softplus 0.313, SiLU minus 0.269, GELU minus 0.159. Right, green bars titled Slope at z equals minus 1: ReLU 0, softplus 0.269, SiLU 0.072, GELU minus 0.083. A note says ReLU sends back nothing, the smooth gates a little, and GELU's slope there is even negative.
DL01.3.3-F3

Values (vermillion) and slopes (green) at \({z = -1}\). Only ReLU passes back exactly nothing.

Use \({\sigma(-1) = 1/(1 + e) = 0.2689}\), \({\Phi(-1) = 0.1587}\) and \({\varphi(1) = 0.2420}\).

  • ReLU. Value 0, slope 0.
  • Softplus. Value \(\ln(1 + e^{-1}) = \ln 1.3679 = 0.3133\), slope \(\sigma(-1) = 0.2689\).
  • SiLU. Value \({-1 \times 0.2689} = -0.2689\). Slope \({0.2689 \times \big(1 - (1 - 0.2689)\big)}\), which is \({0.2689 \times 0.2689} = 0.0723\).
  • GELU. Value \({-1 \times 0.1587} = -0.1587\). Slope \(0.1587 - 0.2420 = -0.0833\).

A unit sitting at \({z = -1}\) is shut under ReLU and learns nothing from this example. Under SiLU it passes back about 7% of the signal; under GELU about 8%, with the opposite sign, because \({z = -1}\) is to the left of GELU’s minimum, where GELU falls as \(z\) rises.

Code

From scratch, with scipy.special.erf for \(\Phi\). Softplus uses the stable form derived above.

import numpy as np
from scipy.special import erf, expit

def softplus(z):  return np.maximum(z, 0) + np.log1p(np.exp(-np.abs(z)))
def silu(z):      return z * expit(z)
def dsilu(z):     s = expit(z); return s * (1 + z * (1 - s))
def Phi(z):       return 0.5 * (1 + erf(z / np.sqrt(2)))
def phi(z):       return np.exp(-z ** 2 / 2) / np.sqrt(2 * np.pi)
def gelu(z):      return z * Phi(z)
def dgelu(z):     return Phi(z) + z * phi(z)

z = np.linspace(-6, 6, 241)
h = 1e-6
for f, df in ((softplus, expit), (silu, dsilu), (gelu, dgelu)):
    err = np.max(np.abs((f(z + h) - f(z - h)) / (2 * h) - df(z)))
    print(f"{f.__name__:8s} largest slope error {err:.1e}")
print("value and slope at z = -1:")
for f, df in ((softplus, expit), (silu, dsilu), (gelu, dgelu)):
    print(f"  {f.__name__:8s} {f(-1.0):+.4f}  {df(-1.0):+.4f}")
softplus largest slope error 5.1e-10
silu     largest slope error 6.8e-10
gelu     largest slope error 5.7e-10
value and slope at z = -1:
  softplus +0.3133  +0.2689
  silu     -0.2689  +0.0723
  gelu     -0.1587  -0.0833

The minima, found with scipy.optimize.minimize_scalar on the interval from \(-3\) to 0, and checked against the formulas derived above:

from scipy.optimize import minimize_scalar

for f in (silu, gelu):
    r = minimize_scalar(f, bounds=(-3, 0), method="bounded", options={"xatol": 1e-10})
    print(f"{f.__name__}: minimum {r.fun:.4f} at z = {r.x:.4f}")
r_silu = minimize_scalar(silu, bounds=(-3, 0), method="bounded", options={"xatol": 1e-10})
r_gelu = minimize_scalar(gelu, bounds=(-3, 0), method="bounded", options={"xatol": 1e-10})
print("SiLU: min value equals z* + 1:", np.isclose(r_silu.fun, r_silu.x + 1))
print("GELU slope at -sqrt(2):", round(float(dgelu(-np.sqrt(2))), 4))
silu: minimum -0.2785 at z = -1.2785
gelu: minimum -0.1700 at z = -0.7518
SiLU: min value equals z* + 1: True
GELU slope at -sqrt(2): -0.1289

Last, the two quick forms of GELU, measured against the exact one on a fine grid:

grid = np.linspace(-6, 6, 120001)
tanh_form = 0.5 * grid * (1 + np.tanh(np.sqrt(2 / np.pi) * (grid + 0.044715 * grid ** 3)))
sig_form = grid * expit(1.702 * grid)
e_tanh = np.abs(tanh_form - gelu(grid))
e_sig = np.abs(sig_form - gelu(grid))
print(f"tanh form:    largest error {e_tanh.max():.2e} at |z| = {abs(grid[e_tanh.argmax()]):.2f}")
print(f"sigmoid form: largest error {e_sig.max():.4f} at |z| = {abs(grid[e_sig.argmax()]):.2f}")
tanh form:    largest error 4.73e-04 at |z| = 2.70
sigmoid form: largest error 0.0203 at |z| = 2.27

There is no GELU in SciPy, so the check against a library is the softplus one above (np.logaddexp(0, z) computes \({\ln(e^0 + e^z)}\) safely). PyTorch, from DL01.7 on, has torch.nn.GELU, whose approximate="tanh" option is the tanh form.

Explore

The module’s interactive, the activation explorer, lives in lesson DL01.3.4. Set the mean to \(-1\) and the spread to 1, then switch between ReLU, SiLU and GELU: ReLU leaves 25 of the 30 units dead, while SiLU and GELU leave none and still pass back a mean slope of about 0.18 and 0.13.

Pitfalls

Two panels of relative error on a logarithmic scale from 10 to the minus 16 up to 1, against z from minus 45 to 5. Left, marked with a cross and titled np.log(1 + np.exp(z)): the error is near 10 to the minus 16 at z equals 0, climbs as a straight line as z falls, labelled error grows like e to the minus z as z falls, and reaches 1 near z equals minus 37, where it stays flat, labelled returns 0: 100% wrong. Right, marked with a tick and titled np.logaddexp(0, z): the error stays flat at the level of rounding everywhere, labelled at the limit of float64 everywhere.
DL01.3.3-F4

Wrong beside right. Adding \(e^{z}\) to 1 loses its digits, and below about \({z = -37}\) all of them.

“Softplus is just log(1 + exp(z)).”

Mathematically yes; in floating point, no. For large positive \(z\), exp(z) overflows. For very negative \(z\), \({1 + e^{z}}\) rounds to 1 and the answer comes out as exactly 0 instead of about \(e^{z}\). Use np.logaddexp(0, z), or \({\max(0, z) + \ln(1 + e^{-\abs{z}})}\) with np.log1p.

“GELU and SiLU are the same function.”

They have the same form, \(z\) times a gate, with different gates. GELU’s gate \(\Phi\) is steeper than the sigmoid, so GELU’s dip is shallower (\(-0.170\) against \(-0.278\)) and nearer zero. Even the sigmoid approximation of GELU, \({z\,\sigma(1.702z)}\), uses a different scale inside the gate.

“A smooth version of ReLU is always better.”

Goodfellow et al. report that softplus, though smooth, did worse than ReLU in comparisons, and they discourage it. Smoothness alone is not the point; GELU’s and SiLU’s record is an empirical one, on particular architectures.

Check yourself

Recall

1. What is the derivative of softplus?

Show answer

The sigmoid: \({\frac{d}{dz}\ln(1 + e^{z}) = e^{z}/(1 + e^{z}) = \sigma(z)}\).

2. Write ReLU, SiLU and GELU as \(z\) times a gate.

Show answer

\({\operatorname{relu}(z) = z\,1[z > 0]}\), \({\operatorname{silu}(z) = z\,\sigma(z)}\), \({\operatorname{gelu}(z) = z\,\Phi(z)}\).

3. Where are the minima of SiLU and GELU, and how deep?

Show answer

SiLU: about \(-0.278\) at \({z \approx -1.278}\). GELU: about \(-0.170\) at \({z \approx -0.752}\).

4. How far above ReLU does softplus lie at most, and where?

Show answer

\({\ln 2 \approx 0.693}\), at \({z = 0}\), since \({\operatorname{softplus} = \operatorname{relu} + \ln(1 + e^{-\abs{z}})}\).

5. What is the tanh approximation of GELU, and how accurate is it?

Show answer

\[ \begin{aligned} &\tfrac12 z\,(1 + \tanh u), \\ &u = \sqrt{2/\pi}\,(z + 0.044715\,z^3), \end{aligned} \]

within about \({4.7 \times 10^{-4}}\) of the exact GELU everywhere.

Exercises

1. ★ Compute softplus, SiLU and GELU at \({z = 2}\), and their slopes there, using \({\sigma(2) = 0.8808}\), \({\Phi(2) = 0.9772}\) and \({\varphi(2) = 0.0540}\).

Show solution

Softplus: \(\ln(1 + e^2) = \ln 8.389 = 2.127\), slope \(0.8808\).

SiLU: \({2 \times 0.8808} = 1.762\), and the slope is \({0.8808\,(1 + {2 \times 0.1192})}\), which is \({0.8808 \times 1.2384} = 1.091\).

GELU: \({2 \times 0.9772} = 1.954\), slope \(0.9772 + {2 \times 0.0540} = 1.085\).

Both slopes are above 1: the overshoot.

2. ★★ Show that \({\operatorname{softplus}(z) - \operatorname{softplus}(-z) = z}\), and use it to show that the slope of softplus at \(-z\) is 1 minus its slope at \(z\).

Show solution

\[ \begin{aligned} &\ln(1 + e^{z}) - \ln(1 + e^{-z}) \\ &\quad= \ln\frac{1 + e^{z}}{1 + e^{-z}} \\ &\quad= \ln\frac{e^{z}(e^{-z} + 1)}{1 + e^{-z}} = z . \end{aligned} \]

Differentiating both sides with the chain rule gives \({\operatorname{softplus}'(z) + \operatorname{softplus}'(-z) = 1}\), that is \({\sigma(z) + \sigma(-z) = 1}\).

3. ★★★ Swish is \({z\,\sigma(\beta z)}\) with \({\beta > 0}\). Show that its minimum value is \({\operatorname{silu}(z^\ast)/\beta}\), where \(z^\ast\) is SiLU’s minimiser, and find where it is reached. What happens to the dip as \({\beta \to \infty}\)?

Show solution

Substitute \({u = \beta z}\): \({z\,\sigma(\beta z) = \tfrac1\beta\,u\,\sigma(u) = \tfrac1\beta\operatorname{silu}(u)}\). As \(z\) runs over the real line so does \(u\), so the minimum is \({\operatorname{silu}(z^\ast)/\beta \approx -0.278/\beta}\), reached at \({u = z^\ast}\), that is \({z = z^\ast/\beta \approx -1.278/\beta}\). As \(\beta\) grows the dip becomes shallower and moves to 0, and Swish tends to ReLU; with \({\beta = 1.702}\) it is the sigmoid approximation of GELU.

Summary card

Soften ReLU’s corner or its gate, and a negative input still sends a gradient back.

The anchor figure again: softplus running smoothly above the ReLU hinge, and SiLU and GELU dipping slightly below zero before following the line a equals z.
DL01.3.3-F1

Smooth corner, soft gate.

  • \({\operatorname{softplus}(z) = \ln(1 + e^{z})}\), slope \(\sigma(z)\), at most \(\ln 2\) above ReLU.
  • \({\operatorname{silu}(z) = z\,\sigma(z)}\), minimum \(-0.278\) at \({z \approx -1.278}\).
  • \({\operatorname{gelu}(z) = z\,\Phi(z)}\), minimum \(-0.170\) at \({z \approx -0.752}\).
  • Gated forms obey \({f'(z) + f'(-z) = 1}\); the tanh form of GELU is within \({4.7 \times 10^{-4}}\).

Sources and next steps

Sources. Each was opened and checked against this lesson on 15 November 2026.

  • Dan Hendrycks and Kevin Gimpel, Gaussian Error Linear Units (GELUs), arXiv:1606.08415, version 5 of June 2023. Section 2 defines \({x\Phi(x)}\), gives the tanh and \({x\sigma(1.702x)}\) approximations and names the SiLU \({x\sigma(x)}\); section 4 says the tanh form was used in every experiment; the abstract and section 5 report GELU ahead of ReLU and ELU on all tasks tried; appendix B says GELU became the default activation of state-of-the-art transformers after BERT and GPT.
  • Stefan Elfwing, Eiji Uchibe and Kenji Doya, Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning, Neural Networks 107, 3 to 11, 2018 (opened as arXiv:1702.03118). Section 2, equation 9, defines the SiLU \({z\sigma(z)}\), notes that it is not monotone, with a global minimum of about \(-0.28\) at \({z \approx -1.28}\).
  • Prajit Ramachandran, Barret Zoph and Quoc V. Le, Searching for Activation Functions, 2017: Swish, \({x \cdot \operatorname{sigmoid}(\beta x)}\), of the third exercise.
  • Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, chapter 6, MIT Press, 2016. Section 6.3.3 introduces softplus as a smooth rectifier, reports that it did worse than ReLU in comparisons and says its use is generally discouraged.
  • Simon J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Figure 3.13c draws softplus, GELU and SiLU, and the notes to chapter 3 trace them and Swish to their papers.
  • Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, BERT, NAACL 2019. Appendix A.2 says that BERT uses a GELU activation rather than the standard ReLU, following OpenAI GPT.

Next. DL01.3.4 Choosing an activation puts all of them in one network and races them.

This lesson unlocks

DL01.3.4 Choosing an activation DL05.4 The transformer block (planned)

Your progress is saved in this browser only. No account, no tracking.
Back to top