What is SELU?

Introduction

SELU Scaled Exponential Linear Unit is a self-normalizing activation function. It is a variant of the ELU . The main advantage of SELU is that we can be sure that the output will always be standardized due to its self-normalizing behavior. That means there is no need to include Batch-Normalization layers. The following equation represents the working of the SELU activation function:

Where λ\lambda and α \alpha are constants with values:

Problem with ReLU

The reason why ReLU was famous is that it solved the problem of vanishing gradient. Nevertheless, ReLU created a problem for itself called the dying ReLU problem. This problem occurs when ReLU outputs 0 on any input.

To counter this, we require our activation function to output values other than zero, so the gradient persists. SELU is one of those functions.

Implementation with Python

import numpy as np
import matplotlib.pyplot as plt
# initializing the constants
λ = 1.0507
α = 1.6732
def SELU(x):
    if x > 0:
        return λ*x
    return λ*α*(np.exp(x) - 1)
x = np.linspace(-5.0, 5.0)
result = []
for i in x:
    result.append(SELU(i))
plt.plot(x, result)
plt.title("SELU activation function")
plt.xlabel("Input")
plt.ylabel("Output")
plt.grid(True)
plt.savefig('output/selu_plot.png')

Explanation

  • Line 10–11: These lines implement the equation mentioned above.
  • Line 13: Here, we use np.linspace to generate evenly spaced numbers between –5.0 to 5.0. By default, it generates a total of 50 numbers.
  • Line 18–23: Here, we use the matplotlib library to plot the output of SELU over the given range.
Free AI Mock Interviews
Coding Interview
Coding PatternsFree Interview
Gain insights and practical experience with coding patterns through targeted MCQs and coding problems, designed to match and challenge your expertise level.
System Design
You TubeFree Interview
Learn to design a video streaming platform like YouTube by tackling functional and non-functional requirements, core components, and high-level to detailed design challenges.

Free Resources

Copyright ©2025 Educative, Inc. All rights reserved