LeakyReLU¶

General¶

LeakyReLU operation is a type of activation function based on ReLU. It has a small slope for negative values with which LeakyReLU can produce small, non-zero, and constant gradients with respect to the negative values. The slope is also called the coefficient of leakage.

Unlike PReLU, the coefficient $$\alpha$$ is constant and defined before training.

LeakyReLU operation applies following formula on every element of $$\src$$ tensor (the variable names follow the standard Naming Conventions):

$\begin{split}dst = \begin{cases} src & \text{if}\ src \ge 0 \\ \alpha src & \text{if}\ src < 0 \end{cases}\end{split}$

Operation attributes¶

Attribute Name

Descr

alpha

Alpha is the coefficient of leakage.

f32

Arbitrary f32 value but usually a small positive value.

Required

Execution arguments¶

The inputs and outputs must be provided according to below index order when constructing an operation.

Inputs¶

Index

Argu

0

src

Required

Outputs¶

Index

Argu

0

dst

Required

Supported data types¶

LeakyReLU operation supports the following data type combinations.

Src

D

f32

f32

bf16

bf16

f16

f16