Contents

LeakyReLU

General

LeakyReLU operation is a type of activation function based on ReLU. It has a small slope for negative values with which LeakyReLU can produce small, non-zero, and constant gradients with respect to the negative values. The slope is also called the coefficient of leakage.

Unlike PReLU, the coefficient \(\alpha\) is constant and defined before training.

LeakyReLU operation applies following formula on every element of \(\src\) tensor (the variable names follow the standard Naming Conventions):

\[\begin{split}dst = \begin{cases} src & \text{if}\ src \ge 0 \\ \alpha src & \text{if}\ src < 0 \end{cases}\end{split}\]

Operation attributes

Attribute Name

Description

Value Type

Supported Values

Required or Optional

alpha

Alpha is the coefficient of leakage.

f32

Arbitrary f32 value but usually a small positive value.

Required

Execution arguments

The inputs and outputs must be provided according to below index order when constructing an operation.

Inputs

Index

Argument Name

Required or Optional

0

src

Required

Outputs

Index

Argument Name

Required or Optional

0

dst

Required

Supported data types

LeakyReLU operation supports the following data type combinations.

Src

Dst

f32

f32

bf16

bf16

f16

f16
LayerNormBackward Log