enum dnnl_normalization_flags_t¶
Overview¶
Flags for normalization primitives. More…
#include <dnnl_types.h> enum dnnl_normalization_flags_t { dnnl_normalization_flags_none = 0x0U, dnnl_use_global_stats = 0x1U, dnnl_use_scale = 0x2U, dnnl_use_shift = 0x4U, dnnl_fuse_norm_relu = 0x8U, dnnl_fuse_norm_add_relu = 0x10U, };
Detailed Documentation¶
Flags for normalization primitives.
Enum Values¶
dnnl_normalization_flags_none
Use no normalization flags.
If specified
on forward training propagation mean and variance are computed and stored as output
on backward propagation compute full derivative wrt data
on backward propagation prop_kind == dnnl_backward_data has the same behavior as prop_kind == dnnl_backward
dnnl_use_global_stats
Use global statistics.
If specified
on forward propagation use mean and variance provided by user (input)
on backward propagation reduces the amount of computations, since mean and variance are considered as constants
If not specified:
on forward propagation mean and variance are computed and stored as output
on backward propagation compute full derivative wrt data
dnnl_use_scale
Use scale parameter.
If specified:
on forward propagation use scale for the normalization results
on backward propagation (for prop_kind == dnnl_backward) compute diff wrt scale (hence one extra output used)
dnnl_use_shift
Use shift parameter.
If specified:
on forward propagation use shift (aka bias) for the normalization results
on backward propagation (for prop_kind == dnnl_backward) compute diff wrt shift (hence one extra output used)
dnnl_fuse_norm_relu
Fuse with ReLU.
The flag implies negative slope being 0. On training this is the only configuration supported. For inference, to use non-zero negative slope consider using Primitive Attributes: Post-ops.
If specified:
on inference this option behaves the same as if the primitive were fused with ReLU using post ops API with zero negative slope.
on training primitive requires workspace (required to be able to perform backward pass)
dnnl_fuse_norm_add_relu
Fuse with Add and then fuse with ReLU.
If specified:
on forward propagation apply element-wise binary Add operation to to the normalization results with an additional input tensor and then apply ReLU with negative slope being 0.
on training primitive requires workspace (required to be able to perform backward pass).
on backward propagation save the result of backward ReLU operation with input tensor and workspace from forward pass to extra output tensor and then perform backward normalization.