A primitive to perform layer normalization. More...

Functions
dnnl_status_t DNNL_API	dnnl_layer_normalization_forward_desc_init (dnnl_layer_normalization_desc_t lnrm_desc, dnnl_prop_kind_t prop_kind, const dnnl_memory_desc_t data_desc, const dnnl_memory_desc_t *stat_desc, float epsilon, unsigned flags)
	Initializes a layer normalization descriptor `lnrm_desc` for forward propagation using `prop_kind` (possible values are dnnl_forward_training and dnnl_forward_inference), memory descriptor `data_desc`, normalization parameter `epsilon`, and `flags` set using bit flags of type dnnl_layer_normalization_desc_t. More...

dnnl_status_t DNNL_API	dnnl_layer_normalization_backward_desc_init (dnnl_layer_normalization_desc_t lnrm_desc, dnnl_prop_kind_t prop_kind, const dnnl_memory_desc_t diff_data_desc, const dnnl_memory_desc_t data_desc, const dnnl_memory_desc_t stat_desc, float epsilon, unsigned flags)
	Initializes a layer normalization descriptor `lnrm_desc` for backward propagation with respect to data and scale-shift parameters using memory descriptors `data_desc` and `diff_data_desc`, normalization parameter `epsilon`, and `flags` set using bit flags of type dnnl_layer_normalization_desc_t. More...

Detailed Description

A primitive to perform layer normalization.

Normalization is performed over the last logical axis of data tensor.

Both forward and backward passes support in-place operation; that is, src and dst point to the same memory for forward pass, and diff_dst and diff_src point to the same memory for backward pass.

Layer normalization supports different flavors controlled by dnnl_layer_normalization_desc_t. For example, layer normalization can compute the mean and variance on its own or take them as inputs. It can either perform scaling and shifting using gamma and beta parameters or not.

See also: dnnl_layer_normalization_desc_t; Layer Normalization in developer guide; layer normalization in C++ API

Function Documentation

◆ dnnl_layer_normalization_forward_desc_init()

dnnl_status_t DNNL_API dnnl_layer_normalization_forward_desc_init	(	dnnl_layer_normalization_desc_t *	lnrm_desc,
		dnnl_prop_kind_t	prop_kind,
		const dnnl_memory_desc_t *	data_desc,
		const dnnl_memory_desc_t *	stat_desc,
		float	epsilon,
		unsigned	flags
	)

Initializes a layer normalization descriptor lnrm_desc for forward propagation using prop_kind (possible values are dnnl_forward_training and dnnl_forward_inference), memory descriptor data_desc, normalization parameter epsilon, and flags set using bit flags of type dnnl_layer_normalization_desc_t.

Inputs:

src (dnnl_query_src_md, 0)
mean (dnnl_query_src_md, 1), if dnnl_use_global_stats bit-flags is set in flags
variance (dnnl_query_src_md, 2), if dnnl_use_global_stats bit-flags is set in flags
scale_and_shift (dnnl_query_weights_md, 0), if dnnl_use_scaleshift bit-flags is set in flags

Outputs:

dst (dnnl_query_dst_md, 0)
mean (dnnl_query_dst_md, 1), if dnnl_use_global_stats bit-flags is not set in flags prop_kind = dnnl_forward_training
variance (dnnl_query_dst_md, 2), if dnnl_use_global_stats bit-flags is not set in flags and prop_kind = dnnl_forward_training

Note: In-place operation is supported; that is, dst points to the same memory as src.

See also: dnnl_layer_normalization_desc_t

◆ dnnl_layer_normalization_backward_desc_init()

dnnl_status_t DNNL_API dnnl_layer_normalization_backward_desc_init	(	dnnl_layer_normalization_desc_t *	lnrm_desc,
		dnnl_prop_kind_t	prop_kind,
		const dnnl_memory_desc_t *	diff_data_desc,
		const dnnl_memory_desc_t *	data_desc,
		const dnnl_memory_desc_t *	stat_desc,
		float	epsilon,
		unsigned	flags
	)

Initializes a layer normalization descriptor lnrm_desc for backward propagation with respect to data and scale-shift parameters using memory descriptors data_desc and diff_data_desc, normalization parameter epsilon, and flags set using bit flags of type dnnl_layer_normalization_desc_t.