.. index:: pair: group; Layer Normalization .. _doxid-group__dnnl__api__layer__normalization: Layer Normalization =================== .. toctree:: :hidden: struct_dnnl_layer_normalization_backward.rst struct_dnnl_layer_normalization_forward.rst Overview ~~~~~~~~ A primitive to perform layer normalization. :ref:`More...` .. ref-code-block:: cpp :class: doxyrest-overview-code-block // structs struct :ref:`dnnl::layer_normalization_backward`; struct :ref:`dnnl::layer_normalization_forward`; // global functions :ref:`dnnl_status_t` DNNL_API :ref:`dnnl_layer_normalization_forward_primitive_desc_create`( :ref:`dnnl_primitive_desc_t`* primitive_desc, :ref:`dnnl_engine_t` engine, :ref:`dnnl_prop_kind_t` prop_kind, :ref:`const_dnnl_memory_desc_t` src_desc, :ref:`const_dnnl_memory_desc_t` dst_desc, :ref:`const_dnnl_memory_desc_t` stat_desc, float epsilon, unsigned flags, :ref:`const_dnnl_primitive_attr_t` attr ); :ref:`dnnl_status_t` DNNL_API :ref:`dnnl_layer_normalization_backward_primitive_desc_create`( :ref:`dnnl_primitive_desc_t`* primitive_desc, :ref:`dnnl_engine_t` engine, :ref:`dnnl_prop_kind_t` prop_kind, :ref:`const_dnnl_memory_desc_t` diff_src_desc, :ref:`const_dnnl_memory_desc_t` diff_dst_desc, :ref:`const_dnnl_memory_desc_t` src_desc, :ref:`const_dnnl_memory_desc_t` stat_desc, float epsilon, unsigned flags, :ref:`const_dnnl_primitive_desc_t` hint_fwd_pd, :ref:`const_dnnl_primitive_attr_t` attr ); :ref:`dnnl_status_t` DNNL_API :ref:`dnnl_layer_normalization_forward_primitive_desc_create_v2`( :ref:`dnnl_primitive_desc_t`* primitive_desc, :ref:`dnnl_engine_t` engine, :ref:`dnnl_prop_kind_t` prop_kind, :ref:`const_dnnl_memory_desc_t` src_desc, :ref:`const_dnnl_memory_desc_t` dst_desc, :ref:`const_dnnl_memory_desc_t` stat_desc, :ref:`dnnl_data_type_t` scale_shift_data_type, float epsilon, unsigned flags, :ref:`const_dnnl_primitive_attr_t` attr ); :ref:`dnnl_status_t` DNNL_API :ref:`dnnl_layer_normalization_backward_primitive_desc_create_v2`( :ref:`dnnl_primitive_desc_t`* primitive_desc, :ref:`dnnl_engine_t` engine, :ref:`dnnl_prop_kind_t` prop_kind, :ref:`const_dnnl_memory_desc_t` diff_src_desc, :ref:`const_dnnl_memory_desc_t` diff_dst_desc, :ref:`const_dnnl_memory_desc_t` src_desc, :ref:`const_dnnl_memory_desc_t` stat_desc, :ref:`dnnl_data_type_t` diff_scale_shift_data_type, :ref:`dnnl_data_type_t` scale_shift_data_type, float epsilon, unsigned flags, :ref:`const_dnnl_primitive_desc_t` hint_fwd_pd, :ref:`const_dnnl_primitive_attr_t` attr ); .. _details-group__dnnl__api__layer__normalization: Detailed Documentation ~~~~~~~~~~~~~~~~~~~~~~ A primitive to perform layer normalization. Normalization is performed within the last logical dimension of data tensor. Both forward and backward propagation primitives support in-place operation; that is, src and dst can refer to the same memory for forward propagation, and diff_dst and diff_src can refer to the same memory for backward propagation. The layer normalization primitives computations can be controlled by specifying different :ref:`dnnl::normalization_flags ` values. For example, layer normalization forward propagation can be configured to either compute the mean and variance or take them as arguments. It can either perform scaling and shifting using gamma and beta parameters or not. .. rubric:: See also: :ref:`Layer Normalization ` in developer guide Global Functions ---------------- .. index:: pair: function; dnnl_layer_normalization_forward_primitive_desc_create .. _doxid-group__dnnl__api__layer__normalization_1ga3f99050f79a43b697bf35cc0f39e21fe: .. ref-code-block:: cpp :class: doxyrest-title-code-block :ref:`dnnl_status_t` DNNL_API dnnl_layer_normalization_forward_primitive_desc_create( :ref:`dnnl_primitive_desc_t`* primitive_desc, :ref:`dnnl_engine_t` engine, :ref:`dnnl_prop_kind_t` prop_kind, :ref:`const_dnnl_memory_desc_t` src_desc, :ref:`const_dnnl_memory_desc_t` dst_desc, :ref:`const_dnnl_memory_desc_t` stat_desc, float epsilon, unsigned flags, :ref:`const_dnnl_primitive_attr_t` attr ) Creates a primitive descriptor for a layer normalization forward propagation primitive. .. note:: In-place operation is supported: the dst can refer to the same memory as the src. .. rubric:: Parameters: .. list-table:: :widths: 20 80 * - primitive_desc - Output primitive_descriptor. * - engine - Engine to use. * - prop_kind - Propagation kind. Possible values are :ref:`dnnl_forward_training ` and :ref:`dnnl_forward_inference `. * - src_desc - Source memory descriptor. * - dst_desc - Destination memory descriptor. * - stat_desc - Memory descriptor for mean and variance. If this parameter is NULL, a zero memory descriptor, or a memory descriptor with format_kind set to :ref:`dnnl_format_kind_undef `, then the memory descriptor for stats is derived from ``src_desc`` by removing the last dimension. * - epsilon - Layer normalization epsilon parameter. * - flags - Layer normalization flags (:ref:`dnnl_normalization_flags_t `). * - attr - Primitive attributes (can be NULL). .. rubric:: Returns: :ref:`dnnl_success ` on success and a status describing the error otherwise. .. index:: pair: function; dnnl_layer_normalization_backward_primitive_desc_create .. _doxid-group__dnnl__api__layer__normalization_1gaea99a932365f2e330420fdefabcfcd05: .. ref-code-block:: cpp :class: doxyrest-title-code-block :ref:`dnnl_status_t` DNNL_API dnnl_layer_normalization_backward_primitive_desc_create( :ref:`dnnl_primitive_desc_t`* primitive_desc, :ref:`dnnl_engine_t` engine, :ref:`dnnl_prop_kind_t` prop_kind, :ref:`const_dnnl_memory_desc_t` diff_src_desc, :ref:`const_dnnl_memory_desc_t` diff_dst_desc, :ref:`const_dnnl_memory_desc_t` src_desc, :ref:`const_dnnl_memory_desc_t` stat_desc, float epsilon, unsigned flags, :ref:`const_dnnl_primitive_desc_t` hint_fwd_pd, :ref:`const_dnnl_primitive_attr_t` attr ) Creates a primitive descriptor for a layer normalization backward propagation primitive. .. note:: In-place operation is supported: the diff_dst can refer to the same memory as the diff_src. .. rubric:: Parameters: .. list-table:: :widths: 20 80 * - primitive_desc - Output primitive_descriptor. * - engine - Engine to use. * - prop_kind - Propagation kind. Possible values are :ref:`dnnl_backward_data ` and :ref:`dnnl_backward ` (diffs for all parameters are computed in this case). * - diff_src_desc - Diff source memory descriptor. * - diff_dst_desc - Diff destination memory descriptor. * - src_desc - Source memory descriptor. * - stat_desc - Memory descriptor for mean and variance. If this parameter is NULL, a zero memory descriptor, or a memory descriptor with format_kind set to :ref:`dnnl_format_kind_undef `, then the memory descriptor for stats is derived from ``src_desc`` by removing the last dimension. * - epsilon - Layer normalization epsilon parameter. * - flags - Layer normalization flags (:ref:`dnnl_normalization_flags_t `). * - hint_fwd_pd - Primitive descriptor for a respective forward propagation primitive. * - attr - Primitive attributes (can be NULL). .. rubric:: Returns: :ref:`dnnl_success ` on success and a status describing the error otherwise. .. index:: pair: function; dnnl_layer_normalization_forward_primitive_desc_create_v2 .. _doxid-group__dnnl__api__layer__normalization_1ga9efbb60f29ecac498a78a828521cb0fd: .. ref-code-block:: cpp :class: doxyrest-title-code-block :ref:`dnnl_status_t` DNNL_API dnnl_layer_normalization_forward_primitive_desc_create_v2( :ref:`dnnl_primitive_desc_t`* primitive_desc, :ref:`dnnl_engine_t` engine, :ref:`dnnl_prop_kind_t` prop_kind, :ref:`const_dnnl_memory_desc_t` src_desc, :ref:`const_dnnl_memory_desc_t` dst_desc, :ref:`const_dnnl_memory_desc_t` stat_desc, :ref:`dnnl_data_type_t` scale_shift_data_type, float epsilon, unsigned flags, :ref:`const_dnnl_primitive_attr_t` attr ) Creates a primitive descriptor for a layer normalization forward propagation primitive with a user-provided data type for the scale and shift memory objects. .. note:: In-place operation is supported: the dst can refer to the same memory as the src. .. rubric:: Parameters: .. list-table:: :widths: 20 80 * - primitive_desc - Output primitive_descriptor. * - engine - Engine to use. * - prop_kind - Propagation kind. Possible values are :ref:`dnnl_forward_training ` and :ref:`dnnl_forward_inference `. * - src_desc - Source memory descriptor. * - dst_desc - Destination memory descriptor. * - stat_desc - Memory descriptor for mean and variance. If this parameter is NULL, a zero memory descriptor, or a memory descriptor with format_kind set to :ref:`dnnl_format_kind_undef `, then the memory descriptor for stats is derived from ``src_desc`` by removing the last dimension. * - scale_shift_data_type - Data type of scale and shift memory. If neither scale nor shift flag are specified the parameter is ignored. * - epsilon - Layer normalization epsilon parameter. * - flags - Layer normalization flags (:ref:`dnnl_normalization_flags_t `). * - attr - Primitive attributes (can be NULL). .. rubric:: Returns: :ref:`dnnl_success ` on success and a status describing the error otherwise. .. index:: pair: function; dnnl_layer_normalization_backward_primitive_desc_create_v2 .. _doxid-group__dnnl__api__layer__normalization_1ga4f398ee5311dcdb475ac18f379958b6a: .. ref-code-block:: cpp :class: doxyrest-title-code-block :ref:`dnnl_status_t` DNNL_API dnnl_layer_normalization_backward_primitive_desc_create_v2( :ref:`dnnl_primitive_desc_t`* primitive_desc, :ref:`dnnl_engine_t` engine, :ref:`dnnl_prop_kind_t` prop_kind, :ref:`const_dnnl_memory_desc_t` diff_src_desc, :ref:`const_dnnl_memory_desc_t` diff_dst_desc, :ref:`const_dnnl_memory_desc_t` src_desc, :ref:`const_dnnl_memory_desc_t` stat_desc, :ref:`dnnl_data_type_t` diff_scale_shift_data_type, :ref:`dnnl_data_type_t` scale_shift_data_type, float epsilon, unsigned flags, :ref:`const_dnnl_primitive_desc_t` hint_fwd_pd, :ref:`const_dnnl_primitive_attr_t` attr ) Creates a primitive descriptor for a layer normalization backward propagation primitive with a user-provided data type for the scale and shift memory objects. .. note:: In-place operation is supported: the diff_dst can refer to the same memory as the diff_src. .. rubric:: Parameters: .. list-table:: :widths: 20 80 * - primitive_desc - Output primitive_descriptor. * - engine - Engine to use. * - prop_kind - Propagation kind. Possible values are :ref:`dnnl_backward_data ` and :ref:`dnnl_backward ` (diffs for all parameters are computed in this case). * - diff_src_desc - Diff source memory descriptor. * - diff_dst_desc - Diff destination memory descriptor. * - src_desc - Source memory descriptor. * - stat_desc - Memory descriptor for mean and variance. If this parameter is NULL, a zero memory descriptor, or a memory descriptor with format_kind set to :ref:`dnnl_format_kind_undef `, then the memory descriptor for stats is derived from ``src_desc`` by removing the last dimension. * - diff_scale_shift_data_type - Data type of diff scale and shift memory. If neither scale nor shift flag are specified the parameter is ignored. * - scale_shift_data_type - Data type of scale and shift memory. If neither scale nor shift flag are specified the parameter is ignored. * - epsilon - Layer normalization epsilon parameter. * - flags - Layer normalization flags (:ref:`dnnl_normalization_flags_t `). * - hint_fwd_pd - Primitive descriptor for a respective forward propagation primitive. * - attr - Primitive attributes (can be NULL). .. rubric:: Returns: :ref:`dnnl_success ` on success and a status describing the error otherwise.