.. index:: pair: page; LayerNorm .. _doxid-dev_guide_op_layernorm: LayerNorm ========= General ~~~~~~~ LayerNorm performs a layer normalization operation on :math:`\src` tensor. The layerNorm operation performs normalization from ``begin_norm_axis`` to last dimension of the data tensor. It is defined by the following formulas which is the same as :ref:`Layer Normalization `. .. math:: \dst(t, n, c) = \gamma(c) \cdot \frac{\src(t, n, c) - \mu(t, n)} {\sqrt{\sigma^2(t, n) + \epsilon}} + \beta(c), where * :math:`\gamma(c), \beta(c)` are optional scale and shift for a channel * :math:`\mu(t, n), \sigma^2(t, n)` are mean and variance (see * :math:`\epsilon` is a constant to improve numerical stability. Mean and variance are computed at runtime or provided by a user. When mean and variance are computed at runtime, the following formulas are used: * :math:`\mu(t, n) = \frac{1}{C} \sum\limits_{c} \src(t, n, c)_{}`, * :math:`\sigma^2(t, n) = \frac{1}{C} \sum\limits_{c} {}_{} (\src(t, n, c) - \mu(t, n))^2`. Operation attributes ~~~~~~~~~~~~~~~~~~~~ ============================================================================================================================ ================================================================================================================================================================================================================================================================================================== =========== ================================================= ===================== Attribute Name Description Value Type Supported Values Required or Optional ============================================================================================================================ ================================================================================================================================================================================================================================================================================================== =========== ================================================= ===================== :ref:`keep_stats ` Indicate whether to output mean and variance which can be later passed to backward op. bool ``false`` , ``true`` (default) Optional :ref:`begin_norm_axis ` ``begin_norm_axis`` is used to indicate which axis to start layer normalization. The normalization is from ``begin_norm_axis`` to last dimension. Negative values means indexing from right to left. This op normalizes over the last dimension by default, e.g. C in TNC for 3D and LDNC for 4D. s64 [-r,r-1],where r=rank(src). -1 is default Optional :ref:`use_affine ` When set to True, this module has learnable per-element affine parameters. bool ``false`` , ``true`` (default) Optional :ref:`epsilon ` The constant to improve numerical stability. f32 Arbitrary positive f32 value, ``1e-5`` (default) Optional ============================================================================================================================ ================================================================================================================================================================================================================================================================================================== =========== ================================================= ===================== Execution arguments ~~~~~~~~~~~~~~~~~~~ The inputs and outputs must be provided according to below index order when constructing an operation. Inputs ------ ====== ============== ===================== Index Argument Name Required or Optional ====== ============== ===================== 0 ``src`` Required 1 ``gamma`` Optional 2 ``beta`` Optional ====== ============== ===================== .. note:: ``gamma`` is scaling for normalized value. ``beta`` is the bias added to the scaled normalized value. They are both 1D tensor with the same span as src’s channel axis and required if attribute ``use_affine`` is set to True. Outputs ------- ====== ============== ===================== Index Argument Name Required or Optional ====== ============== ===================== 0 ``dst`` Required 1 ``mean`` Optional 2 ``variance`` Optional ====== ============== ===================== .. note:: Both ``mean`` and ``variance`` are required if attribute ``keep_stats`` is set to True. Supported data types ~~~~~~~~~~~~~~~~~~~~ LayerNorm operation supports the following data type combinations. ========== =============================== Src / Dst Gamma / Beta / Mean / Variance ========== =============================== f32 f32 bf16 f32, bf16 f16 f32 ========== =============================== Implementation Notes ~~~~~~~~~~~~~~~~~~~~ LayerNorm supports in-place operations, meaning that ``src`` can be used as both input and output (``dst``). In case of in-place operation, the original ``src`` data will be overwritten. This support is limited to cases when data types of ``src`` and ``dst`` are identical. Use in-place operations whenever possible for performance.