.. index:: pair: page; Softmax .. _doxid-dev_guide_softmax: Softmax ======= :ref:`API Reference ` General ~~~~~~~ The softmax primitive performs forward or backward softmax or logsoftmax operation along a particular axis on data with arbitrary dimensions. All other axes are treated as independent (batch). Forward ------- In general form, the operation is defined by the following formulas (the variable names follow the standard :ref:`Naming Conventions `). Softmax: .. math:: \dst(\overline{ou}, c, \overline{in}) = \frac {e^{\src(\overline{ou}, c, \overline{in}) - \nu(\overline{ou}, \overline{in})}} { \sum\limits_{ic} e^{\src(\overline{ou}, ic, \overline{in}) - \nu(\overline{ou}, \overline{in})} } Logsoftmax: .. math:: \dst(\overline{ou}, c, \overline{in}) = \ln\left({\frac { e^{\src(\overline{ou}, c, \overline{in}) - \nu(\overline{ou}, \overline{in})} } { \sum\limits_{ic} e^{\src(\overline{ou}, ic, \overline{in}) - \nu(\overline{ou}, \overline{in})} }}\right) = \left(\src(\overline{ou}, c, \overline{in}) - \nu(\overline{ou}, \overline{in})\right) - \ln\left( \sum\limits_{ic} e^{\src(\overline{ou}, ic, \overline{in}) - \nu(\overline{ou}, \overline{in})} \right) Above * :math:`c` is the axis over which the operation is computed on, * :math:`\overline{ou}` is the outermost index (to the left of the axis), * :math:`\overline{in}` is the innermost index (to the right of the axis), and * :math:`\nu` is used to produce numerically stable results and defined as: .. math:: \nu(\overline{ou}, \overline{in}) = \max\limits_{ic} \src(\overline{ou}, ic, \overline{in}) Difference Between Forward Training and Forward Inference +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ There is no difference between the :ref:`dnnl_forward_training ` and :ref:`dnnl_forward_inference ` propagation kinds. Backward -------- The backward propagation computes :math:`\diffsrc(ou, c, in)`, based on :math:`\diffdst(ou, c, in)` and :math:`\dst(ou, c, in)`. Execution Arguments ~~~~~~~~~~~~~~~~~~~ When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table. ========================================================== ================================================================================================================================================================= ======= Argument Index Type ========================================================== ================================================================================================================================================================= ======= :math:`\src` DNNL_ARG_SRC Input :math:`\dst` DNNL_ARG_DST Output :math:`\diffsrc` DNNL_ARG_DIFF_SRC Output :math:`\diffdst` DNNL_ARG_DIFF_DST Input :math:`src scale` DNNL_ARG_ATTR_SCALES | DNNL_ARG_SRC Input :math:`dst scale` DNNL_ARG_ATTR_SCALES | DNNL_ARG_DST Input :math:`\text{binary post-op}` :ref:`DNNL_ARG_ATTR_MULTIPLE_POST_OP(binary_post_op_position) ` | DNNL_ARG_SRC_1 Input \ :ref:`DNNL_ARG_ATTR_MULTIPLE_POST_OP(binary_post_op_position) ` | DNNL_ARG_SRC_2 Input :ref:`scratchpad ` DNNL_ARG_SCRATCHPAD Output ========================================================== ================================================================================================================================================================= ======= Implementation Details ~~~~~~~~~~~~~~~~~~~~~~ General Notes ------------- #. Both forward and backward propagation support in-place operations, meaning that :math:`\src` can be used as input and output for forward propagation, and :math:`\diffdst` can be used as input and output for backward propagation. In case of in-place operation, the original data will be overwritten. This support is limited to cases when data types of :math:`\src` and :math:`\dst` or :math:`\diffsrc` and :math:`\diffdst` are identical. Post-ops and Attributes ----------------------- Attributes enable you to modify the behavior of the softmax primitive. The following attributes are supported by the softmax primitive: ============ ========== ================================================================================================== ===================================================================================== ======================================================================= Propagation Type Operation Description Restrictions ============ ========== ================================================================================================== ===================================================================================== ======================================================================= forward attribute :ref:`Scales ` Scales the corresponding tensor by the given scale factor(s). Supported only for int8 softmax and one scale per tensor is supported. forward post-op :ref:`Binary ` Applies a :ref:`Binary ` operation to the result General binary post-op restrictions forward Post-op :ref:`Eltwise ` Applies an :ref:`Eltwise ` operation to the result. forward attribute :ref:`Accumulation mode ` Defines the implementation's accumulation arithmetic. Only the values ``strict`` , ``relaxed`` , and ``any`` are supported. ============ ========== ================================================================================================== ===================================================================================== ======================================================================= Accumulation Mode +++++++++++++++++ You can optimize performance of the forward operation when the source and destination floating-point data types of the operation are equal and different from ``f32``. When the destination data type is different from ``f32``, additional memory will be used to accumulate data and store it in the destination memory buffer for a requested data type. Using the additional memory can be opted-out with an accumulation mode setting set to :ref:`relaxed ` or :ref:`any `, which will use the precision of destination data type to accumulate intermediate results directly into the destination memory buffer. This performance optimization, however, results in in a minor decrease in accuracy. Depending on the actual data, the difference between ``strict`` and ``relaxed`` accumulation can reach several units in the last piece (ulps). Data Type Support ----------------- The softmax primitive supports the following combinations of data types: ============ ============================ ============================ Propagation Source Destination ============ ============================ ============================ forward f32, f64, bf16, f16, u8, s8 f32, f64, bf16, f16, u8, s8 backward f32, f64, bf16, f16 f32, f64, bf16, f16 ============ ============================ ============================ Data Representation ------------------- Source, Destination, and Their Gradients ++++++++++++++++++++++++++++++++++++++++ The softmax primitive works with arbitrary data tensors. There is no special meaning associated with any logical dimensions. However, the softmax axis is typically referred to as channels (hence in formulas :math:`c` is used). Implementation Limitations ~~~~~~~~~~~~~~~~~~~~~~~~~~ #. Refer to :ref:`Data Types ` for limitations related to data types support. #. GPU * Only tensors of 6 or fewer dimensions are supported. Performance Tips ~~~~~~~~~~~~~~~~ #. Use in-place operations whenever possible. Examples ~~~~~~~~ See :ref:`Examples and Tutorials ` page for a complete list. Softmax examples are listed in the :ref:`Activation Functions ` section.