.. index:: pair: page; Local Response Normalization (LRN) .. _doxid-dev_guide_lrn: Local Response Normalization (LRN) ================================== :ref:`API Reference ` General ~~~~~~~ The LRN primitive performs a forward or backward local response normalization. Forward ------- The LRN operation is defined by the following formulas (the variable names follow the standard :ref:`Naming Conventions `): LRN across channels: .. math:: \dst(n, c, h, w) = \left\{k + \frac{\alpha}{n_{l}} \sum\limits_{i=-(n_{l}-1)/2}^{(n_{l}+1)/2-1} (\src(n, c+i, h, w))^2 \right\}^{-\beta} \cdot \src(n, c, h, w), LRN within a single channel: .. math:: \dst(n, c, h, w) = \left\{k + \frac{\alpha}{n_{l}} \sum\limits_{i=-(n_{l}-1)/2}^{(n_{l}+1)/2-1} \sum\limits_{j=-(n_{l}-1)/2}^{(n_{l}+1)/2-1} (\src(n, c, h+i, w+j))^2 \right\}^{-\beta} \cdot \src(n, c, h, w), where :math:`n_{l}` is the ``local_size``. Formulas are provided for 2D spatial data case. Backward -------- The backward propagation computes :math:`\diffsrc(n, c, h, w)`, based on :math:`\diffdst(n, c, h, w)` and :math:`\src(n, c, h, w)`. Execution Arguments ~~~~~~~~~~~~~~~~~~~ When executed, the inputs and outputs should be mapped to an execution argument index as specified by the following table. ========================================================== ==================== ============= Argument Index Type ========================================================== ==================== ============= :math:`\src` DNNL_ARG_SRC Input :math:`\dst` DNNL_ARG_DST Output workspace DNNL_ARG_WORKSPACE Input/Output :math:`\diffsrc` DNNL_ARG_DIFF_SRC Output :math:`\diffdst` DNNL_ARG_DIFF_DST Input :ref:`scratchpad ` DNNL_ARG_SCRATCHPAD Output ========================================================== ==================== ============= Implementation Details ~~~~~~~~~~~~~~~~~~~~~~ General Notes ------------- #. During training, LRN might or might not require a workspace on forward and backward passes. The behavior is implementation specific. Optimized implementations typically require a workspace and use it to save some intermediate results from the forward pass that accelerate computations on the backward pass. To check whether a workspace is required, query the LRN primitive descriptor for the workspace. Success indicates that the workspace is required and its description will be returned. Data Type Support ----------------- The LRN primitive supports the following combinations of data types: =================== ===================== Propagation Source / Destination =================== ===================== forward / backward f32, bf16, f16 =================== ===================== .. warning:: There might be hardware and/or implementation specific restrictions. Check the :ref:`Implementation Limitations ` section below. Data Representation ------------------- Source, Destination, and Their Gradients ++++++++++++++++++++++++++++++++++++++++ Like most other primitives, the LRN primitive expects the following tensors: ======== ============================================== Spatial Source / Destination ======== ============================================== 0D :math:`N \times C` 1D :math:`N \times C \times W` 2D :math:`N \times C \times H \times W` 3D :math:`N \times C \times D \times H \times W` ======== ============================================== The LRN primitive is optimized for the following memory formats: ======== =============== ======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================= Spatial Logical tensor Implementations optimized for memory formats ======== =============== ======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================= 2D NCHW :ref:`dnnl_nchw ` ( :ref:`dnnl_abcd ` ), :ref:`dnnl_nhwc ` ( :ref:`dnnl_acdb ` ), *optimized^* ======== =============== ======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================= Here, optimized^ means the format that :ref:`comes out ` of any preceding compute-intensive primitive. Post-ops and Attributes ----------------------- The LRN primitive does not support any post-ops or attributes. :target:`doxid-dev_guide_lrn_1dg_lrn_impl_limits` Implementation Limitations ~~~~~~~~~~~~~~~~~~~~~~~~~~ #. Refer to :ref:`Data Types ` for limitations related to data types support. #. GPU * Supports only 2D spatial case. Performance Tips ~~~~~~~~~~~~~~~~ #. For backward propagation, use the same memory format for ``src``, ``diff_dst``, and ``diff_src`` (the format of the ``diff_dst`` and ``diff_src`` are always the same because of the API). Different formats are functionally supported but lead to highly suboptimal performance. Examples ~~~~~~~~ See :ref:`Examples and Tutorials ` page for a complete list. LRN examples are listed in the :ref:`Normalization Primitives ` section.