.. index:: pair: page; GELU .. _doxid-dev_guide_op_gelu: GELU ==== General ~~~~~~~ GELU operation applies following formula on every element of :math:`\src` tensor (the variable names follow the standard :ref:`Naming Conventions `): .. math:: dst = 0.5 * src * (1.0 + erf(src) / \sqrt2) When the attribute ``mode`` is specified as ``gelu_tanh``, an approximation implementation is used: .. math:: dst = 0.5 * src * (1.0 + \tanh[\sqrt{\frac{2}{\pi}} (src + 0.044715 * s^3)]) Operation attributes ~~~~~~~~~~~~~~~~~~~~ ================================================================================================================= ======================================== =========== ====================================== ===================== Attribute Name Description Value Type Supported Values Required or Optional ================================================================================================================= ======================================== =========== ====================================== ===================== :ref:`mode ` Specifies the computation mode of GELU. string ``gelu_erf`` (default), ``gelu_tanh`` Optional ================================================================================================================= ======================================== =========== ====================================== ===================== Execution arguments ~~~~~~~~~~~~~~~~~~~ The inputs and outputs must be provided according to below index order when constructing an operation. Inputs ------ ====== ============== ===================== Index Argument Name Required or Optional ====== ============== ===================== 0 ``src`` Required ====== ============== ===================== Outputs ------- ====== ============== ===================== Index Argument Name Required or Optional ====== ============== ===================== 0 ``dst`` Required ====== ============== ===================== Supported data types ~~~~~~~~~~~~~~~~~~~~ GELU operation supports the following data type combinations. ===== ===== Src Dst ===== ===== f32 f32 f16 f16 bf16 bf16 ===== ===== Implementation Notes ~~~~~~~~~~~~~~~~~~~~ GELU supports in-place operations, meaning that ``src`` can be used as both input and output (``dst``). In case of in-place operation, the original ``src`` data will be overwritten. Use in-place operations whenever possible for performance.