DynamicQuantize

General

DynamicQuantize operation converts an f32 tensor to a quantized (s8, u8, f8_e4m3, or f8_e5m2) tensor. It supports per-tensor, per-channel, and mask-based linear quantization. The target quantized data type is specified via the data type of the dst logical tensor. Rounding mode is library-implementation defined.

For per-tensor quantization

\[dst = round(src/scales + zps)\]

For per-channel quantization, taking channel axis = 1 as an example:

\[{dst}_{\cdots,i,\cdots,\cdots} = round(src_{\cdots,i,\cdots,\cdots}/scales_i + zps_i),i\in [0,channelNum-1]\]

Operation attributes

Attribute Name	Description	Value Type	Supported Values	Required or Optional
qtype	Specifies which quantization type is used.	string	per_tensor (default), per_channel	Optional
axis	Specifies dimension on which per-channel quantization is applied.	s64	A s64 value in the range of [-r, r-1] where r = rank(src), 1 by default. Negative value means counting the dimension backwards from the end.	Optional
mask	Specifies which dimensions the scales/zps vary over. Bit i set means scales vary on dimension i . When set, qtype must be not set or set to the default.	s64	A non-negative integer where set bits index source dimensions.	Optional

Note

The mask attribute is the recommended way to specify quantization granularity. The qtype and axis attributes are retained for backward compatibility and will be deprecated in the future.

Execution arguments

The inputs and outputs must be provided according to below index order when constructing an operation.

Inputs

Index	Argument Name	Required or Optional
0	src	Required
1	scales	Required
2	zps	Optional

Note

scales is an f32 tensor to be applied to the quantization formula. For qtype = per-tensor, there should be only one element in the scales tensor. For qtype = per-channel, the element number should be equal to the element number of src tensor along the dimension axis. When mask is specified, the scales tensor is a packed tensor whose dimensions correspond to the source dimensions selected by the mask bits.

Note

zps is a tensor with offset values that map to zero. For qtype = per-tensor, there should be only one element in the zps tensor. For qtype = per-channel, the element number should be equal to the element number of input tensor along the dimension axis. When mask is specified, zps must have the same shape as scales. If omitted, zps values are assumed to be zero. The zps input must be omitted when dst has an f8_e4m3 or f8_e5m2 data type.

Outputs

Index	Argument Name	Required or Optional
0	dst	Required

Supported data types

DynamicQuantize operation supports the following data type combinations.

Src	Scales	Zps	Dst
f32	f32	s8, u8, s32	s8
f32	f32	s8, u8, s32	u8
f32	f32	N/A	f8_e4m3
f32	f32	N/A	f8_e5m2