.. index:: pair: page; Convolution int8 inference example with Graph API
.. _doxid-graph_cpu_inference_int8_cpp:

Convolution int8 inference example with Graph API
=================================================

This is an example to demonstrate how to build an int8 graph with Graph API and run it on CPU.

Example code: :ref:`cpu_inference_int8.cpp <doxid-cpu_inference_int8_8cpp-example>`

Some assumptions in this example:

* Only workflow is demonstrated without checking correctness

* Unsupported partitions should be handled by users themselves



.. _doxid-graph_cpu_inference_int8_cpp_1graph_cpu_inference_int8_cpp_headers:

Public headers
~~~~~~~~~~~~~~

To start using oneDNN Graph, we must include the ``dnnl_graph.hpp`` header file in the application. All the C++ APIs reside in namespace ``:ref:`dnnl::graph <doxid-namespacednnl_1_1graph>```.

.. ref-code-block:: cpp

	#include <iostream>
	#include <memory>
	#include <vector>
	#include <unordered_map>
	#include <unordered_set>
	
	#include <assert.h>
	
	#include "oneapi/dnnl/dnnl_graph.hpp"
	
	#include "example_utils.hpp"
	#include "graph_example_utils.hpp"
	
	using namespace :ref:`dnnl::graph <doxid-namespacednnl_1_1graph>`;
	using :ref:`data_type <doxid-classdnnl_1_1graph_1_1logical__tensor_1acddb1dc65b7b4feede7710a719f32227>` = :ref:`logical_tensor::data_type <doxid-classdnnl_1_1graph_1_1logical__tensor_1acddb1dc65b7b4feede7710a719f32227>`;
	using :ref:`layout_type <doxid-classdnnl_1_1graph_1_1logical__tensor_1ad3fcaff44671577e56adb03b770f4867>` = :ref:`logical_tensor::layout_type <doxid-classdnnl_1_1graph_1_1logical__tensor_1ad3fcaff44671577e56adb03b770f4867>`;
	using :ref:`property_type <doxid-classdnnl_1_1graph_1_1logical__tensor_1a037ba7c242d8127d2f42c0c2aef29d82>` = :ref:`logical_tensor::property_type <doxid-classdnnl_1_1graph_1_1logical__tensor_1a037ba7c242d8127d2f42c0c2aef29d82>`;
	using dim = :ref:`logical_tensor::dim <doxid-classdnnl_1_1graph_1_1logical__tensor_1a759c7b96472681049e17716334a2b334>`;
	using dims = :ref:`logical_tensor::dims <doxid-classdnnl_1_1graph_1_1logical__tensor_1a31af724d1ea783a09b6900d69b43ddc7>`;





.. _doxid-graph_cpu_inference_int8_cpp_1graph_cpu_inference_int8_cpp_tutorial:

simple_pattern_int8() function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



.. _doxid-graph_cpu_inference_int8_cpp_1graph_cpu_inference_int8_cpp_get_partition:

Build Graph and Get Partitions
------------------------------

In this section, we are trying to build a graph indicating an int8 convolution with relu post-op. After that, we can get all of partitions which are determined by backend.

Create input/output :ref:`dnnl::graph::logical_tensor <doxid-classdnnl_1_1graph_1_1logical__tensor>` and op for the first ``Dequantize``.

.. ref-code-block:: cpp

	logical_tensor dequant0_src_desc {0, data_type::u8};
	logical_tensor conv_src_desc {1, data_type::f32};
	op dequant0(2, op::kind::Dequantize, {dequant0_src_desc}, {conv_src_desc},
	        "dequant0");
	dequant0.set_attr<std::string>(op::attr::qtype, "per_tensor");
	dequant0.set_attr<std::vector<float>>(op::attr::scales, {0.1f});
	dequant0.set_attr<std::vector<int64_t>>(op::attr::zps, {10});

Create input/output :ref:`dnnl::graph::logical_tensor <doxid-classdnnl_1_1graph_1_1logical__tensor>` and op for the second ``Dequantize``.

.. note:: 

   It's necessary to provide scale and weight information on the ``Dequantize`` on weight.
   
   

.. note:: 

   Users can set weight property type to ``constant`` to enable dnnl weight cache for better performance
   
   


.. ref-code-block:: cpp

	logical_tensor dequant1_src_desc {3, data_type::s8};
	logical_tensor conv_weight_desc {
	        4, data_type::f32, 4, layout_type::undef, property_type::constant};
	op dequant1(5, op::kind::Dequantize, {dequant1_src_desc},
	        {conv_weight_desc}, "dequant1");
	dequant1.set_attr<std::string>(op::attr::qtype, "per_channel");
	// the memory format of weight is XIO, which indicates channel equals
	// to 64 for the convolution.
	std::vector<float> wei_scales(64, 0.1f);
	dims wei_zps(64, 0);
	dequant1.set_attr<std::vector<float>>(op::attr::scales, wei_scales);
	dequant1.set_attr<std::vector<int64_t>>(op::attr::zps, wei_zps);
	dequant1.set_attr<int64_t>(op::attr::axis, 1);













Create input/output :ref:`dnnl::graph::logical_tensor <doxid-classdnnl_1_1graph_1_1logical__tensor>` the op for ``Convolution``.

.. ref-code-block:: cpp

	logical_tensor conv_bias_desc {
	        6, data_type::f32, 1, layout_type::undef, property_type::constant};
	logical_tensor conv_dst_desc {7, data_type::f32, layout_type::undef};

	// create the convolution op
	op conv(8, op::kind::Convolution,
	        {conv_src_desc, conv_weight_desc, conv_bias_desc}, {conv_dst_desc},
	        "conv");
	conv.set_attr<dims>(op::attr::strides, {1, 1});
	conv.set_attr<dims>(op::attr::pads_begin, {0, 0});
	conv.set_attr<dims>(op::attr::pads_end, {0, 0});
	conv.set_attr<dims>(op::attr::dilations, {1, 1});
	conv.set_attr<std::string>(op::attr::data_format, "NXC");
	conv.set_attr<std::string>(op::attr::weights_format, "XIO");
	conv.set_attr<int64_t>(op::attr::groups, 1);











Create input/output :ref:`dnnl::graph::logical_tensor <doxid-classdnnl_1_1graph_1_1logical__tensor>` the op for ``ReLu``.

.. ref-code-block:: cpp

	logical_tensor relu_dst_desc {9, data_type::f32, layout_type::undef};
	op relu(10, op::kind::ReLU, {conv_dst_desc}, {relu_dst_desc}, "relu");









Create input/output :ref:`dnnl::graph::logical_tensor <doxid-classdnnl_1_1graph_1_1logical__tensor>` the op for ``Quantize``.

.. ref-code-block:: cpp

	logical_tensor quant_dst_desc {11, data_type::u8, layout_type::undef};
	op quant(
	        12, op::kind::Quantize, {relu_dst_desc}, {quant_dst_desc}, "quant");
	quant.set_attr<std::string>(op::attr::qtype, "per_tensor");
	quant.set_attr<std::vector<float>>(op::attr::scales, {0.1f});
	quant.set_attr<std::vector<int64_t>>(op::attr::zps, {10});







Finally, those created ops will be added into the graph. The graph inside will maintain a list to store all these ops. To create a graph, :ref:`dnnl::engine::kind <doxid-structdnnl_1_1engine_1a2635da16314dcbdb9bd9ea431316bb1a>` is needed because the returned partitions maybe vary on different devices. For this example, we use CPU engine.

.. note:: 

   The order of adding op doesn't matter. The connection will be obtained through logical tensors.
   
   
Create graph and add ops to the graph

.. ref-code-block:: cpp

	:ref:`graph <doxid-group__dnnl__graph__api__dump__mode_1gga48d03b4285480cd0df9d587ddeec293daf8b0b924ebd7046dbfa85a856e4682c8>` g(:ref:`dnnl::engine::kind::cpu <doxid-structdnnl_1_1engine_1a2635da16314dcbdb9bd9ea431316bb1aad9747e2da342bdb995f6389533ad1a3d>`);

	g.add_op(dequant0);
	g.add_op(dequant1);
	g.add_op(conv);
	g.add_op(relu);
	g.add_op(quant);





After finished above operations, we can get partitions by calling :ref:`dnnl::graph::graph::get_partitions() <doxid-classdnnl_1_1graph_1_1graph_1a116d3552e3b0e6c739a1564329bde014>`.

In this example, the graph will be partitioned into one partition.

.. ref-code-block:: cpp

	auto partitions = g.get_partitions();





.. _doxid-graph_cpu_inference_int8_cpp_1graph_cpu_inference_int8_cpp_compile:

Compile and Execute Partition
-----------------------------

In the real case, users like framework should provide device information at this stage. But in this example, we just use a self-defined device to simulate the real behavior.

Create a :ref:`dnnl::engine <doxid-structdnnl_1_1engine>`. Also, set a user-defined :ref:`dnnl::graph::allocator <doxid-classdnnl_1_1graph_1_1allocator>` to this engine.

.. ref-code-block:: cpp

	allocator alloc {};
	:ref:`dnnl::engine <doxid-structdnnl_1_1engine>` eng
	        = :ref:`make_engine_with_allocator <doxid-group__dnnl__graph__api__engine_1ga42ac93753b2a12d14b29704fe3b0b2fa>`(:ref:`dnnl::engine::kind::cpu <doxid-structdnnl_1_1engine_1a2635da16314dcbdb9bd9ea431316bb1aad9747e2da342bdb995f6389533ad1a3d>`, 0, alloc);
	:ref:`dnnl::stream <doxid-structdnnl_1_1stream>` strm {eng};

Compile the partition to generate compiled partition with the input and output logical tensors.

.. ref-code-block:: cpp

	compiled_partition cp = partition.compile(inputs, outputs, eng);





Execute the compiled partition on the specified stream.

.. ref-code-block:: cpp

	cp.execute(strm, inputs_ts, outputs_ts);