Reorder between CPU and GPU engines

This C++ API example demonstrates programming flow when reordering memory between CPU and GPU engines.

Example code: cross_engine_reorder.cpp

Public headers

To start using oneDNN, we must first include the dnnl.hpp header file in the application. We also include dnnl_debug.h, which contains some debugging facilities such as returning a string representation for common oneDNN C types.

All C++ API types and functions reside in the dnnl namespace. For simplicity of the example we import this namespace.

cross_engine_reorder_tutorial() function

Engine and stream

All oneDNN primitives and memory objects are attached to a particular dnnl::engine, which is an abstraction of a computational device (see also Basic Concepts). The primitives are created and optimized for the device they are attached to, and the memory objects refer to memory residing on the corresponding device. In particular, that means neither memory objects nor primitives that were created for one engine can be used on another.

To create engines, we must specify the dnnl::engine::kind and the index of the device of the given kind. There is only one CPU engine and one GPU engine, so the index for both engines must be 0.

    auto cpu_engine = engine(validate_engine_kind(engine::kind::cpu), 0);
    auto gpu_engine = engine(validate_engine_kind(engine::kind::gpu), 0);

In addition to an engine, all primitives require a dnnl::stream for the execution. The stream encapsulates an execution context and is tied to a particular engine.

In this example, a GPU stream is created.

    auto stream_gpu = stream(gpu_engine, stream::flags::in_order);

Wrapping data into oneDNN GPU memory object

Fill the data in CPU memory first, and then move data from CPU to GPU memory by reorder.

    const auto tz = memory::dims {2, 16, 1, 1};
    auto m_cpu
            = memory({{tz}, memory::data_type::f32, memory::format_tag::nchw},
                    cpu_engine);
    auto m_gpu
            = memory({{tz}, memory::data_type::f32, memory::format_tag::nchw},
                    gpu_engine);
    fill(m_cpu, tz);
    auto r1 = reorder(m_cpu, m_gpu);

Creating a ReLU primitive

Let's now create a ReLU primitive for GPU.

The library implements the ReLU primitive as a particular algorithm of a more general Eltwise primitive, which applies a specified function to each element of the source tensor.

Just as in the case of dnnl::memory, a user should always go through (at least) three creation steps (which, however, can sometimes be combined thanks to C++11):

1. Initialize an operation descriptor (in the case of this example, dnnl::eltwise_forward::desc), which defines the operation parameters including a GPU memory descriptor.
2. Create an operation primitive descriptor (here dnnl::eltwise_forward::primitive_desc) on a GPU engine, which is a lightweight descriptor of the actual algorithm that implements the given operation.
3. Create a primitive (here dnnl::eltwise_forward) that can be executed on GPU memory objects to compute the operation by a GPU engine.

Note

Primitive creation might be a very expensive operation, so consider creating primitive objects once and executing them multiple times.

The code:

    //  ReLU op descriptor (uses a GPU memory as source memory.
    //  no engine- or implementation-specific information)
    auto relu_d = eltwise_forward::desc(prop_kind::forward,
            algorithm::eltwise_relu, m_gpu.get_desc(), 0.0f);
    // ReLU primitive descriptor, which corresponds to a particular
    // implementation in the library. Specify engine type for the ReLU
    // primitive. Use a GPU engine here.
    auto relu_pd = eltwise_forward::primitive_desc(relu_d, gpu_engine);
    // ReLU primitive
    auto relu = eltwise_forward(relu_pd);

Getting results from a oneDNN GPU memory object

After the ReLU operation, users need to get data from GPU to CPU memory by reorder.

    auto r2 = reorder(m_gpu, m_cpu);

Executing all primitives

Finally, let's execute all primitives and wait for their completion via the following sequence:

Reorder(CPU,GPU) -> ReLU -> Reorder(GPU,CPU).

1. After execution of the first Reorder, ReLU has source data in GPU.
2. The input and output memory objects are passed to the ReLU execute() method using a <tag, memory> map. Each tag specifies what kind of tensor each memory object represents. All Eltwise primitives require the map to have two elements: a source memory object (input) and a destination memory (output). For executing on GPU engine, both source and destination memory object must use GPU memory.
3. After the execution of the ReLU on GPU, the second Reorder moves the results from GPU to CPU.

Note

All primitives are executed in the SAME GPU stream (the first parameter of the execute() method).

Execution is asynchronous on GPU. This means that we need to call dnnl::stream::wait before accessing the results.

    // wrap source data from CPU to GPU
    r1.execute(stream_gpu, m_cpu, m_gpu);
    // Execute ReLU on a GPU stream
    relu.execute(stream_gpu, {{DNNL_ARG_SRC, m_gpu}, {DNNL_ARG_DST, m_gpu}});
    // Get result data from GPU to CPU
    r2.execute(stream_gpu, m_gpu, m_cpu);

    stream_gpu.wait();

Validate the result

Now that we have the computed the result on CPU memory, let's validate that it is actually correct.

    if (find_negative(m_cpu, tz) != 0)
        throw std::logic_error(
                "Unexpected output, find a negative value after the ReLU "
                "execution.");

Upon compiling and running the example, the output should be just:

Example passed.