Record and Replay Graph Extension#

Introduced in Level Zero version 1.17

API#

Record and Replay Graph#

This extension allows applications to capture append operations from immediate command lists into a graph, instantiate that graph, and replay it efficiently.

Applications can query support and capabilities using ze_record_replay_graph_ext_properties_t and ze_record_replay_graph_ext_flags_t. Captured graphs can be inspected with zeGraphDumpContentsExt, and ze_record_replay_graph_ext_dump_desc_t controls dump behavior.

Querying the driver and the device#

Query the driver to determine if the Record and Replay extension is available.
Query the device to determine which graph capabilities are available.
Immutable graph support is the minimum required for a functional record and replay graph.

ze_driver_handle_t hDriver; // initialized driver
ze_device_handle_t hDevice; // created device out of the driver

// query extension count
uint32_t extensionsCount = 0;
zeDriverGetExtensionProperties(hDriver, &extensionsCount, nullptr);

ze_driver_extension_properties_t *driverExtensions; // allocate memory under this pointer to fit extensionsCount elements
zeDriverGetExtensionProperties(hDriver, &extensionsCount, driverExtensions);

// discover graph extension
const char *extensionName = "ZE_extension_record_replay_graph";
size_t extensionNameLength = strlen(extensionName);

for (uint32_t i = 0; i < extensionsCount; i++) {
    if (strncmp(driverExtensions[i].name, extensionName, extensionNameLength) == 0) {
        printf("Extension %s found at version %u.%un",
               extensionName,
               ZE_MAJOR_VERSION(driverExtensions[i].version),
               ZE_MINOR_VERSION(driverExtensions[i].version));
    }
}

ze_record_replay_graph_ext_properties_t graphProperties = {
    ZE_STRUCTURE_TYPE_RECORD_REPLAY_GRAPH_EXT_PROPERTIES
};

ze_device_properties_t deviceProps = {
    ZE_STRUCTURE_TYPE_DEVICE_PROPERTIES_1_2   // ZE_STRUCTURE_TYPE_DEVICE_PROPERTIES deprecated since 1.17
};
deviceProps.pNext = &graphProperties;

zeDeviceGetProperties(hDevice, &deviceProps);
if (!(graphProperties.graphFlags & ZE_RECORD_REPLAY_GRAPH_EXT_FLAG_IMMUTABLE_GRAPH)) {
    printf("Basic graph functionality for this device is not availablen");
}

Typical Flow#

Create a graph with zeGraphCreateExt, or begin capture directly on an immediate command list with zeCommandListBeginGraphCaptureExt.
Record append operations and optionally query capture state via zeCommandListIsGraphCaptureEnabledExt and zeCommandListGetGraphExt.
End capture using zeCommandListEndGraphCaptureExt.
Instantiate an executable graph with zeGraphInstantiateExt.
Submit the executable graph using zeCommandListAppendGraphExt.
Optionally dump graph contents with zeGraphDumpContentsExt.
Destroy executable and recorded graph objects using zeExecutableGraphDestroyExt and zeGraphDestroyExt.

External Events#

Counter based events created with ZE_EVENT_COUNTER_BASED_FLAG_GRAPH_EXTERNAL can be used for synchronization outside of a graph, both on the device and on the host.
Counter based events created without this flag can also act as GRAPH_EXTERNAL events when used while recording a graph with
zeCommandListAppendWaitOnEventsWithParameters or zeCommandListAppendSignalEventWithParameters and an event flags descriptor specifying ZE_EVENT_EXT_FLAG_APPEND_GRAPH_EXTERNAL.
The flag affects only the wait or signal operation for which it is specified and does not modify the event object itself.
The state of an external wait event is refreshed when zeCommandListAppendGraphExt is called.
Therefore, if the event was signaled from an immediate command list or from a different graph, the current wait state is propagated to the graph before graph execution.
The state of an external signal event is reattached to the current graph when zeCommandListAppendGraphExt is called.
Therefore, if the event was previously used by a different graph or an immediate command list, it becomes associated with the current graph and subsequent event state updates are driven by the current graph execution. As a result, zeEventQueryStatus may return ZE_RESULT_NOT_READY.

ze_command_list_handle_t hCommandList; // immediate command list in record mode for the root branch
ze_command_list_handle_t hChildCommandList; // immediate command list used for the fork branch
ze_event_handle_t hEvent; // counter based event not created with ZE_EVENT_COUNTER_BASED_FLAG_GRAPH_EXTERNAL

ze_event_ext_flags_desc_t graphExternalDesc = {
    ZE_STRUCTURE_TYPE_EVENT_EXT_FLAGS_DESC,      // stype
    nullptr,                                       // pNext
    ZE_EVENT_EXT_FLAG_APPEND_GRAPH_EXTERNAL      // flags
};

// now the event acts as an external wait event, can receive the signals from outside the graph (for example from another graph, an immediate command list or a mutable command list)
zeCommandListAppendWaitOnEventsWithParameters(hCommandList, &graphExternalDesc, 1, &hEvent);

// now the event acts as an internal signal event to pass a signal within the graph to the fork branch
zeCommandListAppendSignalEvent(hCommandList, hEvent);

// now the event acts as an internal wait event used to receive the signal from the root branch
zeCommandListAppendWaitOnEvents(hChildCommandList, 1, &hEvent);

// some work inside the fork branch
/// ...

// now the event acts as an internal signal event to pass a signal from the fork branch back to the root branch
zeCommandListAppendSignalEvent(hChildCommandList, hEvent);

// now the event acts as an internal wait event used to receive the signal from the fork branch
zeCommandListAppendWaitOnEvents(hCommandList, 1, &hEvent);

// some work inside the root branch
// ...

// now the event acts as an external signal event; the event can be used outside the graph for host or device synchronization
zeCommandListAppendSignalEventWithParameters(hCommandList, &graphExternalDesc, hEvent);