Contents

Compiling and Running Programs

Compiling and Running Programs

Consider the simple example program from Section Writing Intel® SHMEM Programs and assume the code is in a file called ishmem_example.cpp.

To compile the program with the default static host and static device libraries, the necessary flags must be passed to the Intel® oneAPI DPC++/C++ Compiler. For example:

$ icpx -I${ISHMEM_INSTALL_DIR}/include -fsycl -std=gnu++1z ishmem_example.cpp ${ISHMEM_INSTALL_DIR}/lib/libishmem_host.a ${ISHMEM_INSTALL_DIR}/lib/libishmem_device.a -o ishmem_example -lpthread -lze_loader

To link the host library dynamically while keeping the device library static:

$ icpx -I${ISHMEM_INSTALL_DIR}/include -fsycl -std=gnu++1z ishmem_example.cpp ${ISHMEM_INSTALL_DIR}/lib/libishmem_host.so ${ISHMEM_INSTALL_DIR}/lib/libishmem_device.a -o ishmem_example -lpthread -lze_loader

where ISHMEM_INSTALL_DIR is the path to the Intel® SHMEM installation directory.

Alternatively, when building with CMake, the find_package command may be used to define all necessary compiler flags. For example:

find_package(ISHMEM REQUIRED)
add_executable(ishmem_example ishmem_example.cpp)
target_link_libraries(ishmem_example PRIVATE ISHMEM::ISHMEM)

ISHMEM::ISHMEM uses the static host library by default. Configure the consumer with -DISHMEM_HOST_LINK_TYPE=SHARED to select libishmem_host.so. Explicit targets are also provided as ISHMEM::ISHMEM_STATIC and ISHMEM::ISHMEM_SHARED.

If Intel® SHMEM is not sourced via the installed environment script, it may be necessary to prepend the installation path to CMAKE_PREFIX_PATH.

Intel® SHMEM provides a launcher script, ishmrun, that sets CPU and GPU affinity so that each PE is assigned a single SYCL device and a corresponding set of CPU cores with close affinity. To invoke the ishmrun script, pass it as the first argument to your process launcher. The following example assumes the ISHMEM_INSTALL_DIR/bin directory is on your user path and use of the Portable Batch System launcher:

$ aprun -N 12 -n 6 ishmrun ishmem_example

This will launch the example program on 12 PEs with 6 PEs per compute node.

As described in section Building Intel® SHMEM, the following environment variables may be required for execution, depending on the Intel® SHMEM build configuration:

ISHMEM_RUNTIME
ISHMEM_MPI_LIB_NAME
ISHMEM_SHMEM_LIB_NAME
ISHMEM_RUNTIME_USE_OSHMPI

See section Library Constants for more information about these variables.

Launcher-Independent MPI Initialization

Intel® SHMEM also supports launcher-independent MPI initialization with ishmemx_attr_t.use_uid for workflows such as manual process launch or torchrun-style launchers.

This mode requires:

  • ISHMEM_RUNTIME=MPI

  • ishmemx_attr_t.use_uid = true

  • ishmemx_attr_t.nranks and ishmemx_attr_t.rank

  • ishmemx_attr_t.uid set from ishmemx_get_uniqueid

The following environment variables must be set by the launcher or run script:

  • WORLD_SIZE

  • RANK

  • I_MPI_MPCP_SERVER_NAME or MASTER_ADDR

Optional variables:

  • I_MPI_MPCP_SERVER_PORT (defaults to 35555 when unset)

  • MASTER_PORT (if set, it must be different from I_MPI_MPCP_SERVER_PORT)

Example (manual launch on one node):

export WORLD_SIZE=4
export MASTER_ADDR=localhost
export ISHMEM_RUNTIME=MPI
RANK=0 ./app &
RANK=1 ./app &
RANK=2 ./app &
RANK=3 ./app &
wait

See examples/7_uid_mpcp.cpp for a complete example using this flow.

Selecting SPIR-V Compilation Targets

On some systems, you may encounter an error in which the correct SPIR-V targets are not successfully selected when linking with Intel® SHMEM. This may result in problems when using device-initiated communication including compilation warnings:

icpx: warning: linked binaries do not contain expected 'spir64-unknown-unknown' target; found targets: 'spir64_gen-unknown-unknown' [-Wsycl-target]

as well as runtime errors:

terminate called after throwing an instance of 'sycl::_V1::compile_program_error'
  what():  The program was built for 1 devices
Build program log for 'Intel(R) Data Center GPU Max 1550':
Module <0x29941d0>:  Unresolved Symbol <_Z13ishmem_putmemPvPKvmi>
Module <0x29941d0>:  Unresolved Symbol <_Z13ishmem_putmemPvPKvmi>
Module <0x29941d0>:  Unresolved Symbol <_Z13ishmem_putmemPvPKvmi>
Module <0x29941d0>:  Unresolved Symbol <_Z13ishmem_putmemPvPKvmi> -11 (PI_ERROR_BUILD_PROGRAM_FAILURE)

These errors can be resolved by ensuring the desired target(s) match the device IR bundled into the Intel® SHMEM device library. The target(s) are specified at Intel® SHMEM’s configure time using -DISHMEM_AOT_DEVICE_TYPES. The default value is xe-hpc,xe2 to target Intel® Data Center Max and Intel® Arc™ B-Series GPUs, respectively. Below is an example set of flags to add to the linking process for adding these target devices:

-fsycl-targets=spir64_gen --start-no-unused-arguments -Xs "-device xe-hpc,xe2" --end-no-unused-arguments --start-no-unused-arguments -Xsycl-target-backend "-q" --end-no-unused-arguments

When building with CMake, the ISHMEM::ISHMEM interface automatically adds the corresponding target devices to the compilation command.

previous

Writing Intel® SHMEM Programs

next

Using Intel® SHMEM in Intel® Tiber™ AI Cloud