Intel® SHMEM

Introduction / Essentials

• Introduction
  • What is Intel® SHMEM?
  • What is OpenSHMEM?
• Programming Model Overview
  • The Intel® SHMEM APIs Overview
  • The on_queue APIs Overview
  • The work_group APIs Overview
• Memory Model
  • Pointers to Symmetric Objects
  • Atomicity Guarantees
• Execution Model
  • Invoking Intel® SHMEM Operations
• Supported/Unsupported Features

Library Constants and Env Vars

• Library Constants
• Library Handles
• Environment Variables

Intel® SHMEM API

• Library Setup, Exit, and Query Routines
  • ISHMEM_INIT
    • ishmem_init()
  • ISHMEMX_INIT_ATTR
    • ishmemx_runtime_type_t
    • ishmemx_attr_t
    • ishmemx_init_attr()
  • ISHMEM_MY_PE
    • ishmem_my_pe()
  • ISHMEM_N_PES
    • ishmem_n_pes()
  • ISHMEM_FINALIZE
    • ishmem_finalize()
  • ISHMEM_QUERY_INITIALIZED
    • ishmem_query_initialized()
  • ISHMEM_PTR
    • ishmem_ptr()
  • ISHMEM_INFO_GET_VERSION
    • ishmem_info_get_version()
  • ISHMEM_INFO_GET_NAME
    • ishmem_info_get_name()
• Thread Support
  • ISHMEM_INIT_THREAD
    • ishmem_init_thread()
  • ISHMEM_QUERY_THREAD
    • ishmem_query_thread()
• Memory Management
  • ISHMEM_MALLOC
    • ishmem_malloc()
  • ISHMEM_FREE
    • ishmem_free()
  • ISHMEM_ALIGN
    • ishmem_align()
  • ISHMEM_CALLOC
    • ishmem_calloc()
• Team Management Routines
  • Predefined and Application-Defined Teams
  • Team Handles
  • Thread Safety
  • Collective Ordering
  • Team Creation
  • ISHMEM_TEAM_MY_PE
    • ishmem_team_my_pe()
  • ISHMEM_TEAM_N_PES
    • ishmem_team_n_pes()
  • ISHMEM_TEAM_CONFIG_T
    • ishmem_team_config_t
  • ISHMEM_TEAM_GET_CONFIG
    • ishmem_team_get_config()
  • ISHMEM_TEAM_TRANSLATE_PE
    • ishmem_team_translate_pe()
  • ISHMEM_TEAM_SPLIT_STRIDED
    • ishmem_team_split_strided()
  • ISHMEM_TEAM_SPLIT_2D
    • ishmem_team_split_2d()
  • ISHMEM_TEAM_DESTROY
    • ishmem_team_destroy()
• Remote Memory Access (RMA)
  • Blocking RMA
    • ISHMEM_PUT
    • ISHMEMX_PUT_ON_QUEUE
    • ISHMEMX_PUT_WORK_GROUP
    • ISHMEM_P
    • ISHMEM_IPUT
    • ISHMEMX_IPUT_ON_QUEUE
    • ISHMEMX_IPUT_WORK_GROUP
    • ISHMEMX_IBPUT
    • ISHMEMX_IBPUT_ON_QUEUE
    • ISHMEMX_IBPUT_WORK_GROUP
    • ISHMEM_GET
    • ISHMEMX_GET_ON_QUEUE
    • ISHMEMX_GET_WORK_GROUP
    • ISHMEM_G
    • ISHMEM_IGET
    • ISHMEMX_IGET_ON_QUEUE
    • ISHMEMX_IGET_WORK_GROUP
    • ISHMEMX_IBGET
    • ISHMEMX_IBGET_ON_QUEUE
    • ISHMEMX_IBGET_WORK_GROUP
  • Nonblocking RMA
    • ISHMEM_PUT_NBI
    • ISHMEMX_PUT_NBI_ON_QUEUE
    • ISHMEMX_PUT_NBI_WORK_GROUP
    • ISHMEM_GET_NBI
    • ISHMEMX_GET_NBI_ON_QUEUE
    • ISHMEMX_GET_NBI_WORK_GROUP
• Atomic Memory Operations (AMO)
  • Blocking Atomic Memory Operations
    • ISHMEM_ATOMIC_FETCH
    • ISHMEM_ATOMIC_SET
    • ISHMEM_ATOMIC_COMPARE_SWAP
    • ISHMEM_ATOMIC_SWAP
    • ISHMEM_ATOMIC_FETCH_INC
    • ISHMEM_ATOMIC_INC
    • ISHMEM_ATOMIC_FETCH_ADD
    • ISHMEM_ATOMIC_ADD
    • ISHMEM_ATOMIC_FETCH_AND
    • ISHMEM_ATOMIC_AND
    • ISHMEM_ATOMIC_FETCH_OR
    • ISHMEM_ATOMIC_OR
    • ISHMEM_ATOMIC_FETCH_XOR
    • ISHMEM_ATOMIC_XOR
  • Non-blocking Atomic Memory Operations
    • ISHMEM_ATOMIC_FETCH_NBI
    • ISHMEM_ATOMIC_COMPARE_SWAP_NBI
    • ISHMEM_ATOMIC_SWAP_NBI
    • ISHMEM_ATOMIC_FETCH_INC_NBI
    • ISHMEM_ATOMIC_FETCH_ADD_NBI
    • ISHMEM_ATOMIC_FETCH_AND_NBI
    • ISHMEM_ATOMIC_FETCH_OR_NBI
    • ISHMEM_ATOMIC_FETCH_XOR_NBI
• Signaling Operations
  • Atomicity Guarantees for Signaling Operations
  • Available Signal Operators
  • Blocking Put-Signal
    • ISHMEM_PUT_SIGNAL
    • ISHMEMX_PUT_SIGNAL_ON_QUEUE
    • ISHMEMX_PUT_SIGNAL_WORK_GROUP
  • Nonblocking Put-Signal
    • ISHMEM_PUT_SIGNAL_NBI
    • ISHMEMX_PUT_SIGNAL_NBI_ON_QUEUE
    • ISHMEMX_PUT_SIGNAL_NBI_WORK_GROUP
  • ISHMEMX_SIGNAL_ADD
    • ishmemx_signal_add()
  • ISHMEM_SIGNAL_FETCH
    • ishmem_signal_fetch()
  • ISHMEMX_SIGNAL_SET
    • ishmemx_signal_set()
• Collective Operations
  • ISHMEM_BARRIER_ALL
    • ishmem_barrier_all()
  • ISHMEMX_BARRIER_ALL_ON_QUEUE
    • ishmemx_barrier_all_on_queue()
  • ISHMEMX_BARRIER_ALL_WORK_GROUP
    • ishmemx_barrier_all_work_group()
  • ISHMEM_SYNC_ALL
    • ishmem_sync_all()
  • ISHMEMX_SYNC_ALL_ON_QUEUE
    • ishmemx_sync_all_on_queue()
  • ISHMEMX_SYNC_ALL_WORK_GROUP
    • ishmemx_sync_all_work_group()
  • ISHMEM_TEAM_SYNC
    • ishmem_team_sync()
  • ISHMEMX_TEAM_SYNC_ON_QUEUE
    • ishmemx_team_sync_on_queue()
  • ISHMEMX_TEAM_SYNC_WORK_GROUP
    • ishmemx_team_sync_work_group()
  • ISHMEM_ALLTOALL
    • ishmem_alltoall()
    • ishmem_alltoall()
    • ishmem_TYPENAME_alltoall()
    • ishmem_TYPENAME_alltoall()
    • ishmem_alltoallmem()
    • ishmem_alltoallmem()
  • ISHMEMX_ALLTOALL_ON_QUEUE
    • ishmemx_alltoall_on_queue()
    • ishmemx_alltoall_on_queue()
    • ishmemx_TYPENAME_alltoall_on_queue()
    • ishmemx_TYPENAME_alltoall_on_queue()
    • ishmemx_alltoallmem_on_queue()
    • ishmemx_alltoallmem_on_queue()
  • ISHMEMX_ALLTOALL_WORK_GROUP
    • ishmemx_alltoall_work_group()
    • ishmemx_alltoall_work_group()
    • ishmemx_TYPENAME_alltoall_work_group()
    • ishmemx_TYPENAME_alltoall_work_group()
    • ishmemx_alltoallmem_work_group()
    • ishmemx_alltoallmem_work_group()
  • ISHMEM_BROADCAST
    • ishmem_broadcast()
    • ishmem_broadcast()
    • ishmem_TYPENAME_broadcast()
    • ishmem_TYPENAME_broadcast()
    • ishmem_broadcastmem()
    • ishmem_broadcastmem()
  • ISHMEMX_BROADCAST_ON_QUEUE
    • ishmemx_broadcast_on_queue()
    • ishmemx_broadcast_on_queue()
    • ishmemx_TYPENAME_broadcast_on_queue()
    • ishmemx_TYPENAME_broadcast_on_queue()
    • ishmemx_broadcastmem_on_queue()
    • ishmemx_broadcastmem_on_queue()
  • ISHMEMX_BROADCAST_WORK_GROUP
    • ishmemx_broadcast_work_group()
    • ishmemx_broadcast_work_group()
    • ishmemx_TYPENAME_broadcast_work_group()
    • ishmemx_TYPENAME_broadcast_work_group()
    • ishmemx_broadcastmem_work_group()
    • ishmemx_broadcastmem_work_group()
  • ISHMEM_COLLECT, ISHMEM_FCOLLECT
    • ishmem_collect()
    • ishmem_collect()
    • ishmem_fcollect()
    • ishmem_fcollect()
    • ishmem_TYPENAME_collect()
    • ishmem_TYPENAME_collect()
    • ishmem_TYPENAME_fcollect()
    • ishmem_TYPENAME_fcollect()
    • ishmem_collectmem()
    • ishmem_collectmem()
    • ishmem_fcollectmem()
    • ishmem_fcollectmem()
  • ISHMEMX_COLLECT_ON_QUEUE, ISHMEMX_FCOLLECT_ON_QUEUE
    • ishmemx_collect_on_queue()
    • ishmemx_collect_on_queue()
    • ishmemx_fcollect_on_queue()
    • ishmemx_fcollect_on_queue()
    • ishmemx_TYPENAME_collect_on_queue()
    • ishmemx_TYPENAME_collect_on_queue()
    • ishmemx_TYPENAME_fcollect_on_queue()
    • ishmemx_TYPENAME_fcollect_on_queue()
    • ishmemx_collectmem_on_queue()
    • ishmemx_collectmem_on_queue()
    • ishmemx_fcollectmem_on_queue()
    • ishmemx_fcollectmem_on_queue()
  • ISHMEMX_COLLECT_WORK_GROUP, ISHMEMX_FCOLLECT_WORK_GROUP
    • ishmemx_collect_work_group()
    • ishmemx_collect_work_group()
    • ishmemx_fcollect_work_group()
    • ishmemx_fcollect_work_group()
    • ishmemx_TYPENAME_collect_work_group()
    • ishmemx_TYPENAME_collect_work_group()
    • ishmemx_TYPENAME_fcollect_work_group()
    • ishmemx_TYPENAME_fcollect_work_group()
    • ishmemx_collectmem_work_group()
    • ishmemx_collectmem_work_group()
    • ishmemx_fcollectmem_work_group()
    • ishmemx_fcollectmem_work_group()
  • ISHMEM_REDUCE
    • ishmem_and_reduce()
    • ishmem_and_reduce()
    • ishmem_or_reduce()
    • ishmem_or_reduce()
    • ishmem_xor_reduce()
    • ishmem_xor_reduce()
    • ishmem_max_reduce()
    • ishmem_max_reduce()
    • ishmem_min_reduce()
    • ishmem_min_reduce()
    • ishmem_sum_reduce()
    • ishmem_sum_reduce()
    • ishmem_prod_reduce()
    • ishmem_prod_reduce()
    • ishmem_TYPENAME_and_reduce()
    • ishmem_TYPENAME_and_reduce()
    • ishmem_TYPENAME_or_reduce()
    • ishmem_TYPENAME_or_reduce()
    • ishmem_TYPENAME_xor_reduce()
    • ishmem_TYPENAME_xor_reduce()
    • ishmem_TYPENAME_max_reduce()
    • ishmem_TYPENAME_max_reduce()
    • ishmem_TYPENAME_min_reduce()
    • ishmem_TYPENAME_min_reduce()
    • ishmem_TYPENAME_sum_reduce()
    • ishmem_TYPENAME_sum_reduce()
    • ishmem_TYPENAME_prod_reduce()
    • ishmem_TYPENAME_prod_reduce()
  • ISHMEMX_REDUCE_ON_QUEUE
    • ishmemx_and_reduce_on_queue()
    • ishmemx_and_reduce_on_queue()
    • ishmemx_or_reduce_on_queue()
    • ishmemx_or_reduce_on_queue()
    • ishmemx_xor_reduce_on_queue()
    • ishmemx_xor_reduce_on_queue()
    • ishmemx_max_reduce_on_queue()
    • ishmemx_max_reduce_on_queue()
    • ishmemx_min_reduce_on_queue()
    • ishmemx_min_reduce_on_queue()
    • ishmemx_sum_reduce_on_queue()
    • ishmemx_sum_reduce_on_queue()
    • ishmemx_prod_reduce_on_queue()
    • ishmemx_prod_reduce_on_queue()
    • ishmemx_TYPENAME_and_reduce_on_queue()
    • ishmemx_TYPENAME_and_reduce_on_queue()
    • ishmemx_TYPENAME_or_reduce_on_queue()
    • ishmemx_TYPENAME_or_reduce_on_queue()
    • ishmemx_TYPENAME_xor_reduce_on_queue()
    • ishmemx_TYPENAME_xor_reduce_on_queue()
    • ishmemx_TYPENAME_max_reduce_on_queue()
    • ishmemx_TYPENAME_max_reduce_on_queue()
    • ishmemx_TYPENAME_min_reduce_on_queue()
    • ishmemx_TYPENAME_min_reduce_on_queue()
    • ishmemx_TYPENAME_sum_reduce_on_queue()
    • ishmemx_TYPENAME_sum_reduce_on_queue()
    • ishmemx_TYPENAME_prod_reduce_on_queue()
    • ishmemx_TYPENAME_prod_reduce_on_queue()
  • ISHMEMX_REDUCE_WORK_GROUP
    • ishmemx_and_reduce_work_group()
    • ishmemx_and_reduce_work_group()
    • ishmemx_or_reduce_work_group()
    • ishmemx_or_reduce_work_group()
    • ishmemx_xor_reduce_work_group()
    • ishmemx_xor_reduce_work_group()
    • ishmemx_max_reduce_work_group()
    • ishmemx_max_reduce_work_group()
    • ishmemx_min_reduce_work_group()
    • ishmemx_min_reduce_work_group()
    • ishmemx_sum_reduce_work_group()
    • ishmemx_sum_reduce_work_group()
    • ishmemx_prod_reduce_work_group()
    • ishmemx_prod_reduce_work_group()
    • ishmemx_TYPENAME_and_reduce_work_group()
    • ishmemx_TYPENAME_and_reduce_work_group()
    • ishmemx_TYPENAME_or_reduce_work_group()
    • ishmemx_TYPENAME_or_reduce_work_group()
    • ishmemx_TYPENAME_xor_reduce_work_group()
    • ishmemx_TYPENAME_xor_reduce_work_group()
    • ishmemx_TYPENAME_max_reduce_work_group()
    • ishmemx_TYPENAME_max_reduce_work_group()
    • ishmemx_TYPENAME_min_reduce_work_group()
    • ishmemx_TYPENAME_min_reduce_work_group()
    • ishmemx_TYPENAME_sum_reduce_work_group()
    • ishmemx_TYPENAME_sum_reduce_work_group()
    • ishmemx_TYPENAME_prod_reduce_work_group()
    • ishmemx_TYPENAME_prod_reduce_work_group()
  • ISHMEM_INSCAN, ISHMEM_EXSCAN
    • ishmem_sum_inscan()
    • ishmem_sum_inscan()
    • ishmem_sum_exscan()
    • ishmem_sum_exscan()
    • ishmem_TYPENAME_sum_inscan()
    • ishmem_TYPENAME_sum_inscan()
    • ishmem_TYPENAME_sum_exscan()
    • ishmem_TYPENAME_sum_exscan()
  • ISHMEMX_SUM_INSCAN_ON_QUEUE, ISHMEMX_SUM_EXSCAN_ON_QUEUE
    • ishmemx_sum_inscan_on_queue()
    • ishmemx_sum_inscan_on_queue()
    • ishmemx_sum_exscan_on_queue()
    • ishmemx_sum_exscan_on_queue()
    • ishmemx_TYPENAME_sum_inscan_on_queue()
    • ishmemx_TYPENAME_sum_inscan_on_queue()
    • ishmemx_TYPENAME_sum_exscan_on_queue()
    • ishmemx_TYPENAME_sum_exscan_on_queue()
• Point-To-Point Synchronization
  • ISHMEM_WAIT_UNTIL
    • ishmem_wait_until()
    • ishmem_TYPENAME_wait_until()
  • ISHMEMX_WAIT_UNTIL_ON_QUEUE
    • ishmemx_wait_until_on_queue()
    • ishmemx_TYPENAME_wait_until_on_queue()
  • ISHMEMX_WAIT_UNTIL_WORK_GROUP
    • ishmemx_wait_until_work_group()
    • ishmemx_TYPENAME_wait_until_work_group()
  • ISHMEM_WAIT_UNTIL_ALL
    • ishmem_wait_until_all()
    • ishmem_TYPENAME_wait_until_all()
  • ISHMEMX_WAIT_UNTIL_ALL_ON_QUEUE
    • ishmemx_wait_until_all_on_queue()
    • ishmemx_TYPENAME_wait_until_all_on_queue()
  • ISHMEMX_WAIT_UNTIL_ALL_WORK_GROUP
    • ishmemx_wait_until_all_work_group()
    • ishmemx_TYPENAME_wait_until_all_work_group()
  • ISHMEM_WAIT_UNTIL_ANY
    • ishmem_wait_until_any()
    • ishmem_TYPENAME_wait_until_any()
  • ISHMEMX_WAIT_UNTIL_ANY_ON_QUEUE
    • ishmemx_wait_until_any_on_queue()
    • ishmemx_TYPENAME_wait_until_any_on_queue()
  • ISHMEMX_WAIT_UNTIL_ANY_WORK_GROUP
    • ishmemx_wait_until_any_work_group()
    • ishmemx_TYPENAME_wait_until_any_work_group()
  • ISHMEM_WAIT_UNTIL_SOME
    • ishmem_wait_until_some()
    • ishmem_TYPENAME_wait_until_some()
  • ISHMEMX_WAIT_UNTIL_SOME_ON_QUEUE
    • ishmemx_wait_until_some_on_queue()
    • ishmemx_TYPENAME_wait_until_some_on_queue()
  • ISHMEMX_WAIT_UNTIL_SOME_WORK_GROUP
    • ishmemx_wait_until_some_work_group()
    • ishmemx_TYPENAME_wait_until_some_work_group()
  • ISHMEM_WAIT_UNTIL_ALL_VECTOR
    • ishmem_wait_until_all_vector()
    • ishmem_TYPENAME_wait_until_all_vector()
  • ISHMEMX_WAIT_UNTIL_ALL_VECTOR_ON_QUEUE
    • ishmemx_wait_until_all_vector_on_queue()
    • ishmemx_TYPENAME_wait_until_all_on_vector_queue()
  • ISHMEMX_WAIT_UNTIL_ALL_VECTOR_WORK_GROUP
    • ishmemx_wait_until_all_vector_work_group()
    • ishmemx_TYPENAME_wait_until_all_vector_work_group()
  • ISHMEM_WAIT_UNTIL_ANY_VECTOR
    • ishmem_wait_until_any_vector()
    • ishmem_TYPENAME_wait_until_any_vector()
  • ISHMEMX_WAIT_UNTIL_ANY_VECTOR_ON_QUEUE
    • ishmemx_wait_until_any_vector_on_queue()
    • ishmemx_TYPENAME_wait_until_any_vector_on_queue()
  • ISHMEMX_WAIT_UNTIL_ANY_VECTOR_WORK_GROUP
    • ishmemx_wait_until_any_vector_work_group()
    • ishmemx_TYPENAME_wait_until_any_vector_work_group()
  • ISHMEM_WAIT_UNTIL_SOME_VECTOR
    • ishmem_wait_until_some_vector()
    • ishmem_TYPENAME_wait_until_some_vector_()
  • ISHMEMX_WAIT_UNTIL_SOME_VECTOR_ON_QUEUE
    • ishmemx_wait_until_some_vector_on_queue()
    • ishmemx_TYPENAME_wait_until_some_vector_on_queue()
  • ISHMEMX_WAIT_UNTIL_SOME_VECTOR_WORK_GROUP
    • ishmemx_wait_until_some_vector_work_group()
    • ishmemx_TYPENAME_wait_until_some_vector_work_group()
  • ISHMEM_TEST
    • ishmem_test()
    • ishmem_TYPENAME_test()
  • ISHMEMX_TEST_WORK_GROUP
    • ishmemx_test_work_group()
    • ishmemx_TYPENAME_test_work_group()
  • ISHMEM_TEST_ALL
    • ishmem_test_all()
    • ishmem_TYPENAME_test_all()
  • ISHMEMX_TEST_ALL_WORK_GROUP
    • ishmemx_test_all_work_group()
    • ishmemx_TYPENAME_test_all_work_group()
  • ISHMEM_TEST_ANY
    • ishmem_test_any()
    • ishmem_TYPENAME_test_any()
  • ISHMEMX_TEST_ANY_WORK_GROUP
    • ishmemx_test_any_work_group()
    • ishmemx_TYPENAME_test_any_work_group()
  • ISHMEM_TEST_SOME
    • ishmem_test_some()
    • ishmem_TYPENAME_test_some()
  • ISHMEMX_TEST_SOME_WORK_GROUP
    • ishmemx_test_some_work_group()
    • ishmemx_TYPENAME_test_some_work_group()
  • ISHMEM_TEST_ALL_VECTOR
    • ishmem_test_all_vector()
    • ishmem_TYPENAME_test_all_vector()
  • ISHMEMX_TEST_ALL_VECTOR_WORK_GROUP
    • ishmemx_test_all_vector_work_group()
    • ishmemx_TYPENAME_test_all_vector_work_group()
  • ISHMEM_TEST_ANY_VECTOR
    • ishmem_test_any_vector()
    • ishmem_TYPENAME_test_any_vector()
  • ISHMEMX_TEST_ANY_VECTOR_WORK_GROUP
    • ishmemx_test_any_vector_work_group()
    • ishmemx_TYPENAME_test_any_vector_work_group()
  • ISHMEM_TEST_SOME_VECTOR
    • ishmem_test_some_vector()
    • ishmem_TYPENAME_test_some_vector()
  • ISHMEMX_TEST_SOME_VECTOR_WORK_GROUP
    • ishmemx_test_some_vector_work_group()
    • ishmemx_TYPENAME_test_some_vector_work_group()
  • ISHMEM_SIGNAL_WAIT_UNTIL
    • ishmem_signal_wait_until()
  • ISHMEMX_SIGNAL_WAIT_UNTIL_ON_QUEUE
    • ishmemx_signal_wait_until_on_queue()
  • ISHMEMX_SIGNAL_WAIT_UNTIL_WORK_GROUP
    • ishmemx_signal_wait_until_work_group()
• Memory Ordering
  • ISHMEM_FENCE
    • ishmem_fence()
  • ISHMEMX_FENCE_WORK_GROUP
    • ishmemx_fence_work_group()
  • ISHMEM_QUIET
    • ishmem_quiet()
  • ISHMEMX_QUIET_ON_QUEUE
    • ishmemx_quiet_on_queue()
  • ISHMEMX_QUIET_WORK_GROUP
    • ishmemx_quiet_work_group()
• Utility Routines
  • ISHMEMX_PRINT_MSG_TYPE_T
    • ishmemx_print_msg_type_t
  • ISHMEMX_PRINT
    • ishmemx_print()
    • ishmemx_print()
    • ishmemx_print()

Build, Write, and Run Programs

• Building Intel® SHMEM
  • Enabling the OpenSHMEM back-end
  • Enabling the MPI back-end
  • CMake Build Options
• Writing Intel® SHMEM Programs
• Compiling and Running Programs
  • Selecting SPIR-V Compilation Targets
• Using Intel® SHMEM in Intel® Tiber™ AI Cloud

Acknowledgements / Copyright

• Acknowledgements
• Notices & Disclaimers
• Copyright