Mailing List Archive

Initially, rocm.eclass append xnack[1,2] feature flag to gfx9 GPUs,
since ROCm upstream does this in many of their math libraries, e.g.
rocBLAS [3]. The list includes gfx90a:xnack+, indicating xnack is usable
for MI200 series, thus rocm.eclass append :xnack+ to gfx90a.

But it turns out xnack- is also common for MI200 series, restricting to
xnack+ produces incompatible GPU kernel with xnack- mode.

Also, community also explores using xnack on other gfx9 GPU [4,5], which
is previously restricted to xnack- in rocm.eclass.

By not appending xnack feature flag, GPU kernels are compiled to "xnack
any" mode, which can be run in either mode, potentially scarifying some
performance [6,7], with no direct evidence. rocFFT reports no
performance penalty[8].

For the reason above, do not append xnack feature flag to AMDGPU_TARGETS,
which is compatible with GPUs operate in both xnack mode.

[1] https://wiki.gentoo.org/wiki/ROCm#XNACK_target_feature
[2] https://rocm.docs.amd.com/en/latest/conceptual/gpu-memory.html#xnack
[3] https://github.com/ROCm/rocBLAS/blob/release/rocm-rel-5.0/CMakeLists.txt#L201
[4] https://niconiconi.neocities.org/tech-notes/xnack-on-amd-gpus/
[5] https://arxiv.org/abs/2401.02680
[6] https://llvm.org/docs/AMDGPUUsage.html#target-features
[7] https://docs.olcf.ornl.gov/systems/crusher_quick_start_guide.html#compiling-hip-kernels-for-specific-xnack-modes
[8] https://github.com/ROCm/rocFFT/commit/cd2689360ba3b3579d044d8925838ff307b4b4cf

Signed-off-by: Yiyang Wu <xgreenlandforwyy@gmail.com>
---
eclass/rocm.eclass | 19 ++-----------------
1 file changed, 2 insertions(+), 17 deletions(-)

diff --git a/eclass/rocm.eclass b/eclass/rocm.eclass
index 9804ecde97d0..e03e8bdd507a 100644
--- a/eclass/rocm.eclass
+++ b/eclass/rocm.eclass
@@ -1,4 +1,4 @@
-# Copyright 2022-2023 Gentoo Authors
+# Copyright 2022-2024 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

# @ECLASS: rocm.eclass
@@ -201,22 +201,7 @@ unset -f _rocm_set_globals
# Append default target feature to GPU arch. See
# https://llvm.org/docs/AMDGPUUsage.html#target-features
get_amdgpu_flags() {
- local amdgpu_target_flags
- for gpu_target in ${AMDGPU_TARGETS}; do
- local target_feature=
- case ${gpu_target} in
- gfx906|gfx908)
- target_feature=:xnack-
- ;;
- gfx90a)
- target_feature=:xnack+
- ;;
- *)
- ;;
- esac
- amdgpu_target_flags+="${gpu_target}${target_feature};"
- done
- echo "${amdgpu_target_flags}"
+ echo $(printf "%s;" ${AMDGPU_TARGETS[@]})
}

# @FUNCTION: check_amdgpu
--
2.41.0