Skip to content

Conversation

@rjrock
Copy link
Contributor

@rjrock rjrock commented Nov 26, 2025

Purpose

This PR skips tests with ray as the distributed executor backend on ROCm, in order to make the AMD run of the testgroup Distributed Tests (2 GPUs) slightly more successful.

Setting ray as distributed backend executor in vLLM fails on ROCm due to how ray manages GPU resources, how vLLM allocates resources via ray and how importing torch on ROCm caches the value of CUDA_VISIBLE_DEVICES. Ray manages GPU resources per actor with CUDA_VISIBLE_DEVICES, but vLLM doesn't bundle a GPU resource with a CPU resource when spawning a DPEngineCoreActor and consequently must set that value at runtime. Unfortunately for the ROCm flow, CUDA_VISIBLE_DEVICES="" is cached upon importing torch and subsequent value changes won't convince torch that any GPUs are available on the system, resulting in an error.

(TemporaryActor pid=3615141) RuntimeError: No HIP GPUs are available

Setting RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 such that ray won't set CUDA_VISIBLE_DEVICES="" for GPU-less actors also fails due to collisions.

NCCL WARN Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 75000

Test Plan

In the tests directory using the ROCm docker image, TP_SIZE=1 DP_SIZE=2 pytest -s -v v1/distributed/test_async_llm_dp.py::test_load.

Test Result

Status Before After
Summary 4 failed, 8 passed, 4 skipped, 3 warnings in 213.90s 8 passed, 8 skipped, 3 warnings in 180.29s

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: Ryan Rock <ryan.rock@amd.com>
@mergify mergify bot added rocm Related to AMD ROCm v1 labels Nov 26, 2025
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a CI failure on ROCm platforms when using Ray as the distributed executor backend. The change correctly identifies this specific configuration within the test_load test in tests/v1/distributed/test_async_llm_dp.py and skips it using pytest.skip. The implementation is straightforward and effectively resolves the issue described. The code is clean and the change is well-contained. I have no further comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rocm Related to AMD ROCm v1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant