vLLM v0.24.0 reports "NVIDIA driver too old" during startup

Issue

When you use the vLLM v0.24.0 engine to create an endpoint, EngineCore fails to start and the following error appears in the logs:

RuntimeError: The NVIDIA driver on your system is too old (found version 12080).
Please update your GPU driver by downloading and installing a new version...

This issue usually occurs in the following environment:

• The engine version is vLLM v0.24.0.
• The NVIDIA Graphics driver version on the node is earlier than 580.
• The NVIDIA Container Toolkit version on the node is earlier than 1.17.8, or CUDA compat library priority loading is not correctly enabled.
• CUDA Forward Compatibility environment variables are not configured when the endpoint starts.

Cause

The vLLM v0.24.0 image is based on CUDA 13.0 / PyTorch cu130. Native CUDA 13.0 runtime requires NVIDIA Graphics driver 580 or later.

If the NVIDIA Graphics driver version on the node is earlier than 580, the endpoint needs to load the CUDA compatibility library /usr/local/cuda-13.0/compat/libcuda.so.1 in the image first. This compatibility library allows CUDA 13 applications to run on earlier driver versions.

When the container runtime or endpoint environment does not load this compatibility library first, the process loads the older libcuda.so.1 injected from the host. In this case, the CUDA driver API version detected by PyTorch remains too low, and found version 12080 is reported, causing EngineCore startup to fail.

Solution

Select a solution based on the scope of the environment change.

• Option 1: Configure environment variables changes only the current endpoint, so that vLLM enables the CUDA compatibility library during startup. This has the smallest impact scope and is suitable for temporary recovery.

• Option 2: Upgrade NVIDIA Container Toolkit fixes the CUDA compat library loading order at the node-level container runtime. This affects GPU containers on the node and requires a maintenance window.

• Option 3: Upgrade NVIDIA Graphics driver upgrades the node driver to a version that natively supports CUDA 13 and does not rely on compat library as a workaround. This is the long-term solution.

Option 1: Configure environment variables

This option affects only the current vLLM endpoint and does not change the node-level NVIDIA Container Runtime configuration. It is suitable as a temporary recovery solution.

1. Log in to the Neutree management interface.

2. Select the target endpoint in the endpoint list, or click the menu icon (…) on the target endpoint details page and select Edit.

3. On the configuration information page, click Configuration Details and add the following environment variable:

   • Key: VLLM_ENABLE_CUDA_COMPATIBILITY
   • Value: 1

4. Click Save to complete the edit.

Note

If the compat library in the image is not in the default path, or if VLLM_ENABLE_CUDA_COMPATIBILITY=1 has already been configured but troubleshooting shows that the program still does not load the compat library first, add the following environment variable:

• Key: VLLM_CUDA_COMPATIBILITY_PATH
• Value: /usr/local/cuda-13.0/compat

Option 2: Upgrade NVIDIA Container Toolkit

This option changes the node-level NVIDIA Container Runtime behavior and may affect other GPU containers on the same node. Perform the operation within a maintenance window and prepare a rollback plan in advance.

1. Upgrade NVIDIA Container Toolkit on the node to 1.18.1, or at least to 1.17.8 or later.

   For offline environments, see Installing NVIDIA Container Toolkit offline.

2. Confirm that the following configuration exists in /etc/nvidia-container-runtime/config.toml on the node:

   [nvidia-container-runtime.modes.legacy]
   cuda-compat-mode = "ldconfig"

3. If the configuration does not exist, run the following command to add it:

   sudo nvidia-ctk config --in-place \
     --set nvidia-container-runtime.modes.legacy.cuda-compat-mode=ldconfig

4. Restart the related services based on the container runtime type of the node.

   • Docker environment:

     sudo systemctl restart docker

   • Kubernetes / containerd environment:

     sudo systemctl restart containerd
     sudo systemctl restart kubelet

Option 3: Upgrade NVIDIA Graphics driver

As a long-term solution, upgrade the NVIDIA Graphics driver on the node to version 580 or later. Driver 580 or later natively meets CUDA 13 runtime requirements and can reduce the dependency on CUDA Forward Compatibility and the NVIDIA Container Toolkit compat hook.

Verification

Verify the NVIDIA Container Toolkit version

Run the following command on the node to confirm that the NVIDIA Container Toolkit version is 1.17.8 or later. Version 1.18.1 or later is recommended:

nvidia-ctk --version

Verify the compat library loading order

• After entering the endpoint container, run the following command:

  ldconfig -p | grep libcuda.so.1

  If /usr/local/cuda-13.0/compat/libcuda.so.1 appears before the host libcuda.so.1 in the output, the loader cache order is correct. See the following example:

  libcuda.so.1 => /usr/local/cuda-13.0/compat/libcuda.so.1
  libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1

• Run the following command to verify which libcuda.so.1 is actually loaded by the Python process:

  LD_DEBUG=libs python -c 'import torch; torch.ones((1,), device="cuda")' 2>&1 | grep libcuda

  The expected output contains the following path:

  /usr/local/cuda-13.0/compat/libcuda.so.1

Verify the endpoint status

Restart the endpoint. After the endpoint enters the running state, send an inference request and confirm that the request returns normally.

Related links

• NVIDIA CUDA Forward Compatibility
• NVIDIA CUDA Minor Version Compatibility
• NVIDIA Container Toolkit Release Notes
• Installing NVIDIA Container Toolkit offline