Unable to view GPU monitoring metrics when using the AKE cluster

Issue

In an AKE workload cluster, if NVIDIA drivers are installed through the GPU addon and the GPU nodes do not have NVSwitch, the DCGM Exporter managed by the Neutree may fail to complete initialization. As a result, it does not start the /metrics service, and Neutree cannot collect or display GPU metrics.

You can check the DCGM Exporter Pod logs to determine whether this issue has occurred.

Normally, the following log output indicates that DCGM Exporter has completed initialization and started serving metrics:

Registry built successfully
HTTP server started - ready to serve metrics
Listening on [::]:19400

If the logs remain at the following initialization stage and the preceding log output does not appear, DCGM Exporter may not have finished starting:

Initializing system entities of type 'CPU Core'
Not collecting CPU Core metrics; error retrieving DCGM CPU hierarchy: This request is serviced by a module of DCGM that is not currently loaded

Cause

The DCGM Exporter managed by the Neutree collects some NVSwitch-related metrics by default, such as DCGM_FI_DEV_NVSWITCH_LINK_STATUS. On nodes without NVSwitch, these metrics have no practical use, but DCGM may still try to call NSCQ-related libraries to query the NVSwitch status.

If the AKE workload cluster installs NVIDIA drivers through the GPU addon, the nodes usually include libnvidia-nscq.so. When DCGM tries to query the NVSwitch status, it may load this library and enter the NSCQ query path. With some combinations of DCGM, NVIDIA driver, and NSCQ versions, the query may hang. This prevents DCGM Exporter from completing initialization and starting the /metrics service.

Solution

Option 1: Temporarily integrate the AKE DCGM Exporter

If the DCGM Exporter installed by the AKE GPU addon can provide metrics normally, you can temporarily switch Neutree to external accelerator metric component mode. After the switch, the Neutree no longer deploys or collects metrics from its own managed DCGM Exporter. Instead, it collects the Prometheus metrics exposed by the AKE DCGM Exporter.

For how to switch, see Integrating external monitoring components.

After the switch, check whether the GPU-related monitoring dashboards in the Neutree return to normal.

Option 2: Temporarily disable NSCQ in AKE

If no DCGM Exporter is running normally in the environment, you can temporarily disable the NSCQ library that is installed by default when the AKE workload cluster installs NVIDIA drivers on the affected nodes without NVSwitch, and then refresh the dynamic linker cache in the NVIDIA driver root. After NSCQ is disabled, DCGM no longer loads the NSCQ-related libraries to query the NVSwitch status, avoiding the DCGM Exporter hang during initialization.

Procedure

Run the following commands on the affected node:

DRIVER_ROOT=/run/nvidia/driver
LIBDIR="${DRIVER_ROOT}/usr/lib64"
BACKUP="${LIBDIR}/.nscq-disabled-$(date +%Y%m%d%H%M%S)"

sudo mkdir -p "${BACKUP}"
sudo mv "${LIBDIR}"/libnvidia-nscq.so* "${BACKUP}/"
sudo chroot "${DRIVER_ROOT}" ldconfig
sudo chroot "${DRIVER_ROOT}" ldconfig -p | grep -i nscq || true

Restart the DCGM Exporter Pod managed by the Neutree on the affected node:
Terminal window
```
kubectl -n <namespace> delete pod <dcgm-exporter-pod>
```
In the command, <namespace> is the namespace of the DCGM Exporter Pod, and <dcgm-exporter-pod> is the name of the DCGM Exporter Pod to restart.
Check whether the GPU-related monitoring dashboards in the Neutree return to normal.

Related operation

If you need to restore the NSCQ library later, run the following commands on the node:

DRIVER_ROOT=/run/nvidia/driver
LIBDIR="${DRIVER_ROOT}/usr/lib64"
BACKUP=<backup-dir>

sudo mv "${BACKUP}"/libnvidia-nscq.so* "${LIBDIR}/"
sudo chroot "${DRIVER_ROOT}" ldconfig

In the commands, <backup-dir> is the BACKUP directory that was set when NSCQ library was disabled.