NVIDIA GPU Operator operations guide

By default, NVIDIA GPU Operator deploys NVIDIA Device Plugin and DCGM Exporter on GPU nodes. Depending on Neutree cluster configuration, you may need to disable one or more of these components to prevent the same GPU nodes from being managed repeatedly or having metrics collected repeatedly.

After Accelerator Virtualization is enabled, the Neutree deploys and manages the HAMi device plugin. Both the HAMi device plugin and the Device Plugin included with NVIDIA GPU Operator report nvidia.com/gpu to Kubernetes, so they cannot manage the same GPU nodes at the same time. Before enabling Accelerator Virtualization, disable the Device Plugin of NVIDIA GPU Operator.
If the Neutree already manages accelerator metric collection components, or if another DCGM Exporter is already deployed in the cluster, disable the DCGM Exporter of NVIDIA GPU Operator to avoid deploying duplicate GPU metric collection components.

Disabling NVIDIA Device Plugin

AKE clusters

If the target cluster is an AKE workload cluster, modify the NVIDIA GPU Operator addon configuration in Arcfra Operation Center.

Log in to Arcfra Operation Center.
Go to the addon management page of the target AKE workload cluster.
Edit the NVIDIA GPU Operator addon configuration.
Disable NVIDIA Device Plugin.
```
devicePlugin:
  enabled: false
```
Save the configuration and wait for the addon update to complete.

In the cluster, confirm that the NVIDIA GPU Operator Device Plugin DaemonSet has stopped or is no longer scheduled to GPU nodes.

kubectl get clusterpolicy
kubectl get ds -A | grep -i nvidia-device-plugin
kubectl describe node <gpu-node> | grep -A5 nvidia.com/gpu

Self-managed Kubernetes clusters

If NVIDIA GPU Operator is installed by using Helm, keep the existing configuration and disable only NVIDIA Device Plugin:

helm upgrade gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  --reuse-values \
  --set devicePlugin.enabled=false

If you are installing NVIDIA GPU Operator for the first time, disable NVIDIA Device Plugin during installation:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update nvidia

helm install gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  --create-namespace \
  --set devicePlugin.enabled=false

If NVIDIA GPU Operator is not managed by Helm, modify the ClusterPolicy of GPU Operator to disable spec.devicePlugin.enabled:

kubectl get clusterpolicy

kubectl patch clusterpolicy <cluster-policy-name> \
  --type=merge \
  -p '{"spec":{"devicePlugin":{"enabled":false}}}'

Disabling DCGM Exporter

AKE clusters

If the target cluster is an AKE workload cluster, modify the NVIDIA GPU Operator addon configuration in Arcfra Operation Center.

Log in to Arcfra Operation Center.
Go to the addon management page of the target AKE workload cluster.
Edit the NVIDIA GPU Operator addon configuration.
Disable DCGM Exporter.
```
dcgmExporter:
  enabled: false
```
Save the configuration and wait for the addon update to complete.
In the cluster, confirm that the NVIDIA GPU Operator DCGM Exporter DaemonSet has stopped or is no longer scheduled to GPU nodes.
Terminal window
```
kubectl get clusterpolicy
kubectl get ds -A | grep -i nvidia-dcgm-exporter
kubectl get pods -A | grep -i nvidia-dcgm-exporter
```

Self-managed Kubernetes clusters

If NVIDIA GPU Operator is installed by using Helm, keep the existing configuration and disable only DCGM Exporter:

helm upgrade gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  --reuse-values \
  --set dcgmExporter.enabled=false

If you are installing NVIDIA GPU Operator for the first time, disable DCGM Exporter during installation:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update nvidia

helm install gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  --create-namespace \
  --set dcgmExporter.enabled=false

If NVIDIA GPU Operator is not managed by Helm, modify the ClusterPolicy of GPU Operator to disable spec.dcgmExporter.enabled:

kubectl get clusterpolicy

kubectl patch clusterpolicy <cluster-policy-name> \
  --type=merge \
  -p '{"spec":{"dcgmExporter":{"enabled":false}}}'

Verifying the configuration

After disabling these components, confirm that NVIDIA official device plugin and HAMi device plugin no longer exist on the same GPU nodes at the same time, and confirm that DCGM Exporter managed by NVIDIA GPU Operator is no longer running.

kubectl get ds -A | grep -E 'nvidia-device-plugin|nvidia-dcgm-exporter'
kubectl get pods -A | grep -E 'nvidia-device-plugin|nvidia-dcgm-exporter'
kubectl describe node <gpu-node> | grep -A5 nvidia.com/gpu

If the nvidia-device-plugin or nvidia-dcgm-exporter query command still returns results, check whether the AKE addon configuration, Helm values, or ClusterPolicy has taken effect.

If NVIDIA GPU Operator still needs to manage components such as NVIDIA drivers, Container Toolkit, and MIG Manager, disable only devicePlugin.enabled and dcgmExporter.enabled. Do not uninstall the entire NVIDIA GPU Operator.