HAMi Scheduler not ready after upgrading or updating Accelerator Virtualization

Issue

In a Kubernetes cluster with only one schedulable GPU worker node, HAMi Scheduler might fail to complete a rolling update when you upgrade the Neutree control plane from an earlier version or update Accelerator Virtualization after disabling it. In this case, the cluster status might become Failed, and the Accelerator Virtualization component status might become NotReady.

You might observe the following symptoms:

• The hami-scheduler Deployment has both an old Ready Pod and a new Pending Pod.
• Scheduling events for the new hami-scheduler Pod include failure reasons related to Pod anti-affinity.

Cause

The hami-scheduler Pod in affected earlier versions uses a hard Pod anti-affinity rule. In a cluster with only one schedulable GPU worker node, where the control-plane node is unschedulable or has taints that are not tolerated, the new Pod might fail to be scheduled after the old Pod occupies the only schedulable node. As a result, the rolling update cannot converge.

The current version has adjusted the rolling update strategy and Pod anti-affinity rule of HAMi Scheduler. After you perform one manual recovery and the new HAMi Scheduler version takes effect, subsequent upgrades or Accelerator Virtualization updates no longer require this compatibility workaround for earlier versions.

Solution

Delete the old hami-scheduler Pod.

Note:

Deleting the hami-scheduler Pod temporarily affects Accelerator Virtualization scheduling. Perform this operation during a maintenance window, and confirm that no virtual-card endpoints are being created or changed.

Prerequisites

• You have obtained access to the target Kubernetes cluster.
• You have confirmed that the issue occurs in a cluster with only one schedulable GPU worker node.
• You have confirmed that the new hami-scheduler Pod is in the Pending state due to the Pod anti-affinity rule.

Procedure

1. Get the namespace of the cluster:

   kubectl get ns \
     -l neutree.ai/neutree-cluster=<cluster-name>,neutree.ai/neutree-workspace=<workspace-name>

   Parameter	Description
   <cluster-name>	Cluster name.
   <workspace-name>	Name of the workspace to which the cluster belongs.

2. Check the status of the hami-scheduler Pod:

   kubectl -n <namespace> get pod \
     -l app.kubernetes.io/component=hami-scheduler \
     -o wide

3. Check the scheduling events of the Pending Pod, and confirm that the failure reason is related to Pod anti-affinity:

   kubectl -n <namespace> describe pod <pending-hami-scheduler-pod>

4. Delete the hami-scheduler Pod:

   kubectl -n <namespace> delete pod \
     -l app.kubernetes.io/component=hami-scheduler

   The Kubernetes Deployment automatically recreates the hami-scheduler Pod.

Validation

1. Confirm that the hami-scheduler Deployment is ready:

   kubectl -n <namespace> rollout status deployment/hami-scheduler

2. Confirm that the hami-scheduler Pod is in the Running state:

   kubectl -n <namespace> get pod \
     -l app.kubernetes.io/component=hami-scheduler

3. In the Neutree management interface, check the cluster status and confirm that the cluster has recovered to Running and that the Accelerator Virtualization component has recovered to Ready.