Managing Kubernetes clusters

Managing Kubernetes clusters

Node requirements

All nodes in the Kubernetes cluster must have nfs-utils installed to support NFS mounting. Neutree needs to mount model directories from NFS file system model registries and download models to the local host.

Port requirements

If there is a firewall between your standard Kubernetes cluster and the Neutree management plane, open the corresponding ports on the destination side according to the following list to ensure Neutree can manage your Kubernetes cluster. Unless otherwise specified, all ports listed below are TCP ports.

Source	Destination	Port	Purpose
Control plane	Kubernetes cluster nodes	6443	Manage and deploy Neutree cluster components and inference endpoints.
	LoadBalancer IP associated with Kubernetes cluster nodes	8000	Forward requests from the AI gateway to the specific cluster.
	Control Plane virtual IP of Kubernetes cluster nodes	Assigned NodePort	Forward requests from the AI gateway to the specific cluster.
Kubernetes cluster nodes	Node where monitoring components are deployed	8480	Required when monitoring components are deployed on a server or VM to upload monitoring metrics.
	LoadBalancer IP allocated to monitoring components deployed on Kubernetes	8480	Required when monitoring components are deployed on a Kubernetes cluster to upload monitoring metrics.

Configuring accelerators

If the Kubernetes nodes include accelerators, complete the corresponding configuration based on the accelerator type.

@tab NVIDIA GPU

When used with AKE, enable the NVIDIA GPU Operator plugin for the AKE workload cluster in Arcfra Operation Center. For details, see the Configuring cluster addons section in the Arcfra Kubernetes Engine Administration Guide.

When used with other standard Kubernetes clusters, refer to the Installing the NVIDIA GPU Operator section in NVIDIA GPU Operator from NVIDIA’s official documentation.

To enable GPU virtualization capabilities for slicing the GPU memory and compute power of a single GPU across multiple endpoints, disable Device Plugin first. For more information, see NVIDIA GPU Operator operations guide to disable NVIDIA Device Plugin.

Neutree proactively installs monitoring collection components in compute clusters. If DCGM Exporter or DCGM is already deployed in the Kubernetes cluster, cluster monitoring components may become abnormal. Disable the existing DCGM Exporter or DCGM first. If DCGM Exporter is managed by NVIDIA GPU Operator, see the procedure for disabling DCGM Exporter in NVIDIA GPU Operator operations guide. If you want Neutree to directly use the already deployed DCGM Exporter or DCGM, see Integrating external monitoring components to switch the monitoring mode.

@tab AMD GPU

Refer to AMD’s official documentation to complete the following:

1. The cluster image currently supports ROCM software version 6.3.3. Install the corresponding version of the AMDGPU driver. See the Install AMD GPU Driver section in the AMD GPU Driver (amdgpu).

2. The cluster image currently supports ROCM software version 6.3.3. Install the corresponding version of the AMD GPU Device Plugin. See the AMD GPU Device Plugin for Kubernetes section in the Device Plugin Documentation.

Creating a cluster

Follow the steps below to create a cluster. If the cluster nodes cannot access Docker Hub or the connection is slow, you can manually import cluster images.

1. Log in to the Neutree management interface, click Clusters in the left navigation pane, then click Create on the right.

2. Fill in the configuration.

   • Basic Information

     Parameter	Description	Editable after creation
     Name	The name of the cluster.	No
     Workspace	The workspace to which the cluster belongs.	No

   • Image Registry

     Select a container registry for the cluster to store cluster-related container images. If no registry is available, see Creating a container registry; if no registry is available in your environment, see Setting up a temporary container registry. This field is not editable after creation.

   • Cluster Type

     The cluster type. Select Kubernetes. Not editable after creation.

   • Version

     The cluster version. The system automatically retrieves available versions from the selected registry. Can be updated after creation via Upgrading the cluster version.

   • Provider

     Enter the Kubeconfig string for the cluster to access the Kubernetes cluster. Not editable after creation.

   • Router

     Parameter	Description	Editable after creation
     Access Mode	Routing component access mode: LoadBalancer or NodePort. When selecting LoadBalancer, ensure the Kubernetes cluster supports LoadBalancer services.	Yes
     Replicas	Number of replicas for the routing component. Recommended: at least 2 for high availability.	Yes
     CPU	Number of CPUs for the routing component. Recommended: 0.5 cores.	Yes
     Memory	Memory capacity for the routing component. Recommended: 1 GiB.	Yes

   • Accelerator Virtualization

     After accelerator virtualization is enabled, the Neutree deploys and manages the HAMi component in the Kubernetes cluster to support GPU virtualization capabilities for NVIDIA GPUs. You can edit this setting after the cluster is created.

     To adjust the HAMi scheduling policy, see Adjusting the HAMi scheduling policy through config_patch.

   • Model Caches

     Parameter	Description	Editable after creation
     Name	The name of the model cache.	No
     Cache Type	Supported cache types: Host Path: Local cache. NFS: NFS cache. PVC: Persistent storage, ReadWriteMany only.	Yes
     Cache Path	The path for model caching. When cache type is Host Path, specify the host path. When cache type is NFS, specify the NFS server path. When cache type is PVC, this field is not required.	Yes
     NFS Server Address	The IP address or domain name of the NFS server. Required only when cache type is NFS.	Yes
     Storage	Specify the storage capacity for model caching. Required only when cache type is PVC.	Yes
     Storage Class Name	Specify the storage class name for model caching. Required only when cache type is PVC.	No

     If model cache is not configured during creation, it can be added after the cluster is created.

3. After confirming the configuration is correct, click Save to complete creation.

Manually importing a cluster image

When upgrading the cluster version or when the network environment is restricted, you can manually import the required cluster images into the Neutree container registry.

Procedure

1. Download version 1.1.0 of the Neutree CLI tool and the cluster offline image for the specified accelerator type, based on the server CPU architecture.

2. Upload the cluster offline image to the specified registry using the CLI tool:

   ./neutree-cli-<arch> import cluster \
   --package <cluster_package> \
   --mirror-registry <mirror_registry> \
   [--registry-project <registry_project>] \
   --registry-username <registry_username> \
   --registry-password <registry_password>

   Parameter	Description
   <arch>	CPU architecture of the server: amd64 or arm64.
   <cluster_package>	Cluster offline image name, format: neutree-cluster-k8s-v1.1.0-<arch>.tar.gz.
   <mirror_registry>	Registry address. Must match the address used when uploading images with the CLI tool during Neutree management plane deployment. Enter an OCI-compatible registry address without the https:// prefix.
   --registry-project <registry_project>	Optional. Registry project name. Ensure the corresponding project has been pre-created in the registry.
   <registry_username>	Registry username. Must have upload permissions.
   <registry_password>	Registry password or access key (such as a token).

Viewing clusters

Log in to the Neutree management interface, click Clusters in the left navigation pane. The cluster list on the right shows all current clusters. Click a cluster name to view details. On the details page, you can view Basic and Monitor as needed.

The possible statuses during cluster operation and their descriptions are as follows:

Status	Description
Initializing	The cluster is performing initial initialization.
Running	The cluster is operating normally.
Updating	The cluster configuration has changed and the new configuration is being applied.
Upgrading	The cluster is undergoing a version upgrade.
Failed	The cluster is experiencing an error. Check node status, error messages, and logs. If the routing, monitoring, or accelerator virtualization component is not ready, the error message may include the status reason of the related Deployment, Pod, Event, Service, or HAMi.
Deleting	The cluster is being deleted; resources are being cleaned up.

Editing a cluster

After creation, you can modify the routing, accelerator virtualization, and model cache configuration of the cluster as needed.

1. Log in to the Neutree management interface, click the menu icon (…) in the cluster list or details page, and select Edit.

2. On the configuration page, modify as needed. For parameter descriptions, see Creating a cluster.

3. After confirming the configuration is correct, click Save to complete editing.

Note

If there are still endpoints using GPU virtualization resources in the current cluster, you cannot disable Accelerator Virtualization for the cluster. Delete the related endpoints first, then disable Accelerator Virtualization.

Deleting clusters

You can delete one or more clusters at a time.

1. Log in to the Neutree management interface, click the menu icon (…) in the cluster list or details page, and select Delete; or select multiple clusters in the list and click Delete above the list.

2. In the dialog that appears, confirm again and click Delete. The selected clusters will be permanently deleted.