Skip to content
Neutree Documentation

Managing static node clusters

Managing static node clusters

Section titled “Managing static node clusters”

Servers or virtual machines can be added as nodes to form a static node cluster. There are two node types: head node and worker node.

  • Head node: Both a control node and a worker node. It runs management services and can also run AI workloads.
  • Worker node: Runs AI workloads only; does not run management services.

The minimum cluster size is a single node (head node only, with no worker nodes required). In this configuration, the head node runs both management services and AI workloads simultaneously.

In a multi-node cluster, it is recommended to use a node without accelerators as the head node for management services, and add nodes with accelerators as worker nodes dedicated to AI workloads.

Plan the number of worker nodes in advance based on your business requirements.

Node requirements

Section titled “Node requirements”

Ensure node configurations meet the following requirements:

  • Resource configuration

    • System disk: 200 GiB
    • CPU: at least 8-core vCPU
    • Memory: at least 16 GiB

  • OS image

    Accelerator typeOS image
    CPU or NVIDIA GPURocky-8.10-x86_64-minimal.iso
    AMD GPUUbuntu-22.04.5-live-server-amd64.iso

  • NFS client

    All nodes must have nfs-utils installed to support NFS mounting. Neutree needs to mount model directories from NFS file system model registries and download models to the local host.

  • Port requirements

    If there is a firewall between your static node cluster and Neutree, open the following ports on the target side. Unless otherwise specified, all ports listed below are TCP ports.

    Source Destination Port Purpose
    Control plane All nodes 22 Used for remote login, static node initialization, and maintenance. If the node uses a non-standard SSH port, see Using a non-standard SSH port to configure the node.
    54311 Scrape node runtime status data.
    44217 Retrieve monitoring data for auto-scaling.
    44227 Export monitoring data for dashboards.
    All nodes All nodes 10002-20000 Core channel for inter-node data exchange and distributed computing.
    8077 Used for node management.
    8076 Used for shared memory object access and distribution.
    56999 Used for managing execution environments (dependencies, etc.) on each node.
    Head node All nodes 52365, 8078 Used as the proxy for delivering dashboard-related commands.
    19100 Port for collecting node monitoring metrics.
    19101 Port for collecting metrics from Neutree monitoring components.
    19400 Port for collecting accelerator monitoring metrics.
    54311 Used to collect runtime status data from each node.
    44217 Used to obtain monitoring data related to autoscaling.
    44227 Used to export monitoring data for dashboards.
    Control plane Head node 8265, 8079 Access the graphical management interface.
    8000 Entry point for the model inference service.
    Worker node Head node 6379 Ray cluster metadata center.
    Developer Head node 10001 Allows connecting to the cluster from remote scripts to run jobs.
    All nodes Node where monitoring components are deployed 8480 Required when monitoring components are deployed on a server or VM to upload monitoring metrics.
    LoadBalancer IP allocated to monitoring components deployed on Kubernetes 8480 Required when monitoring components are deployed on a Kubernetes cluster to upload monitoring metrics.

Configuring the operating system

Section titled “Configuring the operating system”

Configure the system and install Docker as the container runtime according to the OS type of your nodes.

System configuration

Section titled “System configuration”

@tab RHEL/Rocky Linux

  1. Configure static IP addresses:

    Terminal window
    sudo vi /etc/sysconfig/network-scripts/ifcfg-<interface>

    Replace <interface> with the network interface name, for example eth0.

  2. Configure the DNS server:

    Terminal window
    sudo vi /etc/resolv.conf

  3. Disable the firewall:

    Terminal window
    sudo systemctl stop firewalld && sudo systemctl disable firewalld

  4. Disable SELinux:

    Terminal window
    echo -e "SELINUX=disabled\nSELINUXTYPE=targeted" | sudo tee /etc/selinux/config
    sudo setenforce 0

  5. Install dependencies:

    sudo dnf install rsync pciutils -y

@tab Ubuntu

  1. Configure static IP addresses and DNS server:

    Terminal window
    sudo vi /etc/netplan/50-cloud-init.yaml

  2. Apply the network configuration:

    Terminal window
    sudo netplan apply

  3. Disable the firewall:

    Terminal window
    sudo ufw disable

  4. Optional: disable AppArmor if needed:

    Terminal window
    sudo systemctl disable apparmor && sudo systemctl stop apparmor

  5. Install dependencies:

    sudo apt-get update && sudo apt-get install rsync pciutils -y

  6. Restart the OS for the configuration to take effect:

    Terminal window
    sudo reboot

Installing Docker

Section titled “Installing Docker”

@tab RHEL/Rocky Linux

  1. Install Docker CE:

    Terminal window
    sudo dnf -y install dnf-plugins-core
    sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
    sudo dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

  2. Start the Docker service:

    Terminal window
    sudo systemctl enable --now docker

  3. Confirm Docker is installed successfully:

    Terminal window
    docker --version

  4. Restart the OS for the configuration to take effect:

    Terminal window
    sudo reboot

@tab Ubuntu

  1. Update the package index:

    Terminal window
    sudo apt-get update
    sudo apt-get -y install ca-certificates curl

  2. Add the Docker GPG key:

    Terminal window
    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc

  3. Add the Docker repository:

    Terminal window
    echo \
     "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
     $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
     sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

  4. Install Docker CE:

    Terminal window
    sudo apt-get update
    sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

  5. Start the Docker service:

    Terminal window
    sudo systemctl enable --now docker

  6. Confirm Docker is installed successfully:

    Terminal window
    docker --version

Configuring accelerators

Section titled “Configuring accelerators”

If the nodes include accelerators, complete the corresponding configuration based on accelerator type:

@tab NVIDIA GPU

Follow NVIDIA’s official documentation to complete the following:

  1. Disable the NVIDIA GPU Nouveau driver.

  2. Download and install an NVIDIA Graphics driver with a version no higher than 590.x.x and no lower than 530.x.x.

  3. Install the NVIDIA Container Toolkit.

Neutree proactively installs monitoring collection components in static node clusters. If DCGM Exporter or DCGM is already deployed on static nodes, cluster monitoring components may become abnormal. Disable the existing DCGM Exporter or DCGM first. If you want Neutree to directly use the already deployed DCGM Exporter or DCGM, see Integrating external monitoring components to switch the monitoring mode.

@tab AMD GPU

The cluster image currently supports ROCM software version 6.3.3. Install the corresponding version of the AMDGPU driver and AMD Container Toolkit. See the Quick Start Guide section in the AMD Container Toolkit Documentation from AMD’s official documentation.

Preparing SSH private key

Section titled “Preparing SSH private key”

Before creating a static node cluster, you need to prepare an SSH private key for node authentication. The control plane uses the SSH private key to securely connect to and manage nodes in the cluster via the SSH protocol.

Creating an SSH key pair

Section titled “Creating an SSH key pair”

An SSH key pair consists of a public key and a private key, used for node authentication and secure communication. For security, it is recommended to use different SSH key pairs for different clusters.

If you do not have an SSH key pair, create one with the following steps:

  1. Run the following command on the control plane or a local machine to generate an SSH key pair:

    Terminal window
    ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f ~/.ssh/neutree_cluster_key

    Parameter descriptions:

    ParameterDescription
    -t rsaSpecifies the key encryption algorithm as RSA.
    -b 4096Specifies the key length as 4096 bits.
    -C "your_email@example.com"Adds a comment to the key, typically an email address.
    -f ~/.ssh/neutree_cluster_keySpecifies the save path and file name for the key.

  2. When prompted with Enter passphrase (empty for no passphrase), press Enter to leave it empty (no passphrase).

  3. After the command completes, the key pair is generated at the specified location:

    • The private key file is at ~/.ssh/neutree_cluster_key. The private key is sensitive information; keep it safe and do not share it.
    • The public key file is at ~/.ssh/neutree_cluster_key.pub. This must be configured on all nodes in the cluster.

Configuring the public key on target nodes

Section titled “Configuring the public key on target nodes”

The ~/.ssh/authorized_keys file on each node stores the public keys allowed to access it. After configuring the public key on all nodes following this section, SSH will automatically use key-based login without prompting for a password.

  1. Copy the public key content to the ~/.ssh/authorized_keys file on the target node using one of the following methods:

    @tab Automatic configuration (using ssh-copy-id)

    Terminal window
    ssh-copy-id -i ~/.ssh/neutree_cluster_key.pub <username>@<node_ip>

    @tab Manual configuration

    Terminal window
    cat ~/.ssh/neutree_cluster_key.pub | ssh <username>@<node_ip> "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
    ParameterDescription
    <username>SSH username. Must be root or a user with root privileges.
    <node_ip>The IP address of the cluster node.

  2. Set the permissions for the ~/.ssh/authorized_keys file and its parent directory on the target node:

    Terminal window
    ssh <username>@<node_ip> "chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys"

Verifying SSH connectivity

Section titled “Verifying SSH connectivity”

Run the following commands on all nodes in the cluster to test SSH connectivity and ensure each node connects successfully.

  1. Test SSH connectivity using the private key:

    Terminal window
    ssh -i ~/.ssh/neutree_cluster_key <username>@<node_ip>

    You should be able to log in without a password, indicating that the SSH key is configured correctly.

  2. For non-root users, test root privileges:

    Terminal window
    ssh -i ~/.ssh/neutree_cluster_key <username>@<node_ip> "sudo whoami"

    The expected output is root, indicating the user has sudo privileges.

Retrieving the private key content

Section titled “Retrieving the private key content”

Before creating the cluster, retrieve the private key content with the following command:

Terminal window
cat ~/.ssh/neutree_cluster_key

The private key content looks similar to:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn
...
-----END OPENSSH PRIVATE KEY-----

Creating a cluster

Section titled “Creating a cluster”

Follow the steps below to create a cluster. If the cluster nodes cannot access Docker Hub or the connection is slow, you can manually import cluster images.

  1. Log in to the Neutree management interface. Click Clusters in the left navigation pane, then click Create on the right.

  2. Fill in the configuration.

    • Basic Information

      ParameterDescriptionEditable after creation
      NameThe name of the cluster.No
      WorkspaceThe workspace to which the cluster belongs.No

    • Image Registry

      Select a container registry for the cluster to store cluster-related container images. If no registry is available, see the Creating a container registry section. If no registry is available in your environment, see the Set up a temporary container registry section. This field is not editable after creation.

    • Cluster Type

      The cluster type. Select Static Nodes. This field is not editable after creation.

    • Version

      The cluster version. The system automatically retrieves available cluster versions from the selected container registry. After creation, the version can be updated via Upgrading the cluster version.

    • Provider

      Parameter Description Editable after creation
      Head Node IP The IP address of the head node. No
      Worker Node IPs The IP address of worker nodes.
      • Not required for single-node clusters.
      • For multi-node clusters, enter an IP address and click + Add to add the next one.
      		 Yes

    • Node Authentication

      ParameterDescriptionEditable after creation
      SSH UserSSH username. Must be root or a user with root privileges.No
      SSH Private KeyThe SSH private key string. See the Preparing SSH private key section for how to obtain it.No

    • Model Caches

      ParameterDescriptionEditable after creation
      NameThe name of the model cache.No
      Cache TypeStatic node clusters only support Host Path.No
      Cache PathThe host path for the model cache.Yes

      If model cache is not configured during creation, it cannot be added after the cluster is created.

  3. After confirming the configuration is correct, click Save to complete creation.

Manually importing a cluster image

Section titled “Manually importing a cluster image”

When upgrading the cluster version or when the network environment is restricted, you can manually import the required cluster images into the Neutree container registry.

Procedure

  1. Download version 1.1.0 of the Neutree CLI tool and the cluster offline image for the required accelerator type, according to the server’s CPU architecture.

  2. Use the CLI tool to upload the cluster offline image to the specified registry:

    Terminal window
    ./neutree-cli-<arch> import cluster \
    --package <cluster_package> \
    --mirror-registry <mirror_registry> \
    [--registry-project <registry_project>] \
    --registry-username <registry_username> \
    --registry-password <registry_password>
    ParameterDescription
    <arch>The server’s CPU architecture: amd64 or arm64.
    <cluster_package>The cluster offline image name, in the format neutree-cluster-ssh-v1.1.0-<arch>.tar.gz.
    <mirror_registry>The registry address must match the registry address used when uploading images with the CLI tool during Neutree management plane deployment. Enter an OCI-compatible image registry address without the https:// prefix.
    --registry-project <registry_project>Optional. The registry project name. Ensure the corresponding project has been created in the registry in advance.
    <registry_username>The username for the registry, must have image upload permissions.
    <registry_password>The login password or access key (such as a token) for the registry user.

Viewing clusters

Section titled “Viewing clusters”

Log in to the Neutree management interface, click Clusters in the left navigation pane. The cluster list on the right shows all current clusters. Click a cluster name to view details. On the details page, you can view Basic and Monitor as needed.

The possible statuses during cluster operation and their descriptions are as follows:

StatusDescription
InitializingThe cluster is performing initial initialization.
RunningThe cluster is operating normally.
UpdatingThe cluster configuration has changed and the new configuration is being applied.
UpgradingThe cluster is undergoing a version upgrade.
FailedThe cluster is experiencing an error. Check node status, error messages, and logs.
DeletingThe cluster is being deleted; resources are being cleaned up.

If cluster monitoring or node resources show no data, first check the error message in the cluster status and confirm that the static nodes can access port 8480 of the monitoring component.

Editing a cluster

Section titled “Editing a cluster”

After a cluster is created, you can modify the worker node configuration and model cache path as needed.

  1. Log in to the Neutree management interface. In the cluster list or details page, click the menu icon (…) and select Edit.

  2. Modify the configuration as needed. For parameter descriptions, see Creating a cluster.

  3. After confirming the configuration is correct, click Save to complete the edit.

Deleting clusters

Section titled “Deleting clusters”

You can delete one or more clusters at the same time.

  1. Log in to the Neutree management interface. In the cluster list or details page, click the menu icon (…) and select Delete; or select multiple clusters in the list and click Delete above the list.

  2. In the confirmation dialog, confirm the deletion and click Delete. The selected clusters will be permanently deleted.