SSH Non-Standard Port Configuration

Problem

When managing Static Node clusters, Neutree uses SSH to connect to nodes. By default, SSH connections use port 22. If your nodes use a non-standard SSH port, you need to configure SSH settings manually.

Solution

Since Ray (the underlying cluster management tool) does not support specifying SSH ports directly in its configuration, you need to use the SSH config file to define custom ports for each host.

Step 1: Create SSH Config File

Create an SSH config file that will be mounted into the Neutree container:

mkdir -p ./ssh-config
vim ./ssh-config/config

Step 2: Add Host Configuration

Add an entry for each node that uses a non-standard SSH port:

Host node1
    HostName 192.168.1.100
    Port 2222
    User ubuntu

Host node2
    HostName 192.168.1.101
    Port 2222
    User ubuntu

Step 3: Use Host Alias in Neutree

When adding nodes to your Static Node cluster in Neutree, use the Host alias (e.g., node1, node2) instead of the IP address directly.

Example

If you have a node at 192.168.1.100 with SSH running on port 2222:

1. Add to ./ssh-config/config:

   Host gpu-node-1
       HostName 192.168.1.100
       Port 2222
       User root

2. Mount the config file into the container (see Persistence in Different Deployments below).

3. In Neutree, when configuring the Static Node cluster, enter gpu-node-1 as the node address.

Persistence in Different Deployments

The SSH config file needs to be mounted into the neutree-core container.

Docker Compose

1. Create the SSH config file on the host:

   mkdir -p ./ssh-config
   cat > ./ssh-config/config << 'EOF'
   Host gpu-node-1
       HostName 192.168.1.100
       Port 2222
       User root
   EOF
   chmod 600 ./ssh-config/config

2. Update your docker-compose.yaml to mount the config:

   services:
     neutree-core:
       # ... other configurations
       volumes:
         - ./ssh-config:/root/.ssh:ro

3. Restart the service:

   docker compose up -d

Kubernetes

1. Create a ConfigMap with the SSH config:

   kubectl create configmap ssh-config --from-file=config=./ssh-config -n neutree

   Or using YAML:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ssh-config
     namespace: neutree
   data:
     config: |
       Host gpu-node-1
           HostName 192.168.1.100
           Port 2222
           User root

2. Update the neutree-core Deployment to mount the ConfigMap:

   spec:
     template:
       spec:
         containers:
           - name: neutree-core
             # ... other configurations
             volumeMounts:
               - name: ssh-config
                 mountPath: /root/.ssh
                 readOnly: true
         volumes:
           - name: ssh-config
             configMap:
               name: ssh-config
               defaultMode: 0600

3. Apply the changes:

   kubectl apply -f neutree-core-deployment.yaml -n neutree

Notes

Caution

This is a workaround using SSH config, not a built-in Neutree feature. You need to ensure the SSH config is properly set up and mounted into the container before adding nodes.

Note

When upgrading the Neutree control plane, your custom volume mounts may be overwritten. After upgrading, remember to re-apply the SSH config mount in your docker-compose.yaml or neutree-core Deployment YAML.

• Test the connection manually before adding to Neutree: ssh gpu-node-1
• For more context on this limitation, see Ray Discuss: Specify SSH port to cluster YAMLs