Managing engines

Neutree provides built-in engines, and also supports creating custom engines via the CLI tool, adding new versions to built-in engines, and deleting engine versions.

Current built-in engines:

Name	Version	Description
vllm	v0.8.5	vLLM community version v0.8.5.
	v0.11.2	vLLM community version v0.11.2.
	v0.17.1	vLLM community version v0.17.1.
	v0.24.0	vLLM community version v0.24.0. This is the default version for both Kubernetes clusters and static node clusters.
sglang	v0.5.10	SGLang community version v0.5.10.
llama-cpp	v0.3.7	Llama-cpp Python high-level implementation (llama-cpp commit: 794fe23f29fb40104975c91fe19f23798f7c726e).

Viewing engines

Log in to the Neutree management interface. Click Engines in the left navigation pane. The engine list on the right shows all built-in engines. Click an engine name to enter the details page and view the task types and parameters supported by the engine.

Importing an engine

You can create an engine via the CLI tool, or add a new version to an existing engine.

Procedure

Create an API key and save it securely.
Download version 1.1.0 of the Neutree CLI tool and the specified engine package for your server’s CPU architecture.

Import the engine using the CLI tool:

@tab Import an engine package

In offline scenarios, cluster nodes cannot access Docker Hub or other external image registries. You need to import the complete engine metadata to Neutree and import the engine image to an internal image registry. Run the following command to import the engine package and upload the images in the package to the internal image registry.

./neutree-cli-<arch> import engine \
--package <engine_version_package> \
--mirror-registry <mirror_registry> \
[--registry-project <registry_project>] \
--registry-username <registry_username> \
--registry-password <registry_password> \
--api-key <api_key> \
--server-url <server_url>

Parameter	Description
`<arch>`	Replace with the server’s CPU architecture: `amd64` or `arm64`.
`<engine_version_package>`	The engine package name, for example `vllm-v0.8.5.tar.gz`.
`<mirror_registry>`	The registry address. Must match the registry address used when uploading images with the CLI tool during Neutree management plane deployment. Enter an OCI-compatible registry address without the `https://` prefix.
`[--registry-project <registry_project>]`	Optional. The project name in the registry. The project must be created in advance. For example, specifying `--registry-project neutree-ai` will push the image to `<mirror_registry>/neutree-ai/vllm/vllm-openai:v0.XX.XX`.
`<registry_username>`	The registry username. Must have permission to push images.
`<registry_password>`	The registry user’s login password or access token (for example, a token).
`<api_key>`	The API key created in step 1.
`<server_url>`	The address of the control plane, for example `http://localhost:3000`.

@tab Import engine metadata

Published engine images are uploaded to Docker Hub by default. If your environment can access Docker Hub and Neutree uses Docker Hub as the ImageRegistry, you can import only the engine metadata without importing the engine image.

./neutree-cli-<arch> import engine --skip-image-push \
--package <manifest.yaml> \
--api-key <api_key> \
--server-url <server_url>

Parameter	Description
`<arch>`	Replace with the server’s CPU architecture: `amd64` or `arm64`.
`<manifest.yaml>`	The path to the manifest.yaml file, which contains the engine metadata.
`<api_key>`	The API key created in step 1.
`<server_url>`	The address of the control plane, for example `http://localhost:3000`.

After importing, log in to the Neutree management interface. Click Engines in the left navigation pane and confirm that the new engine or engine version appears in the engine list.

Deleting an engine version

You can delete engine versions using the CLI tool. When only one version remains, use the --force parameter to force-delete that version and remove the entire engine at the same time.

Prerequisites

Confirm that the engine version is not currently in use by any endpoint.

Procedure

Run the following command:

./neutree-cli-<arch> engine remove-version \
  --name <engine_name> \
  --version <engine_version> \
  --api-key <api_key> \
  --server-url <server_url>

Parameter descriptions:

Parameter	Description
`<arch>`	Replace with the server’s CPU architecture: `amd64` or `arm64`.
`<engine_name>`	The name of the engine, for example `vllm`.
`<engine_version>`	The version to delete, for example `v0.8.5`.
`<api_key>`	The API key created in the prerequisites.
`<server_url>`	The address of the control plane, for example `http://localhost:3000`.
`--force`	Optional. Required when only one version remains — force-deletes that version and removes the entire engine.