Managing engines

Neutree provides built-in engines, and also supports creating custom engines via the CLI tool, adding new versions to built-in engines, and deleting engine versions.

Current built-in engines:

Name	Version	Description
vllm	v0.8.5	vLLM community version v0.8.5.
	v0.11.2	vLLM community version v0.11.2.
	v0.17.1	vLLM community version v0.17.1.
	v0.24.0	vLLM community version v0.24.0. This is the default version for both Kubernetes clusters and static node clusters.
sglang	v0.5.10	SGLang community version v0.5.10.
llama-cpp	v0.3.7	Llama-cpp Python high-level implementation (llama-cpp commit: 794fe23f29fb40104975c91fe19f23798f7c726e).

FunASR and MinerU can be integrated into Neutree by importing Flex Engine packages, and then used as custom engines to create endpoints.

Viewing engines

Log in to the Neutree management interface. Click Engines in the left navigation pane. The engine list on the right shows all engines. Click an engine name to enter the details page and view the task types and parameters supported by the engine.

Importing an engine

You can create an engine via the CLI tool, or add a new version to an existing engine.

Procedure

1. Create an API key and save it securely.

2. Download version 1.1.1 of the Neutree CLI tool and the specified engine package for your server’s CPU architecture.

3. Import the engine using the CLI tool:

   Import an engine package

   In offline scenarios, cluster nodes cannot access Docker Hub or other external image registries. You need to import the complete engine metadata to Neutree and import the engine image to an internal image registry. Run the following command to import the engine package and upload the images in the package to the internal image registry.

   ./neutree-cli-<arch> import engine \
   --package <engine_version_package> \
   --mirror-registry <mirror_registry> \
   [--registry-project <registry_project>] \
   --registry-username <registry_username> \
   --registry-password <registry_password> \
   --api-key <api_key> \
   --server-url <server_url>

   Parameter	Description
   <arch>	Replace with the server’s CPU architecture: amd64 or arm64.
   <engine_version_package>	The engine package name, for example vllm-v0.8.5.tar.gz.
   <mirror_registry>	The registry address. Must match the registry address used when uploading images with the CLI tool during Neutree management plane deployment. Enter an OCI-compatible registry address without the https:// prefix.
   [--registry-project <registry_project>]	Optional. The project name in the registry. The project must be created in advance. For example, specifying --registry-project neutree-ai will push the image to <mirror_registry>/neutree-ai/vllm/vllm-openai:v0.XX.XX.
   <registry_username>	The registry username. Must have permission to push images.
   <registry_password>	The registry user’s login password or access token (for example, a token).
   <api_key>	The API key created in step 1.
   <server_url>	The address of the control plane, for example http://localhost:3000.

   Import engine metadata

   Published engine images are uploaded to Docker Hub by default. If your environment can access Docker Hub and Neutree uses Docker Hub as the ImageRegistry, you can import only the engine metadata without importing the engine image.

   ./neutree-cli-<arch> import engine --skip-image-push \
   --package <manifest.yaml> \
   --api-key <api_key> \
   --server-url <server_url>

   Parameter	Description
   <arch>	Replace with the server’s CPU architecture: amd64 or arm64.
   <manifest.yaml>	The path to the manifest.yaml file, which contains the engine metadata.
   <api_key>	The API key created in step 1.
   <server_url>	The address of the control plane, for example http://localhost:3000.

4. After importing, log in to the Neutree management interface. Click Engines in the left navigation pane and confirm that the new engine or engine version appears in the engine list.

Deleting an engine version

You can delete engine versions using the CLI tool. When only one version remains, use the --force parameter to force-delete that version and remove the entire engine at the same time.

Prerequisites

Confirm that the engine version is not currently in use by any endpoint.

Procedure

Run the following command:

./neutree-cli-<arch> engine remove-version \
  --name <engine_name> \
  --version <engine_version> \
  --api-key <api_key> \
  --server-url <server_url>

Parameter descriptions:

Parameter	Description
<arch>	Replace with the server’s CPU architecture: amd64 or arm64.
<engine_name>	The name of the engine, for example vllm.
<engine_version>	The version to delete, for example v0.8.5.
<api_key>	The API key created in the prerequisites.
<server_url>	The address of the control plane, for example http://localhost:3000.
--force	Optional. Required when only one version remains — force-deletes that version and removes the entire engine.

If you delete a built-in engine, the system automatically rebuilds it.