Quickly using the latest engine version in a Kubernetes cluster

If Neutree has not yet built in the required vLLM or SGLang version, you cannot quickly use the latest upstream engine version in a Kubernetes cluster. This section provides a temporary testing procedure. The procedure uses the standalone script build-engine-package.sh to generate an engine manifest or a complete package that contains images, and then imports the engine version to Neutree.

Information:

This solution is not intended for production deployment and does not mean that Neutree provides long-term support or compatibility guarantees for this version. For production workloads, use a built-in and verified engine version.

The build-engine-package.sh script has built-in minimal schemas and Kubernetes templates for vLLM and SGLang. It can generate an engine manifest or a complete package that contains images, without calling or modifying the builder in the repository. We recommend that you use this script to generate a manifest first. After generating the manifest, manually upload the community image to the container registry configured for the Kubernetes cluster, and then import only the engine metadata. If you need to transfer images between restricted networks or prepare an offline package, you can use the script to generate a complete package instead.

Prerequisites

Before you begin, confirm the following requirements:

• The target cluster is a Kubernetes cluster.
• The engine is vLLM or SGLang, and the image reference contains an explicit tag, such as vllm/vllm-openai:v0.25.0. The script does not accept digest references.
• The administrator host has Docker and version 1.2.0 of neutree-cli installed, and can access the Neutree API.
• You have prepared a Neutree API key and the target workspace.
• You have confirmed the container registry and project configured for the target cluster, and have permission to upload images to the registry.
• Docker has permission to pull the upstream image. For private upstream images, log in to the corresponding registry in advance.

Procedure

Step 1: Download the script

Download build-engine-package.sh from the internal file server.

Step 2: Generate and import a vLLM engine

The following example verifies vllm/vllm-openai:v0.25.0:

ENGINE="vllm"
ENGINE_VERSION="v0.25.0"
UPSTREAM_IMAGE="vllm/vllm-openai:v0.25.0"
ACCELERATOR="<accelerator>"
NEUTREE_WORKSPACE="<workspace>"
export NEUTREE_SERVER_URL="<server_url>"
export NEUTREE_API_KEY="<api_key>"

TARGET_REGISTRY="<mirror_registry>"
TARGET_PROJECT="<registry_project>"
CLUSTER_IMAGE_PREFIX="${TARGET_REGISTRY}/${TARGET_PROJECT}"
TARGET_IMAGE="${CLUSTER_IMAGE_PREFIX}/vllm/vllm-openai:v0.25.0"

Parameter	Description
<accelerator>	Accelerator type, such as nvidia_gpu.
<mirror_registry>	Image registry address and optional port, without http://, https://, or the project.
<registry_project>	Image registry project. <mirror_registry> and <registry_project> must match the container registry configuration associated with the target cluster.
<TARGET_IMAGE>	Target image path. You must keep the upstream repository path vllm/vllm-openai intact, that is, <mirror_registry>/<registry_project>/vllm/vllm-openai:v0.25.0; do not shorten it to <mirror_registry>/<registry_project>/vllm-openai:v0.25.0.

Choose one of the following approaches based on the actual environment. If you can access both the upstream registry and the target registry from the administrator host, use the metadata-only approach.

Import metadata only (recommended)

This approach generates only the manifest and does not process image files. The community image must be manually uploaded to the container registry path configured for the target Kubernetes cluster.

bash build-engine-package.sh \
  --engine "$ENGINE" \
  --manifest-only \
  --version "$ENGINE_VERSION" \
  --image-ref "$ACCELERATOR=$UPSTREAM_IMAGE"

docker login "$TARGET_REGISTRY"
docker pull "$UPSTREAM_IMAGE"
docker tag "$UPSTREAM_IMAGE" "$TARGET_IMAGE"
docker push "$TARGET_IMAGE"
docker pull "$TARGET_IMAGE"

neutree-cli import engine \
  --workspace "$NEUTREE_WORKSPACE" \
  -p "dist/${ENGINE}-${ENGINE_VERSION}-manifest.yaml" \
  --skip-image-push

docker pull "$TARGET_IMAGE" verifies that the image has been uploaded to the container registry path associated with the target Kubernetes cluster. --skip-image-push only registers the engine metadata and does not check or upload images, so the manual upload step cannot be skipped.

Import complete package

This approach is suitable for scenarios where images need to be packaged for transfer into a restricted network.

If $UPSTREAM_IMAGE is not local, the script automatically pulls the image and generates a .tar.gz complete package:

bash build-engine-package.sh \
  --engine "$ENGINE" \
  --version "$ENGINE_VERSION" \
  --image-ref "$ACCELERATOR=$UPSTREAM_IMAGE"

Configure registry credentials on the import host, then import the .tar.gz package:

REGISTRY_USERNAME="<registry_username>"
REGISTRY_PASSWORD="<registry_password>"

neutree-cli import engine \
  --workspace "$NEUTREE_WORKSPACE" \
  -p "dist/${ENGINE}-${ENGINE_VERSION}.tar.gz" \
  --mirror-registry "$TARGET_REGISTRY" \
  --registry-project "$TARGET_PROJECT" \
  --registry-username "$REGISTRY_USERNAME" \
  --registry-password "$REGISTRY_PASSWORD"

Step 3: Switch to SGLang

When verifying SGLang, replace the corresponding variables in Step 2 with the following values:

ENGINE="sglang"
ENGINE_VERSION="nightly"
UPSTREAM_IMAGE="lmsysorg/sglang:nightly"
TARGET_IMAGE="${CLUSTER_IMAGE_PREFIX}/lmsysorg/sglang:nightly"

No other variables need to be changed. After setting them, refer to Step 2 and choose either approach to generate and import the engine.

The built-in SGLang profile supports text-generation and text-embedding, but not text-rerank.

Validating the engine and endpoint

1. Log in to the Neutree management interface and confirm that the imported $ENGINE_VERSION is shown under the corresponding engine.

2. Create a temporary endpoint and select the imported engine version. Do not use an unverified version for production workloads.

3. Observe the endpoint Pod, Events, and engine logs. Wait for the endpoint status to become Running.

4. Run a test request that matches the task type in the endpoint Playground or API. vLLM can be validated with text-generation, text-embedding, or text-rerank; SGLang can only be validated with text-generation or text-embedding.

Rollback and considerations

Before rolling back, delete the test endpoint that uses this engine version, or switch it to another verified version, and then run:

neutree-cli engine remove-version \
  --workspace "$NEUTREE_WORKSPACE" \
  --name "$ENGINE" \
  --version "$ENGINE_VERSION"

Removing an engine version does not delete the image from the container registry. Clean up $TARGET_IMAGE separately according to your registry retention policy. If you need to preserve the problem state or rollback point, do not delete the image in advance.

Upstream versions may change startup parameters, dependencies, or image contents. This temporary procedure does not provide production support guarantees. For production workloads, use a built-in and verified engine version.