Skip to content
Neutree Documentation

What's in this release

What’s new

Section titled “What’s new”
  • Supports virtualizing NVIDIA GPUs in Kubernetes clusters, enabling flexible slicing of GPU memory and computing power.
  • Built-in inference engines:
    • Adds SGLang v0.5.10, supporting text generation and text embedding tasks.
    • Adds vLLM v0.24.0, with support for models such as MiniMax M3, Gemma 4, and Mellum v2, and enhanced capabilities for DeepSeek, Qwen, and other models, along with improvements to inference performance, memory and parallelism capabilities, and hardware ecosystem support for FP8, ROCm/MI300, and CUDA.
  • Adds an Access Log page for viewing request records, response status, token usage, as well as request and response details for endpoints and external endpoints.
  • Adds a Model Usage page for viewing token usage statistics by date, API key, endpoint type, and model.
  • Model catalog supports variant and feature combinations, and supports quick endpoint deployment from the model catalog list.
  • The AI gateway supports limiting the invocation scope of endpoints and external endpoints based on workspace permissions inherited by API keys.
  • API keys support configuring token quotas, request rate limits, concurrency limits, and allowed models, and can be disabled or re-enabled.
  • Adds a ModelDownloading status for endpoints to indicate the model download phase during endpoint startup.
  • Supports visualizing computing resource usage during model deployment, improving deployment efficiency.

Improvements

Section titled “Improvements”
  • Cluster details now display node-level GPU card information, including card model, UUID, health status, video memory, and compute usage.
  • Neutree supports automatic installation of cluster monitoring collection components on managed static node clusters and Kubernetes clusters.
  • In Endpoints, the Resource Settings and Configuration Details sections support displaying the resource usage of cluster GPUs and the resource usage of instance replicas.
  • Model catalog supports card-style display, search by name, viewing variant and feature information, and editing the spec of imported model catalogs.
  • The AI gateway usage statistics support recording extended fields, including cached tokens, inference tokens, cost, and response message ID.
  • The aggregation frequency for API key token quota usage has been optimized from approximately 5 minutes to 30 seconds, reducing the synchronization delay between quota usage and actual call volume.
  • The CLI tool supports validating version compatibility when starting the control plane to prevent mismatched core component versions.
  • The CLI tool supports importing models directly via a local NFS mount path when pushing models, reducing the time required to upload large models.
  • Enhances critical error information when a static node cluster fails to start.

Resolved issues

Section titled “Resolved issues”
  • Anthropic protocol conversion had compatibility issues in scenarios such as streaming output, tool calls, image content, and role handling. The issue has been resolved in this release.
  • When the AI gateway forwards OpenAI or Anthropic-compatible requests, empty tools, stop, or stop_sequences fields could cause processing errors in some upstream services. The issue has been resolved in this release.
  • Paused Kubernetes endpoints did not trigger reconciliation correctly after resuming. The issue has been resolved in this release.
  • Endpoint names in the Kubernetes engine template were not quoted, causing some names to fail deployment. The issue has been resolved in this release.
  • External endpoint service addresses were not synchronized promptly after being changed. The issue has been resolved in this release.
  • Boolean engine variables set to false in Kubernetes endpoint templates were handled incorrectly. The issue has been resolved in this release.
  • Engine variables of object or array type were not saved as structured data when creating or editing endpoints. The issue has been resolved in this release.
  • When importing offline package images using the CLI tool, non-Docker Hub image paths might be incorrectly appended with the library/ namespace. The issue has been resolved in this release.
  • An error occurred in node IP selection during control plane docker compose installation.
  • The upstream API key of an external endpoint could be retrieved via the credentials API. After saving, the key is no longer returned through the page or API. The issue has been resolved in this release.
  • When editing a cluster, image registry, model registry, or external endpoint, empty credential fields that were not modified could overwrite existing configurations. The issue has been resolved in this release.
  • Models referenced by endpoints could be deleted, causing the related endpoints to become unavailable. The issue has been resolved in this release.