Managing endpoints

Managing endpoints

It is recommended to use the model catalog to create endpoints. Neutree will automatically populate the relevant configurations from the model catalog. After making any necessary modifications, you can quickly create an endpoint. Neutree also supports creating endpoints without selecting a model catalog by manually entering the configuration parameters.

Creating an endpoint

1. Log in to the Neutree management interface. In the left navigation pane, click Endpoints, then click Create on the right-side page.

   You can also click Deploy on a model catalog card on the Model Catalogs page to go to the endpoint creation page with that model catalog automatically selected.

2. Fill in the configuration information.

   Parameter	Description	Editable after creation
   Name	The name of the endpoint.	No
   Workspace	The workspace that the endpoint belongs to.	No
   Cluster	The target to schedule the endpoint to. The page displays the number of nodes and available resources based on the selected cluster.	Yes
   Model registry	The model registry that the endpoint belongs to.	Yes
   Model catalog (template)	Optional. After you select a standard model catalog, the built-in model, resource, engine, and custom configurations of the model catalog are populated into the endpoint configuration, which you can modify as needed. After you select a model catalog that includes variants and features, you need to continue selecting variants and features. Neutree generates the final configuration based on the selected combination. If no model catalog is selected, you need to fill in the model configuration manually.	No
   CPU	The number of CPU cores allocated to each replica of the endpoint.	Yes
   Memory	The amount of memory allocated to each replica of the endpoint.	Yes
   Accelerator	Select the accelerator type and model. The page displays available accelerators based on the selected cluster. For a model catalog marked as verified hardware, the page preferentially displays verified accelerators that match the current cluster.	Yes
   Card Count	The number of accelerators allocated to each replica. When Accelerator Virtualization is not enabled, this is the number of physical accelerator cards. When Accelerator Virtualization is enabled and Virtual card split resources are configured, this is the number of physical cards to slice.	Yes
   Virtual card split resources	Configure only when Accelerator Virtualization is enabled for the selected Kubernetes cluster. Allows you to set the VRAM and Core Limit for each virtual card.	Yes

   Note

   • When using a model catalog, ensure that the model registry contains a model that matches the model catalog configuration. Otherwise, endpoint creation may fail.
   • If the CPU, memory, full accelerator, or virtual accelerator resources exceed the available capacity of the current cluster, the page displays an insufficient resource prompt and does not allow you to save the endpoint configuration. Adjust the resource configuration or select another cluster and try again.
   • When using virtual accelerator resources, the target cluster must be a Kubernetes cluster with Accelerator Virtualization enabled and the virtualization components ready. Virtual accelerators support only NVIDIA GPUs, and you must specify the accelerator model and the GPU memory for each virtual accelerator. The GPU memory of each virtual accelerator cannot exceed the GPU memory of the corresponding physical GPU.

3. If you select a model catalog that includes variants and features, select and confirm the template configuration in the Recipe options area as needed.

   • Variant: Select a model variant. Different variants can correspond to different models, resources, minimum GPU memory, and parameter overrides.
   • Features: Select or enter the features to enable. Different features may correspond to different engine variables or environment variables. Features that conflict with each other cannot be enabled at the same time.

   You can view the verified hardware declared by the model catalog, whether GPU memory conflicts exist, and the model, resources, engine variables, and environment variables generated from the selected basic configuration, variant, and feature combination.

4. Click Customize Settings and fill in the custom configuration as needed.

   If you specified a standard model catalog in the basic configuration, the custom configuration fields are automatically populated with the built-in parameters of that model catalog. You can edit them as needed. Changes made here do not affect the model catalog. If you select a model catalog that includes variants and features, the final configuration is generated from the basic configuration, variants, and features. If no model catalog is specified, fill in the custom configuration as needed.

   • Model Settings

     Parameter	Description	Editable after creation
     Model name	The name of the model used by the endpoint.	Yes
     Model version	The version of the model used by the endpoint. For multi-version models, you can specify a particular version as needed. If left empty, Neutree will automatically use the latest model version: the main version for Hugging Face model registries, and the latest version for file system model registries.	Yes
     Model file	The model file used by the endpoint. Enter the entry file in the model folder. Models of the safetensors type do not require this field. For models of the GGUF type, select the GGUF file of the required quantization version as the entry file. For example, for the 8-bit quantization version, select the file ending in *8_0.gguf as the entry file.	Yes

   • Engine

     Parameter	Description	Editable after creation
     Engine	The engine for the endpoint. vllm: A mainstream open-source engine with efficient model inference capabilities, suitable for NVIDIA GPU and AMD GPU scenarios. sglang: A high-performance engine suitable for NVIDIA GPU scenarios. Supports text generation and text embedding tasks only. llama-cpp: A lightweight engine suitable for CPU-only scenarios. Requires models in GGUF format.	Yes
     Engine version	The engine version for the endpoint. Defaults to the latest currently available engine version; you can select a specific version as needed.	Yes
     Task type	The task type for the endpoint. An endpoint may support multiple task types; select the task type that matches the model you are using. Text generation: The most common LLM inference scenario. Text embedding: Generates vector representations of text, commonly used for text similarity computation. Text reranking: Ranks text based on similarity, commonly used for search result reranking.	Yes

   • Replica Settings

     Parameter	Description	Editable After Creation
     Replicas	The replica policy for the endpoint. Multiple replicas enable high availability; each replica consumes one set of resource allocations.	Yes
     Scheduler type	When using the vLLM engine, it is recommended to use the Consistent hashing policy. The scheduler will be aware of the KVCache distribution in the engine and intelligently optimize inference request routing to effectively improve KVCache hit rates. For other scenarios, the Round robin policy is recommended to maintain even load distribution.	Yes

   • Advanced Options

     • Engine variables: You can select parameters defined in the engine or enter custom key-value pairs. If a selected parameter’s data type is an object or array, enter its value in JSON object or array format. After the endpoint is created, you can edit existing engine variables and add new key-value pairs. When using the vLLM or SGLang engine with multiple accelerators configured, the system will automatically set the corresponding tensor parallelism parameter equal to the number of accelerators; no manual configuration is required.
     • Environment variables: You can enter custom key-value pairs. After the endpoint is created, you can edit existing environment variables and add new key-value pairs.

5. After confirming the configuration is correct, click Save to complete the creation.

Viewing endpoints

Log in to the Neutree management interface. In the left navigation pane, click Endpoints. The endpoint list on the right will display all current endpoints. Click an endpoint name to view its details. On the details page, you can view Basic, Monitor, Logs, and Playground as needed.

The possible statuses during endpoint operation and their descriptions are as follows:

Status	Description
Pending	The endpoint has been created and is waiting to be scheduled or initialized.
Deploying	The endpoint is creating the workloads required to run.
Model Downloading	The endpoint is downloading or preparing model files.
Running	The endpoint is operating normally and can serve requests.
Paused	The endpoint is paused and not serving requests.
Failed	The endpoint encountered an error. Check the error message and logs.
Deleting	The endpoint is being deleted; resources are being cleaned up.
Deleted	The endpoint has been deleted.

Getting the API URL

On the Basic tab of the details page, the Service URL is the API URL that external services use to call this endpoint. After creating an API key for any OpenAI API-compatible client, the client can use the API key to call this API URL for integration.

Note

When using the API URL, make sure to append /v1 to the URL.

For example, to view the models of an endpoint via the API:

curl  <API_URL>/v1/models \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer <api_key>'

Viewing monitoring and logs

Neutree provides monitoring and logs to help you understand the running status and performance metrics of endpoints, enabling you to identify and resolve issues promptly.

• Monitor: Select the Monitor tab on the details page to view real-time monitoring information for the endpoint. You can filter monitoring data by time range and set the auto-refresh frequency for monitoring data.

  • For multi-replica endpoints, you can use the replica selector to view monitoring metrics for individual replicas.
  • For endpoints using the vLLM or SGLang engine, you can view monitoring dashboards such as overview, latency, throughput, queue, and cache.
  • Latency data in engine monitoring dashboards shows aggregated values from the past 1 minute.

• Logs: Select the Logs tab on the details page to view application logs, error output, and standard output of the endpoint. You can quickly filter logs by keyword and time range, with support for log download and auto-refresh.

• Resources: When the cluster can report GPU card-level resource usage, the details page displays the node, GPU card model, VRAM usage, and compute usage for each endpoint replica, allowing you to confirm resource scheduling results. For endpoints using virtual accelerator resources, the Basic tab also shows Allocated Resources, including accelerator model, VRAM, Core Units, and the node and device information for each replica.

Quick testing

Neutree provides a Playground where you can quickly verify that an endpoint’s functionality is working correctly and that the model’s output meets expectations, enabling you to adjust the endpoint configuration promptly.

Select the Playground tab on the details page. The platform will display the corresponding test interface based on the endpoint’s task type.

• For Text Generation tasks, a conversation window is displayed for testing the basic functionality of the inference API.

• For Text Embedding tasks, a set of editable text entries is displayed. After clicking Generate, the similarity relationships between the generated text vectors are displayed.

• For Text Rerank tasks, an editable prompt and a set of editable related texts are displayed. After clicking Generate, the reranking results showing the relevance between the texts and the prompt are displayed.

Editing endpoints

After an endpoint is created, you can modify its resource, model, engine, replica, and advanced options configuration based on actual requirements.

1. Log in to the Neutree management interface. In the endpoint list or details page, click the menu icon (…) and select Edit.

2. Modify the configuration as needed. For detailed parameter descriptions, refer to Creating an endpoint.

3. After confirming the configuration is correct, click Save to complete the edit.

Pausing an endpoint

When an endpoint is temporarily not needed, you can suspend it to release the resources it occupies. After suspending an endpoint, the accelerator resources (such as GPUs), CPU, and memory it occupies will be released, and you can allocate these resources to other endpoints or applications. During suspension, the endpoint’s configuration, model files, and history are fully retained, and you can resume the endpoint at any time to continue providing inference services.

Note

After an endpoint is suspended, it will no longer be able to provide inference services. Make sure to notify relevant users or systems before suspending.

Procedure

1. Log in to the Neutree management interface. In the endpoint list or details page, click the menu icon (…) and select Pause. The endpoint will enter the suspended state and its resources will be released.

2. Verify that the endpoint status has been updated to Paused.

Follow-up

To resume an endpoint, click the menu icon (…) in the endpoint list or details page and select Resume. The endpoint will be reallocated resources and resume operation.

Deleting endpoints

You can delete one or more endpoints at the same time.

1. Log in to the Neutree management interface. In the endpoint list or details page, click the menu icon (…) and select Delete; or in the endpoint list, select multiple endpoints to delete and click Delete above the list.

2. In the dialog box that appears, confirm the deletion and click Delete. The selected endpoints will be permanently deleted.