Skip to content
Neutree Documentation

Managing external endpoints

External endpoints allow you to connect external API services, such as OpenAI, Azure OpenAI, other third-party inference services, or Neutree internal endpoints, to the AI Gateway through a unified proxy. This provides unified authentication, routing, and usage statistics, and avoids exposing provider API keys on the client side.

Creating an external endpoint

  1. Log in to the Neutree management interface, click External Endpoints in the left navigation pane, then click Create on the right.

  2. Fill in the configuration.

    • Basic Information

      ParameterDescriptionEditable after creation
      NameThe name of the external endpoint. The name must contain only lowercase letters, digits, _, -, or ., can be up to 63 characters long, and must start and end with a letter or digit.No
      WorkspaceThe workspace the external endpoint belongs to.No

    • Configuration

      Set the timeout for waiting for an upstream response. The default is 60 seconds; 5 minutes is recommended. The timeout is the time Kong gateway waits for a complete response from the upstream after the connection is established. If the upstream server stalls and the idle time exceeds the configured value, Kong gateway triggers a timeout and returns a 504 Gateway Timeout response. Editable after creation.

    • Exposed model

      An exposed model is the model name that clients use when calling an external endpoint. You can select a routing strategy for each exposed model and bind it to a specific model in one or more upstream channels.

      ParameterDescriptionEditable after creation
      Exposed ModelThe model name exposed to clients. The model name in client requests must match this name.Yes
      Model sourceThe source type of the model. You can select Private self-hosted, Private access, Public cloud, or Hybrid. If no source is explicitly selected and all targets of the exposed model are internal endpoints, Neutree identifies the model source as Private self-hosted.Yes
      Routing StrategyHow requests are distributed among upstream models. Single Route, Active & Standby, and Weighted Routing are supported.You can switch only from Single Route to Active & Standby or Weighted Routing.

      The behavior and configuration fields for the three routing strategies are as follows:

      Forwards all requests for the exposed model to one upstream model.

      ParameterDescription
      Upstream ChannelThe upstream channel to which requests are forwarded.
      Upstream Model NameThe model name actually received by the upstream service. After you select an upstream channel, the system suggests model names based on the model list returned by that channel. You can also type a model name manually.
      Max concurrent requestsThe maximum number of requests that the upstream model can process simultaneously. The platform determines whether the limit is reached based on the number of requests that this target is currently processing and has not yet completed. Leave this field empty or set it to 0 for no limit. When the limit is reached, requests fail if no other target is available.

    • Upstream channels

      Upstream channels define the services to which requests are actually forwarded. You can configure one or more upstream channels and select a target through the Upstream Channel field in an exposed model’s routing configuration. Upstream channels support External and Endpoint Ref types.

      ParameterDescriptionEditable after creation
      Upstream nameThe identifier of the upstream channel. Upstream names cannot be duplicated within the same external endpoint.Yes
      Upstream typeExternal forwards requests to an external API service; Endpoint Ref forwards requests to an internal endpoint in the same workspace.Yes
      Upstream URLFor the External upstream type, enter the complete URL of the external API, including the API path, for example https://api.openai.com/v1.Yes
      API keyFor the External upstream type, enter the Bearer token for the upstream API.Yes
      Internal endpointFor the Endpoint Ref upstream type, select an internal endpoint in the same workspace.Yes

      Compatibility with legacy model mappings:

      External endpoints created before the upgrade that still use legacy model mappings continue to run. When you edit such an external endpoint, the Neutree management interface converts each legacy model mapping to a Single Route: the old mapping’s exposed model name becomes the new Exposed model, the upstream model name is retained, and the upstream containing the old mapping becomes the new Upstream Channel. After you save, Neutree uses the new exposed-model routing configuration and no longer retains the legacy model mappings.

  3. (Optional) Test the connection status of the upstream service. Click the Test Connection button in the upstream configuration. The system automatically verifies that the upstream URL and API key are available. After a successful test, the response latency and the number of available models are displayed. If the upstream model list fails to be obtained, Upstream connection failed is displayed next to the selected upstream channel. Check the upstream URL, API key, and network connectivity. A connection failure does not prevent you from manually entering an upstream model name.

  4. After confirming the configuration, click Save to complete creation.

Viewing external endpoints

Log in to the Neutree management interface and click External Endpoints in the left navigation pane. The top of the right panel shows an overview of external endpoints, and the list below shows all current external endpoints. Click an external endpoint name to view its details.

The external endpoint details page displays the routing strategy, model source, upstream channel, upstream model, traffic weight or active/standby role, and maximum concurrent requests by exposed model. For an external endpoint with exposed model routing configured, you can go to the Monitor tab to view model routing monitoring, or click View monitoring to open the full monitoring panel. External endpoints that do not have exposed model routing configured and use only model mapping do not provide routing monitoring.

The possible statuses of an external endpoint during operation are described below:

StatusDescription
PendingThe external endpoint is waiting for the first synchronization.
RunningThe external endpoint is running normally, and all upstreams are available.
DegradedAt least one upstream in the external endpoint is unavailable. Neutree excludes abnormal upstreams from the gateway configuration and continues to deliver configurations for other available upstreams. Clients can still call models corresponding to other available upstreams.
FailedThe external endpoint is abnormal. If all upstreams are unavailable, the external endpoint enters this status. Check the upstream status, error information, and logs.
DeletedThe external endpoint has been deleted.

On the external endpoint details page, you can also view the status of each upstream. The possible upstream statuses are described below:

StatusDescription
ReadyThe upstream has been successfully delivered to the gateway configuration.
FailedThe upstream is unavailable and has been excluded from the gateway configuration. The details page displays error information and Unavailable Models. Before the abnormal upstream recovers, requests sent to these models fail. Requests sent to other available models are not affected.

Getting the API URL

On the details page, Service URL is the API URL that external services use to call this external endpoint. Usage is the same as for a regular endpoint. After creating an API key for any OpenAI- or Anthropic-compatible client, you can use the API key to call this API URL for integration.

Information:

When using the API URL, make sure to append /v1 to the URL.

For example, to view the models of an external endpoint via API:

Terminal window
curl  <API_URL>/v1/models \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer <api_key>'

Curl examples

  1. Under Curl Example on the details page, select a model name, then click the copy icon Copy icon to copy the corresponding command.

  2. Create an API key for any OpenAI- or Anthropic-compatible client and save it securely.

  3. Replace $ENDPOINT_API_KEY in the command copied in step 1 with the API key created in step 2 (or set the ENDPOINT_API_KEY environment variable), then run the command on the client to call the external endpoint’s API URL.

Editing an external endpoint

After an external endpoint is created, you can modify its upstream configuration as needed.

  1. Log in to the Neutree management interface, click the menu icon (…) in the external endpoint list or details page, and select Edit.

  2. Modify the configuration as needed. For parameter descriptions, see Creating an external endpoint.

  3. After confirming the configuration, click Save to complete editing.

Deleting external endpoints

You can delete one or more external endpoints at a time.

  1. Log in to the Neutree management interface, click the menu icon (…) in the external endpoint list or details page, and select Delete; or select multiple external endpoints in the list and click Delete above the list.

  2. In the dialog that appears, confirm and click Delete. The selected external endpoints will be permanently deleted.