Integrating external monitoring components

Integrating external monitoring components

By default, the Neutree manages monitoring components on compute clusters. If external GPU monitoring components have already been deployed in the target Kubernetes cluster or static node cluster, you can adjust how monitoring components are integrated through cluster.spec.config.metrics. Currently, only accelerator metric collection components can be switched to external mode. Node metric collection components are still managed by Neutree.

The following management modes are supported for accelerator metric collection components:

Configuration item	Value	Description
spec.config.metrics.accelerator_exporter.mode	managed	Default value. Neutree deploys and collects metrics from managed accelerator metric components.
spec.config.metrics.accelerator_exporter.mode	external	Neutree does not deploy managed accelerator metric components. Instead, it collects metrics from existing external accelerator metric components in the cluster.

Configuration example

apiVersion: v1
kind: Cluster
metadata:
  name: <cluster-name>
  workspace: <workspace-name>
spec:
  type: kubernetes
  version: "v1.1.0"
  image_registry: <image-registry>
  config:
    metrics:
      accelerator_exporter:
        mode: external
    kubernetes_config:
      kubeconfig: <kubeconfig>

Operation example

To switch an existing cluster to external accelerator metric component mode, refer to the following example:

1. Export the existing cluster configuration:

   neutree-cli-<arch> get Cluster <cluster-name> \
     -w <workspace-name> \
     -o yaml \
     --api-key <api_key> \
     --server-url <server_url> \
     > external-accelerator-exporter.yaml

2. Edit external-accelerator-exporter.yaml and add or adjust the following fields:

   spec:
     config:
       metrics:
         accelerator_exporter:
           mode: external

3. Save the file and run:

   neutree-cli-<arch> apply -f external-accelerator-exporter.yaml \
     --api-key <api_key> \
     --server-url <server_url>

4. View the updated cluster configuration:

   neutree-cli-<arch> get Cluster <cluster-name> \
     -w <workspace-name> \
     -o yaml \
     --api-key <api_key> \
     --server-url <server_url>

   When using external mode, make sure that an external accelerator metric component that can be collected by the Neutree has already been deployed in the target cluster. In NVIDIA GPU scenarios, the external DCGM Exporter must expose metrics in Prometheus format. It typically uses the app=nvidia-dcgm-exporter label and port 9400.

5. After configuration is complete, confirm that the external component is accessible.

   For Kubernetes clusters, run the following commands:

   kubectl get pods -A -l app=nvidia-dcgm-exporter
   kubectl port-forward -n <namespace> <dcgm-exporter-pod> 9400:9400
   curl http://127.0.0.1:9400/metrics

   For static node clusters, confirm that Neutree can access the corresponding Prometheus metrics port based on the existing DCGM Exporter or DCGM deployment method.

   If the external component label, port, or metric format does not meet the preceding requirements, the Neutree may fail to collect GPU monitoring data.