Manually importing models to the model registry

Issue

Neutree supports pushing models to the model registry using the neutree-cli-<arch> model push command (see Pushing a model), but this method uploads model files via the API and can be time-consuming for large models. You may also encounter scenarios where you need to manually import already-downloaded model files into the Neutree model registry:

Model files have been transferred to a node via USB drive, hard disk copy, or other offline methods and need to be imported for use.
Model files were manually downloaded from Hugging Face or another source and need to be registered in the Neutree model registry.
The network environment is restricted and Neutree cannot automatically download models, so models must be downloaded in another environment and then imported.

Solution

If the model files are already on the node where the NFS storage is mounted, this section provides the import-model.sh script to import model files directly into the model registry. The script imports by local file copy, bypassing the network transfer step for faster and simpler operation.

Prerequisites

Model files have been downloaded to a local directory on the node.
The node has NFS storage mounted.
The node has Python 3 installed. Used to auto-generate the version number; can be ignored if you specify the version number manually.

Step 1: Create the script

Create the import script import-model.sh on the node where the model files are located, with the following content:

#!/bin/bash
# Usage: ./import-model.sh <source_dir> <model_name> [version] [nfs_path]

set -e

SOURCE_DIR="$1"
MODEL_NAME="$2"
VERSION="$3"
NFS_PATH="${4:-/mnt/bentoml}"

if [ -z "$SOURCE_DIR" ] || [ -z "$MODEL_NAME" ]; then
    echo "Usage: $0 <source_dir> <model_name> [version] [nfs_path]"
    echo "Example: $0 /tmp/qwen2-0.5b-instruct qwen2-0.5b-instruct"
    echo "Example: $0 /tmp/qwen2-0.5b-instruct qwen2-0.5b-instruct v1.0"
    echo "Example: $0 /tmp/qwen2-0.5b-instruct qwen2-0.5b-instruct v1.0 /mnt/bentoml"
    exit 1
fi

# Convert model name to lowercase
MODEL_NAME=$(echo "$MODEL_NAME" | tr '[:upper:]' '[:lower:]')

# Auto-generate version if not provided
if [ -z "$VERSION" ]; then
    VERSION=$(python3 -c "import uuid, base64; u=uuid.uuid1(); print(base64.b32encode(u.bytes[:6]+u.bytes[8:12]).decode().lower().rstrip('='))")
fi
VERSION=$(echo "$VERSION" | tr '[:upper:]' '[:lower:]')

TARGET_DIR="${NFS_PATH}/models/${MODEL_NAME}/${VERSION}"

echo "Importing model: ${MODEL_NAME}"
echo "Version: ${VERSION}"
echo "Target directory: ${TARGET_DIR}"

# Create directory
mkdir -p "${TARGET_DIR}"

# Copy files
echo "Copying model files..."
cp -r "${SOURCE_DIR}"/* "${TARGET_DIR}/"

# Calculate directory size
SIZE=$(du -sh "${TARGET_DIR}" | cut -f1)

# Generate model.yaml
cat > "${TARGET_DIR}/model.yaml" << EOF
name: ${MODEL_NAME}
version: ${VERSION}
size: ${SIZE}
module: ""
api_version: v1
signatures: {}
labels: {}
options: {}
metadata: {}
context:
    framework_name: transformers
    framework_versions: {}
    bentoml_version: 1.4.6
    python_version: "3.12"
creation_time: "$(date -u +"%Y-%m-%dT%H:%M:%S.000000+00:00")"
EOF

# Update latest
echo -n "${VERSION}" > "${NFS_PATH}/models/${MODEL_NAME}/latest"

# Set permissions
chmod -R 755 "${NFS_PATH}/models/${MODEL_NAME}"

echo "Import complete!"
echo "Model tag: ${MODEL_NAME}:${VERSION}"

Grant the script executable permissions:

chmod +x import-model.sh

Step 2: Run the script to import the model

Run the script to import the model into the model registry:

./import-model.sh <source_dir> <model_name> [version] [nfs_path]

Parameter description:

Parameter	Required	Description
`<source_dir>`	Yes	Local directory path where the model files are located.
`<model_name>`	Yes	Model name. The script automatically converts it to lowercase.
`[version]`	No	Model version number. If not specified, the script auto-generates a unique version number.
`[nfs_path]`	No	Mount path of the NFS storage. Default is `/mnt/bentoml`.

Examples

Import a model with default settings (auto-generated version number, default NFS path):
Terminal window
```
./import-model.sh /tmp/qwen2-0.5b-instruct qwen2-0.5b-instruct
```

Import a model with a specified version number:

./import-model.sh /tmp/qwen2-0.5b-instruct qwen2-0.5b-instruct v1.0

Import a model with a specified version number and custom NFS path:

./import-model.sh /tmp/qwen2-0.5b-instruct qwen2-0.5b-instruct v1.0 /mnt/bentoml

Step 3: Verify the import result

After the import is complete, the script outputs the model tag in the format model_name:version. You can use the neutree-cli-<arch> model list command to verify that the model is correctly recognized by Neutree. For details, see Viewing models.