Managing Nodes

This guide covers all aspects of node management, from adding new machines to configuring settings and handling customer requests.

Adding New Nodes

Prerequisites

Before adding an node, ensure:

1. Node Software Running: Your GPU machine has the node program running
2. Network Access: Machine is accessible from the internet
3. Miner Server Connected: Your central miner server is running and connected
4. Wallet Setup: EVM address is associated with your hotkey

Registration Process

1. Access Add Executor Modal:

   • Navigate to the Executors page
   • Click the "Add Executor" button
   • A popup modal will appear

2. Fill Required Information:

   • GPU Type: Select from available options (RTX 4090, A100, etc.)
   • GPU Count: Enter the number of GPUs in the machine
   • Machine IP Address: Public IP address of your node
   • Port: Port number where the node program is running
   • GPU Price: Hourly price per GPU (in USD)

3. Submit Registration:

   • Review all information for accuracy
   • Click "Add Executor" to register

Synchronization

Sync From the Central Miner Server

Sync nodes from your miner server to the portal:

• Purpose: Import node configurations from your miner server
• When to Use: After setting up nodes directly on your server
• Process: Click "Sync From Miner Server" button
• Result: Nodes configured on your server appear in the portal

Note

When you first login on portal, the sync process will be done automatically for you.

Sync Into Miner Server

Sync node configurations from the portal to your miner server:

• Purpose: Export portal configurations to your miner server
• When to Use: After making changes in the portal that need to be reflected on your server if your server lost connection at that moment.
• Process: Click "Sync Into Miner Server" button
• Result: Portal configurations are applied to your miner server

Note

The syncing process is done automatically as long as your miner server is connected to the portal backend and you barely need to do manual sync at all.

Synchronization

Always ensure your miner server is running and accessible before attempting synchronization.

Post-Registration Steps

After successful registration:

1. Deposit Collateral: Fund your node to start receiving score and incentives on subnet
2. Monitor Status: Check that the node appears as active
3. Configure Settings: Set notice periods and other parameters

Price Management

Updating Prices

1. Access Price Settings:

   • Navigate to your node details page
   • Click the "Update Price" button
   • Enter the new hourly rate per GPU (in USD) in the GPU Price field
   • Note: The Machine Price field is deprecated and will be ignored

2. Price Considerations:

   • Minimum Viable Price: Set prices that cover your operational costs
   • Competitive Pricing: Research market rates to stay competitive
   • GPU Count Impact: Total rental cost = GPU Price × Number of GPUs rented

Price Updates

Price changes take effect immediately and apply to all new rental requests. Existing active rentals continue at their original rate until they end.

GPU Splitting

GPU splitting allows you to rent individual GPUs from a single node to multiple customers simultaneously, enabling more efficient resource utilization and flexible pricing.

Prerequisites

To enable GPU splitting, you must configure your Docker storage driver to use XFS as the backing filesystem with overlay2. This setup is required for proper GPU isolation and resource management.

Before you begin:

• Ensure you have administrative access to the node
• Prepare a new drive or partition for Docker storage
• Back up any existing Docker data before making changes

1. Prepare New Drive/Partition

   • Identify or create a new partition for Docker storage
   • Ensure sufficient space for your Docker images and containers

2. Stop Docker Services

   sudo systemctl stop docker
   sudo systemctl stop docker.socket

3. Configure overlay2 Storage Driver If your Docker storage driver is not already set to overlay2, configure the Docker daemon:

   # Open Docker daemon configuration
   sudo nano /etc/docker/daemon.json

   Add or update the configuration:

   {
     "storage-driver": "overlay2"
   }

4. Set Up XFS Backing Filesystem

   # Set your new partition device name
   new_partition=<your-new-partition-device-name>
   
   # Copy the contents of /var/lib/docker to a temporary location
   sudo rsync -aXS /var/lib/docker/ /tmp/docker-backup/
   
   # Format partition to XFS filesystem with ftype=1
   sudo mkfs.xfs -n ftype=1 $new_partition -f
   
   # Mount partition to /var/lib/docker with pquota enabled
   sudo mount -t xfs -o defaults,inode64,pquota $new_partition /var/lib/docker
   
   # Restore the backup content to /var/lib/docker
   sudo rsync -aXS /tmp/docker-backup/ /var/lib/docker/
   
   # Add this mount to /etc/fstab to make it permanent
   UUID=$(sudo blkid -s UUID -o value $new_partition)
   echo "UUID=${UUID}  /var/lib/docker  xfs  defaults,inode64,pquota  0  2" | sudo tee -a /etc/fstab

5. Restart Docker

   sudo systemctl start docker

6. Verify the Setup Run the docker info command and verify the following settings:

   • Storage Driver: overlay2
   • Backing Filesystem: xfs
   • Supports d_type: true

   Example output:

   Server:
    Containers: 6
     Running: 6
     Paused: 0
     Stopped: 0
    Images: 8
    Server Version: 28.3.2
    Storage Driver: overlay2
     Backing Filesystem: xfs
     Supports d_type: true
     Using metacopy: false
     Native Overlay Diff: true
     userxattr: false

Setting Minimum GPU Count for Rental

After configuring the prerequisites, you must set a minimum GPU count to enable GPU splitting on your node.

1. Access GPU Splitting Settings:

   • Navigate to your node details page in the portal
   • Locate the "GPU Splitting" section in the details panel
   • If your node meets all prerequisites, you will see an "Edit" button

2. Configure Minimum GPU Count:

   • Click the "Edit" button
   • Set the minimum GPU count (must be greater than 1)
   • Save your changes

Minimum GPU Count

GPU splitting requires a minimum GPU count to be configured. If the minimum GPU count is not set, the node will not support GPU splitting, and customers will only be able to rent all GPUs on the node. Once configured, customers can rent any number of GPUs from the minimum up to the total available GPUs on the node.

Disabling GPU Splitting: You can remove the minimum GPU count to disable GPU splitting. However, you can only remove it when there are no pods currently renting a partial number of GPUs.

Notice Period Management

Understanding Notice Periods

Notice periods allow you to schedule maintenance for rented machines without deposited collateral being slashed:

• Requirement
  • Notice period should be scheduled 24 hours ahead.
  • Maximum period is 60 mins
• Purpose:
  • Give customers advance warning of planned downtime
  • Give miners safe machine maintenance period without collateral getting slashed

Setting Notice Periods

• Go to node management page
• Find the "Notice Period" section
• Select the start time. (minimum 24 hours later from now)
• Enter desired notice period (maximum 60 mins)

Note

Once the notice period is scheduled, it will send an email to the customer who rented the machine and let them know.

Notify Machine Request from Customer

Respond to customer requests efficiently:

1. Access Notify Feature:

   • Go to your node details page
   • Click the "Notify" button in the top-right corner
   • A modal window displays available machine requests

2. Select and Notify:

   • Review available machine requests
   • Select appropriate requests based on your node's capabilities
   • Click "Notify" to send notification to the renter

Node Deletion

When to Delete an Node

Consider deleting an node when:

• Hardware Issues: Machine is no longer functional
• Upgrade Plans: Replacing with better hardware
• Cost Optimization: Reducing operational costs
• Relocation: Moving to a different location

Deletion Process

Important

Deleting an node without proper reclaim process will result in collateral slashing. Always follow the reclaim procedure first.

1. Reclaim Collateral First:

   • Submit a reclaim request
   • Wait for approval
   • Finalize the reclaim process

2. Delete Node:

   • Delete from the portal

Troubleshooting

Common Issues

Node Not Appearing

• Deposit Collateral: Verify the required collateral amount is deposited
• Check Registration: Verify all required fields were filled correctly
• Network Connectivity: Ensure machine is accessible from the internet
• Miner Server: Confirm your miner server is running and connected
• Validation: Wait for validation process (minimum 15 minutes) to complete